Recherche avancée

Sur d’autres sites (9122)

• Libavcodec : How to tell end of access unit when decoding H.264 stream

  22 septembre 2011, par Flame

  I'm receiving H.264 video over RTP and decoding it with libavcodec. I'm unpackaging the NAL units from the RTP packets before feeding them to avcodec (including reassembling fragmentation units).

  I'm trying to show effective decoding frame rate. I used to log the time after a successful decode video call where *got_picture_ptr is non-zero. So far this worked since I only ever got video where there was one slice per frame. But now I receive video where both I and P frames consist of 2 NAL units each, of types 5 and 1 respectively. Now when I feed the either slice of a frame, decode_video return that it got a picture, and the pAVFrame->coded_picture_number is increased from every slice.

  How do I go about reliably finding the beginning or end of a video frame/picture/access unit ?

  I've dumped out a few NAL units from the stream and run them through h264_analyze from h264bitstream.

  Output from h264_analyze on 4 NAL Units

   !! Found NAL at offset 695262 (0xA9BDE), size 25 (0x0019) 
  ==================== NAL ====================
   forbidden_zero_bit : 0 
   nal_ref_idc : 1 
   nal_unit_type : 7 ( Sequence parameter set ) 
  ======= SPS =======
   profile_idc : 66 
   constraint_set0_flag : 1 
   constraint_set1_flag : 1 
   constraint_set2_flag : 1 
   constraint_set3_flag : 0 
   reserved_zero_4bits : 0 
   level_idc : 32 
   seq_parameter_set_id : 0 
   chroma_format_idc : 0 
   residual_colour_transform_flag : 0 
   bit_depth_luma_minus8 : 0 
   bit_depth_chroma_minus8 : 0 
   qpprime_y_zero_transform_bypass_flag : 0 
   seq_scaling_matrix_present_flag : 0 
   log2_max_frame_num_minus4 : 12 
   pic_order_cnt_type : 2 
     log2_max_pic_order_cnt_lsb_minus4 : 0 
     delta_pic_order_always_zero_flag : 0 
     offset_for_non_ref_pic : 0 
     offset_for_top_to_bottom_field : 0 
     num_ref_frames_in_pic_order_cnt_cycle : 0 
   num_ref_frames : 1 
   gaps_in_frame_num_value_allowed_flag : 0 
   pic_width_in_mbs_minus1 : 79 
   pic_height_in_map_units_minus1 : 44 
   frame_mbs_only_flag : 1 
   mb_adaptive_frame_field_flag : 0 
   direct_8x8_inference_flag : 1 
   frame_cropping_flag : 0 
     frame_crop_left_offset : 0 
     frame_crop_right_offset : 0 
     frame_crop_top_offset : 0 
     frame_crop_bottom_offset : 0 
   vui_parameters_present_flag : 1 
  === VUI ===
   aspect_ratio_info_present_flag : 1 
     aspect_ratio_idc : 1 
       sar_width : 0 
       sar_height : 0 
   overscan_info_present_flag : 0 
     overscan_appropriate_flag : 0 
   video_signal_type_present_flag : 1 
     video_format : 5 
     video_full_range_flag : 1 
     colour_description_present_flag : 0 
       colour_primaries : 0 
     transfer_characteristics : 0 
     matrix_coefficients : 0 
   chroma_loc_info_present_flag : 0 
     chroma_sample_loc_type_top_field : 0 
     chroma_sample_loc_type_bottom_field : 0 
   timing_info_present_flag : 1 
     num_units_in_tick : 1 
     time_scale : 25 
     fixed_frame_rate_flag : 0 
   nal_hrd_parameters_present_flag : 0 
   vcl_hrd_parameters_present_flag : 0 
     low_delay_hrd_flag : 0 
   pic_struct_present_flag : 0 
   bitstream_restriction_flag : 1 
     motion_vectors_over_pic_boundaries_flag : 1 
     max_bytes_per_pic_denom : 0 
     max_bits_per_mb_denom : 0 
     log2_max_mv_length_horizontal : 6 
     log2_max_mv_length_vertical : 6 
     num_reorder_frames : 0 
     max_dec_frame_buffering : 1 
  === HRD ===
   cpb_cnt_minus1 : 0 
   bit_rate_scale : 0 
   cpb_size_scale : 0 
   initial_cpb_removal_delay_length_minus1 : 0 
   cpb_removal_delay_length_minus1 : 0 
   dpb_output_delay_length_minus1 : 0 
   time_offset_length : 0 
   !! Found NAL at offset 695290 (0xA9BFA), size 4 (0x0004) 
  ==================== NAL ====================
   forbidden_zero_bit : 0 
   nal_ref_idc : 1 
   nal_unit_type : 8 ( Picture parameter set ) 
  ======= PPS =======
   pic_parameter_set_id : 0 
   seq_parameter_set_id : 0 
   entropy_coding_mode_flag : 0 
   pic_order_present_flag : 0 
   num_slice_groups_minus1 : 0 
   slice_group_map_type : 0 
   num_ref_idx_l0_active_minus1 : 0 
   num_ref_idx_l1_active_minus1 : 0 
   weighted_pred_flag : 0 
   weighted_bipred_idc : 0 
   pic_init_qp_minus26 : 3 
   pic_init_qs_minus26 : 0 
   chroma_qp_index_offset : 0 
   deblocking_filter_control_present_flag : 1 
   constrained_intra_pred_flag : 0 
   redundant_pic_cnt_present_flag : 0 
   transform_8x8_mode_flag : 1 
   pic_scaling_matrix_present_flag : 0 
   second_chroma_qp_index_offset : 1 
   !! Found NAL at offset 695297 (0xA9C01), size 50725 (0xC625) 
  ==================== NAL ====================
   forbidden_zero_bit : 0 
   nal_ref_idc : 1 
   nal_unit_type : 5 ( Coded slice of an IDR picture ) 
  ======= Slice Header =======
   first_mb_in_slice : 0 
   slice_type : 2 ( I slice ) 
   pic_parameter_set_id : 0 
   frame_num : 0 
   field_pic_flag : 0 
   bottom_field_flag : 0 
   idr_pic_id : 0 
   pic_order_cnt_lsb : 0 
   delta_pic_order_cnt_bottom : 0 
   redundant_pic_cnt : 0 
   direct_spatial_mv_pred_flag : 0 
   num_ref_idx_active_override_flag : 0 
   num_ref_idx_l0_active_minus1 : 0 
   num_ref_idx_l1_active_minus1 : 0 
   cabac_init_idc : 0 
   slice_qp_delta : 5 
   sp_for_switch_flag : 0 
   slice_qs_delta : 0 
   disable_deblocking_filter_idc : 0 
   slice_alpha_c0_offset_div2 : 0 
   slice_beta_offset_div2 : 0 
   slice_group_change_cycle : 0 
  === Prediction Weight Table ===
   luma_log2_weight_denom : 0 
   chroma_log2_weight_denom : 0 
   luma_weight_l0_flag : 0 
   chroma_weight_l0_flag : 0 
   luma_weight_l1_flag : 0 
   chroma_weight_l1_flag : 0 
  === Ref Pic List Reordering ===
   ref_pic_list_reordering_flag_l0 : 0 
   ref_pic_list_reordering_flag_l1 : 0 
  === Decoded Ref Pic Marking ===
   no_output_of_prior_pics_flag : 0 
   long_term_reference_flag : 0 
   adaptive_ref_pic_marking_mode_flag : 0 
   !! Found NAL at offset 746025 (0xB6229), size 38612 (0x96D4) 
  ==================== NAL ====================
   forbidden_zero_bit : 0 
   nal_ref_idc : 1 
   nal_unit_type : 5 ( Coded slice of an IDR picture ) 
  ======= Slice Header =======
   first_mb_in_slice : 1840 
   slice_type : 2 ( I slice ) 
   pic_parameter_set_id : 0 
   frame_num : 0 
   field_pic_flag : 0 
   bottom_field_flag : 0 
   idr_pic_id : 0 
   pic_order_cnt_lsb : 0 
   delta_pic_order_cnt_bottom : 0 
   redundant_pic_cnt : 0 
   direct_spatial_mv_pred_flag : 0 
   num_ref_idx_active_override_flag : 0 
   num_ref_idx_l0_active_minus1 : 0 
   num_ref_idx_l1_active_minus1 : 0 
   cabac_init_idc : 0 
   slice_qp_delta : 5 
   sp_for_switch_flag : 0 
   slice_qs_delta : 0 
   disable_deblocking_filter_idc : 0 
   slice_alpha_c0_offset_div2 : 0 
   slice_beta_offset_div2 : 0 
   slice_group_change_cycle : 0 
  === Prediction Weight Table ===
   luma_log2_weight_denom : 0 
   chroma_log2_weight_denom : 0 
   luma_weight_l0_flag : 0 
   chroma_weight_l0_flag : 0 
   luma_weight_l1_flag : 0 
   chroma_weight_l1_flag : 0 
  === Ref Pic List Reordering ===
   ref_pic_list_reordering_flag_l0 : 0 
   ref_pic_list_reordering_flag_l1 : 0 
  === Decoded Ref Pic Marking ===
   no_output_of_prior_pics_flag : 0 
   long_term_reference_flag : 0 
   adaptive_ref_pic_marking_mode_flag : 0 

  Both I slices show the frame_num = 0. The next 2 (not shown) have frame_num = 1.

• Building FFmpeg for Android to use command line arguments

  28 juin 2013, par Zargoon

  I am trying to build the FFmpeg library to use in my android app with the NDK. The reason for this is because I am using the native video capture feature in android because I really don't want to write my own video recorder. However, the native video capture only allows for either high-quality encoding, or low quality encoding. I want something in between, and I believe that the solution is to use the FFmpeg library to re-encode the high quality video to be lighter.

  So far I have been able to build the FFmpeg library according to this guide : http://www.roman10.net/how-to-build-ffmpeg-for-android/ and which a few tweaks I have been able to get it to work.

  However, everything that I've found seems to be about writing your own encoder, which seems like overkill to me. All that I really want to do is send a string in command line format to the main() function of FFmpeg and re-encode my video. However, I can't seem to figure out how I build FFmpeg to give me access to the main method. I found this post : Compile ffmpeg.c and call its main() via JNI which links to a project doing what I want more of less, but for the life of me I cannot figure out what is going on. It also seems like he is compiling more than I want, and I would really like to keep my application as light weight as possible.

  Some additional direction would be extremely helpful. Thank you.

• Working on images asynchronously

  15 décembre 2013, par Mikko Koppanen — Imagick, PHP stuff

  To get my quota on buzzwords for the day we are going to look at using ZeroMQ and Imagick to create a simple asynchronous image processing system. Why asynchronous ? First of all, separating the image handling from a interactive PHP scripts allows us to scale the image processing separately from the web heads. For example we could do the image processing on separate servers, which have SSDs attached and more memory. In this example making the images available to all worker nodes is left to the reader.

  Secondly, separating the image processing from a web script can provide more responsive experience to the user. This doesn’t necessarily mean faster, but let’s say in a multiple image upload scenario this method allows the user to do something else on the site while we process the images in the background. This can be beneficial especially in cases where users upload hundreds of images at a time. To achieve a simple distributed image processing infrastructure we are going to use ZeroMQ for communicating between different components and Imagick to work on the images.

  The first part we are going to create is a simple “Worker” -process skeleton. Naturally for a live environment you would like to have more error handling and possibly use pcntl for process control, but for the sake of brewity the example is barebones :

  php

  < view plain text >

  1. < ?php
  2.  
  3. define (’THUMBNAIL_ADDR’, ’tcp ://127.0.0.1:5000’) ;
  4. define (’COLLECTOR_ADDR’, ’tcp ://127.0.0.1:5001’) ;
  5.  
  6. class Worker {
  7.  
  8.   private $in ;
  9.   private $out ;
  10.  
  11.   public function __construct ($in_addr, $out_addr)
  12.   {
  13.     $context = new ZMQContext () ;
  14.  
  15.     $this->in = new ZMQSocket ($context, ZMQ: :SOCKET_PULL) ;
  16.     $this->in->bind ($in_addr) ;
  17.  
  18.     $this->out = new ZMQSocket ($context, ZMQ: :SOCKET_PUSH) ;
  19.     $this->out->connect ($out_addr) ;
  20.   }
  21.  
  22.   public function work () {
  23.     while ($command = $this->in->recvMulti ()) {
  24.       if (isset ($this->commands [$command [0]])) {
  25.         echo "Received work" . PHP_EOL ;
  26.  
  27.         $callback = $this->commands [$command [0]] ;
  28.  
  29.         array_shift ($command) ;
  30.         $response = call_user_func_array ($callback, $command) ;
  31.  
  32.         if (is_array ($response))
  33.           $this->out->sendMulti ($response) ;
  34.         else
  35.           $this->out->send ($response) ;
  36.       }
  37.       else {
  38.         error_log ("There is no registered worker for $command [0]") ;
  39.       }
  40.     }
  41.   }
  42.  
  43.   public function register ($command, $callback)
  44.   {
  45.     $this->commands [$command] = $callback ;
  46.   }
  47. }
  48.  ?>

  The Worker class allows us to register commands with callbacks associated with them. In our case the Worker class doesn’t actually care or know about the parameters being passed to the actual callback, it just blindly passes them on. We are using two separate sockets in this example, one for incoming work requests and one for passing the results onwards. This allows us to create a simple pipeline by adding more workers in the mix. For example we could first have a watermark worker, which takes the original image and composites a watermark on it, passes the file onwards to thumbnail worker, which then creates different sizes of thumbnails and passes the final results to event collector.

  The next part we are going to create a is a simple worker script that does the actual thumbnailing of the images :

  php

  < view plain text >

  1. < ?php
  2. include __DIR__ . ’/common.php’ ;
  3.  
  4. // Create worker class and bind the inbound address to ’THUMBNAIL_ADDR’ and connect outbound to ’COLLECTOR_ADDR’
  5. $worker = new Worker (THUMBNAIL_ADDR, COLLECTOR_ADDR) ;
  6.  
  7. // Register our thumbnail callback, nothing special here
  8. $worker->register (’thumbnail’, function ($filename, $width, $height) {
  9.                   $info = pathinfo ($filename) ;
  10.  
  11.                   $out = sprintf ("%s/%s_%dx%d.%s",
  12.                           $info [’dirname’],
  13.                           $info [’filename’],
  14.                           $width,
  15.                           $height,
  16.                           $info [’extension’]) ;
  17.  
  18.                   $status = 1 ;
  19.                   $message = ’’ ;
  20.  
  21.                   try {
  22.                     $im = new Imagick ($filename) ;
  23.                     $im->thumbnailImage ($width, $height) ;
  24.                     $im->writeImage ($out) ;
  25.                   }
  26.                   catch (Exception $e) {
  27.                     $status = 0 ;
  28.                     $message = $e->getMessage () ;
  29.                   }
  30.  
  31.                   return array (
  32.                         ’status’  => $status,
  33.                         ’filename’ => $filename,
  34.                         ’thumbnail’ => $out,
  35.                         ’message’ => $message,
  36.                     ) ;
  37.                 }) ;
  38.  
  39. // Run the worker, will block
  40. echo "Running thumbnail worker.." . PHP_EOL ;
  41. $worker->work () ;

  As you can see from the code the thumbnail worker registers a callback for ‘thumbnail’ command. The callback does the thumbnailing based on input and returns the status, original filename and the thumbnail filename. We have connected our Workers “outbound” socket to event collector, which will receive the results from the thumbnail worker and do something with them. What the “something” is depends on you. For example you could push the response into a websocket to show immediate feeedback to the user or store the results into a database.

  Our example event collector will just do a var_dump on every event it receives from the thumbnailer :

  php

  < view plain text >

  1. < ?php
  2. include __DIR__ . ’/common.php’ ;
  3.  
  4. $socket = new ZMQSocket (new ZMQContext (), ZMQ: :SOCKET_PULL) ;
  5. $socket->bind (COLLECTOR_ADDR) ;
  6.  
  7. echo "Waiting for events.." . PHP_EOL ;
  8. while (($message = $socket->recvMulti ())) {
  9.   var_dump ($message) ;
  10. }
  11.  ?>

  The final piece of the puzzle is the client that pumps messages into the pipeline. The client connects to the thumbnail worker, passes on filename and desired dimensions :

  php

  < view plain text >

  1. < ?php
  2. include __DIR__ . ’/common.php’ ;
  3.  
  4. $socket = new ZMQSocket (new ZMQContext (), ZMQ: :SOCKET_PUSH) ;
  5. $socket->connect (THUMBNAIL_ADDR) ;
  6.  
  7. $socket->sendMulti (
  8.       array (
  9.         ’thumbnail’,
  10.         realpath (’./test.jpg’),
  11.         50,
  12.         50,
  13.       )
  14. ) ;
  15. echo "Sent request" . PHP_EOL ;
  16.  ?>

  After this our processing pipeline will look like this :

  

  Now, if we notice that thumbnail workers or the event collectors can’t keep up with the rate of images we are pushing through we can start scaling the pipeline by adding more processes on each layer. ZeroMQ PUSH socket will automatically round-robin between all connected nodes, which makes adding more workers and event collectors simple. After adding more workers our pipeline will look like this :

  

  Using ZeroMQ also allows us to create more flexible architectures by adding forwarding devices in the middle, adding request-reply workers etc. So, the last thing to do is to run our pipeline and see the results :

  Let’s create our test image first :

  $ convert magick:rose test.jpg
  

  From the command-line run the thumbnail script :

  $ php thumbnail.php 
  Running thumbnail worker..
  

  In a separate terminal window run the event collector :

  $ php collector.php 
  Waiting for events..
  

  And finally run the client to send the thumbnail request :

  $ php client.php 
  Sent request
  $
  

  If everything went according to the plan you should now see the following output in the event collector window :

  array(4) 
    [0]=>
    string(1) "1"
    [1]=>
    string(56) "/test.jpg"
    [2]=>
    string(62) "/test_50x50.jpg"
    [3]=>
    string(0) ""
  
  

  Happy hacking !