Recherche avancée

Sur d’autres sites (1419)

• FFmpeg C audio video streams (mic, webcam) sync to mp4

  22 février 2020, par NIAZ

  I am trying to capture audio and video using a microphone and a webcam into an mp4 file. The recorded file is playable but over the time the audio starts drifting away from video and for a longer period of time the gap increases. Both the audio and video is handled in separate threads, for the audio I am using audiofifo adapted from the transcode_acc.c example https://github.com/FFmpeg/FFmpeg/blob/master/doc/examples/transcode_aac.c
  Here is how I am setting up the streams
  Video :

  video_output_codec_ctx = video_stream->codec;
  
  video_output_codec_ctx->bit_rate = 2000000;
  
  video_output_codec_ctx->codec_id = AV_CODEC_ID_MPEG4;
  
  video_output_codec_ctx->width = 640;
  
  video_output_codec_ctx->height = 480;
  
  video_stream->time_base = (AVRational){1, fps};
  
  video_output_codec_ctx->time_base = video_stream->time_base;
  
  video_output_codec_ctx->pix_fmt = AV_PIX_FMT_YUV420P;
  
  video_output_codec_ctx->codec_type = AVMEDIA_TYPE_VIDEO;

  Audio :

  audio_output_codec_ctx->channels       = OUTPUT_CHANNELS; // 2
  
  audio_output_codec_ctx->channel_layout = av_get_default_channel_layout(OUTPUT_CHANNELS);
  
  audio_output_codec_ctx->sample_rate    = audio_input_codec_ctx->sample_rate;
  
  audio_output_codec_ctx->sample_fmt     = audio_output_codec->sample_fmts[0];
  
  audio_output_codec_ctx->bit_rate       = OUTPUT_BIT_RATE; // 96000
  
  audio_output_codec_ctx->strict_std_compliance = FF_COMPLIANCE_EXPERIMENTAL;
  
  
  
  /* Set the sample rate for the container. */
  
  audio_stream->time_base.den = audio_input_codec_ctx->sample_rate;
  
  audio_stream->time_base.num = 1;

  For the video pts, I increment the index by 1 once the frame is encoded and before sending the frame to the encoder I use rescale and also after receiving the frame,afterwards the packets are written via av_interleaved_write_frame().

  output_frame->pts = av_rescale_q(video_frame_index, video_output_codec_ctx->time_base, video_input_format_ctx->streams[0]->time_base);
  
  
  
  error = avcodec_send_frame(video_output_codec_ctx, output_frame);
  
  
  
  error = avcodec_receive_packet(video_output_codec_ctx, &output_packet);
  
  
  
  output_packet.stream_index = video_index;
  
  
  
  output_packet.pts = av_rescale_q_rnd(output_packet.pts, video_input_format_ctx->streams[0]->time_base, output_format_context->streams[video_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.dts = av_rescale_q_rnd(output_packet.dts, video_input_format_ctx->streams[0]->time_base, output_format_context->streams[video_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.duration = ((output_format_context->streams[0]->time_base.den / output_format_context->streams[0]->time_base.num) / video_output_codec_ctx->time_base.den);
  
  
  
  output_packet.pos = -1;  
  
  
  
  video_frame_index++;

  For the audio pts, I increment by frame->nb_samples once the frame is encoded I use rescale, afterwards the packets are written via av_interleaved_write_frame().

  frame->pts = aud_pts;
  
  aud_pts += frame->nb_samples;
  
  
  
  error = avcodec_send_frame(audio_output_codec_ctx, frame);
  
  
  
  error = avcodec_receive_packet(audio_output_codec_ctx, &output_packet);
  
  
  
  output_packet.stream_index = audio_index;
  
  
  
  output_packet.pts = av_rescale_q_rnd(output_packet.pts, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.dts = av_rescale_q_rnd(output_packet.dts, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.duration = av_rescale_q(output_packet.duration, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base);

  I am new to the FFmpeg C API, have tried various resources / posts over the internet but still haven’t been able to sync the audio and video in a robust way. Here are few question that I want to understand which will help me to resolve this. Any thoughts are really appreciated on this.

  1. Can the FFmpeg C API handle sync internally or this is something which needs to be handled from the caller side ?

  2. Am I setting up the PTS correctly both for audio and video in the first place ? I have noticed that when I use fps lower than 20, I get Invalid pts (66667) <= last (66667) Operation not permitted from the encoder. This must be something wrong with the way I am currently setting the video PTS. How can I set up the video PTS to handle lower fps ?

  3. I am also trying to adopt the idea of clocks sync from dranger’s tutorial, not sure whether this would be suitable for my use case, things like where to setup the audio and video clocks as he used only decoders, for the audio I am using fifo and not sure how to adjust the samples based on the clocks sync, also the way to call and setup the refresh timer ?

  4. Is there a better mechanism for creating a robust sync for my use case which can handle both audio and video if they go out of sync, would be great to have an idea about the samples and frames adjustment based on that ?

• Proper use of `nalu_process` callback in x264

  26 janvier 2021, par gspr

  I wish to make use of libx264's low-latency encoding mechanism, whereby a user-provided callback is called as soon as a single NAL unit is available instead of having to wait for a whole frame to be encoded before starting processing.

  &#xA;

  The x264 documentation states the following about that facility :

  &#xA;

  /* Optional low-level callback for low-latency encoding.  Called for each output NAL unit
 * immediately after the NAL unit is finished encoding.  This allows the calling application
 * to begin processing video data (e.g. by sending packets over a network) before the frame
 * is done encoding.
 *
 * This callback MUST do the following in order to work correctly:
 * 1) Have available an output buffer of at least size nal->i_payload*3/2 + 5 + 64.
 * 2) Call x264_nal_encode( h, dst, nal ), where dst is the output buffer.
 * After these steps, the content of nal is valid and can be used in the same way as if
 * the NAL unit were output by x264_encoder_encode.
 *
 * This does not need to be synchronous with the encoding process: the data pointed to
 * by nal (both before and after x264_nal_encode) will remain valid until the next
 * x264_encoder_encode call.  The callback must be re-entrant.
 *
 * This callback does not work with frame-based threads; threads must be disabled
 * or sliced-threads enabled.  This callback also does not work as one would expect
 * with HRD -- since the buffering period SEI cannot be calculated until the frame
 * is finished encoding, it will not be sent via this callback.
 *
 * Note also that the NALs are not necessarily returned in order when sliced threads is
 * enabled.  Accordingly, the variable i_first_mb and i_last_mb are available in
 * x264_nal_t to help the calling application reorder the slices if necessary.
 *
 * When this callback is enabled, x264_encoder_encode does not return valid NALs;
 * the calling application is expected to acquire all output NALs through the callback.
 *
 * It is generally sensible to combine this callback with a use of slice-max-mbs or
 * slice-max-size.
 *
 * The opaque pointer is the opaque pointer from the input frame associated with this
 * NAL unit. This helps distinguish between nalu_process calls from different sources,
 * e.g. if doing multiple encodes in one process.
 */
void (*nalu_process)( x264_t *h, x264_nal_t *nal, void *opaque );


  &#xA;

  This seems straight forward enough. However, when I run the following dummy code, I get a segfault on the marked line. I've tried to add some debugging to x264_nal_encode itself to understand where it goes wrong, but it seems to be the function call itself that results in a segfault. Am I missing something here ? (Let's ignore the fact that the use of assert probably makes cb non-reentrant – it's only there to indicate to the reader that my workspace buffer is more than large enough.)

  &#xA;

  #include 
#include 
#include 
#include 
#include 
#include 

#define WS_SIZE 10000000
uint8_t * workspace;

void cb(x264_t * h, x264_nal_t * nal, void * opaque)
{
  assert((nal->i_payload*3)/2 + 5 + 64 < WS_SIZE);
  x264_nal_encode(h, workspace, nal); // Segfault here.
  // Removed: Process nal.
}

int main(int argc, char ** argv)
{
  uint8_t * fake_frame = malloc(1280*720*3);
  memset(fake_frame, 0, 1280*720*3);

  workspace = malloc(WS_SIZE);

  x264_param_t param;
  int status = x264_param_default_preset(&param, "ultrafast", "zerolatency");
  assert(status == 0);

  param.i_csp = X264_CSP_RGB;
  param.i_width = 1280;
  param.i_height = 720;
  param.i_threads = 1;
  param.i_lookahead_threads = 1;
  param.i_frame_total = 0;
  param.i_fps_num = 30;
  param.i_fps_den = 1;
  param.i_slice_max_size = 1024;
  param.b_annexb = 1;
  param.nalu_process = cb;

  status = x264_param_apply_profile(&param, "high444");
  assert(status == 0);

  x264_t * h = x264_encoder_open(&param);
  assert(h);

  x264_picture_t pic;
  status = x264_picture_alloc(&pic, param.i_csp, param.i_width, param.i_height);
  assert(pic.img.i_plane == 1);

  x264_picture_t pic_out;
  x264_nal_t * nal; // Not used. We process NALs in cb.
  int i_nal;

  for (int i = 0; i < 100; ++i)
  {
    pic.i_pts = i;
    pic.img.plane[0] = fake_frame;
    status = x264_encoder_encode(h, &nal, &i_nal, &pic, &pic_out);
  }

  x264_encoder_close(h);
  x264_picture_clean(&pic);
  free(workspace);
  free(fake_frame);
  return 0;
}


  &#xA;

  Edit : The segfault happens the first time cb calls x264_nal_encode. If I switch to a different preset, where more frames are encoded before the first callback happens, then several successful calls to x264_encoder_encode are made before the first callback, and hence segfault, occurs.

  &#xA;

• ffmpeg to Youtube Live stops working. ffmpeg continues to run

  10 février 2020, par Robert Pringle

  I’m streaming a mobotix camera to Youtube Live.

  This works correctly for a period of time. Youtube then reports the stream as "noData".
  ffmpeg continues to run, but no video is being received by Youtube.

  The problem is intermittant.

  Restarting the ffmpeg process makes Youtube start receiving the stream again.

  ffmpeg is being run as a systemd service.

  ffmpeg is run with the following command :

  ffmpeg -nostdin  -f lavfi  -i anullsrc  -rtsp_transport tcp -thread_queue_size 512 -i "rtsp://@192.168.1.1:554/mobotix.h264"   -tune zerolatency  -vcodec libx264  -pix_fmt + -c:v copy  -c:a aac  -strict experimental  -f flv rtmp://a.rtmp.youtube.com/live2/

  Logs only show : ffmpeg[47357]: [48.0K blob data]

  Any ideas on how to solve this ? Or improve the logging.

  Edit :
  Logs from starting ffmpeg :

      Feb 07 17:57:00 localhost.localdomain ffmpeg[49525]: [48.0K blob data]
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:       encoder         : Lavc58.54.100 aac
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:     Metadata:
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:     Stream #0:1: Audio: aac (LC) ([10][0][0][0] / 0x000A), 44100 Hz, stereo, fltp, 128 kb/s
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:     Stream #0:0: Video: h264 (Constrained Baseline) ([7][0][0][0] / 0x0007), yuv420p(progressive), 1024x768, q=2-31, 12.50 tbr, 1k tbn, 90k tbc
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:     encoder         : Lavf58.29.100
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:   Metadata:
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]: Output #0, flv, to 'rtmp://a.rtmp.youtube.com/live2/':
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:   Stream #0:0 -> #0:1 (pcm_u8 (native) -> aac (native))
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:   Stream #1:0 -> #0:0 (copy)
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]: Stream mapping:
  
      Feb 07 17:52:46 localhost.localdomain ffmpeg[49525]:     Stream #1:0: Video: h264 (Constrained Baseline), yuv420p(progressive), 1024x768, 12.50 tbr, 90k tbn, 180k tbc
  
      Feb 07 17:52:46 localhost.localdomain ffmpeg[49525]:   Duration: N/A, start: 136.139322, bitrate: N/A
  
      Feb 07 17:52:46 localhost.localdomain ffmpeg[49525]:     comment         : mobotix.h264
  
      Feb 07 17:52:46 localhost.localdomain ffmpeg[49525]:     title           : Mobotix IP-Camera (H.264, unicast)
  
      Feb 07 17:52:46 localhost.localdomain ffmpeg[49525]:   Metadata:
  
      Feb 07 17:52:46 localhost.localdomain ffmpeg[49525]: Input #1, rtsp, from 'rtsp://@192.168.1.1:554/mobotix.h264':
  
      Feb 07 17:52:44 localhost.localdomain ffmpeg[49525]: [h264 @ 0x559ee7857940] concealing 2400 DC, 2400 AC, 2400 MV errors in P frame
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:     Stream #0:0: Audio: pcm_u8, 44100 Hz, stereo, u8, 705 kb/s
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   Duration: N/A, start: 0.000000, bitrate: 705 kb/s
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]: Input #0, lavfi, from 'anullsrc':
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libpostproc    55.  5.100 / 55.  5.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libswresample   3.  5.100 /  3.  5.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libswscale      5.  5.100 /  5.  5.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libavresample   4.  0.  0 /  4.  0.  0
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libavfilter     7. 57.100 /  7. 57.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libavdevice    58.  8.100 / 58.  8.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libavformat    58. 29.100 / 58. 29.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libavcodec     58. 54.100 / 58. 54.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libavutil      56. 31.100 / 56. 31.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 >
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   built with gcc 8 (GCC)
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]: ffmpeg version 4.2.1 Copyright (c) 2000-2019 the FFmpeg developers

  Running the command manually gave me logs and exited, whereas the systemd service keeps running.

      [flv @ 0x560541a8b3c0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
  
      av_interleaved_write_frame(): Broken pipekB time=03:35:18.84 bitrate=3131.8kbits/s speed=0.999x    
  
          Last message repeated 1 times
  
      [flv @ 0x560541a8b3c0] Failed to update header with correct duration.
  
      [flv @ 0x560541a8b3c0] Failed to update header with correct filesize.
  
      Error writing trailer of rtmp://a.rtmp.youtube.com/live2/: Broken pipe
  
      frame=134427 fps= 10 q=-1.0 Lsize= 4938869kB time=03:35:18.96 bitrate=3131.8kbits/s speed=0.997x    
  
      video:4923745kB audio:3296kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.240068%
  
      [aac @ 0x560541a6d680] Qavg: 65536.000
  
      Conversion failed!