Recherche avancée

Sur d’autres sites (14256)

• FFmpeg C audio video streams (mic, webcam) sync to mp4

  22 février 2020, par NIAZ

  I am trying to capture audio and video using a microphone and a webcam into an mp4 file. The recorded file is playable but over the time the audio starts drifting away from video and for a longer period of time the gap increases. Both the audio and video is handled in separate threads, for the audio I am using audiofifo adapted from the transcode_acc.c example https://github.com/FFmpeg/FFmpeg/blob/master/doc/examples/transcode_aac.c
  Here is how I am setting up the streams
  Video :

  video_output_codec_ctx = video_stream->codec;
  
  video_output_codec_ctx->bit_rate = 2000000;
  
  video_output_codec_ctx->codec_id = AV_CODEC_ID_MPEG4;
  
  video_output_codec_ctx->width = 640;
  
  video_output_codec_ctx->height = 480;
  
  video_stream->time_base = (AVRational){1, fps};
  
  video_output_codec_ctx->time_base = video_stream->time_base;
  
  video_output_codec_ctx->pix_fmt = AV_PIX_FMT_YUV420P;
  
  video_output_codec_ctx->codec_type = AVMEDIA_TYPE_VIDEO;

  Audio :

  audio_output_codec_ctx->channels       = OUTPUT_CHANNELS; // 2
  
  audio_output_codec_ctx->channel_layout = av_get_default_channel_layout(OUTPUT_CHANNELS);
  
  audio_output_codec_ctx->sample_rate    = audio_input_codec_ctx->sample_rate;
  
  audio_output_codec_ctx->sample_fmt     = audio_output_codec->sample_fmts[0];
  
  audio_output_codec_ctx->bit_rate       = OUTPUT_BIT_RATE; // 96000
  
  audio_output_codec_ctx->strict_std_compliance = FF_COMPLIANCE_EXPERIMENTAL;
  
  
  
  /* Set the sample rate for the container. */
  
  audio_stream->time_base.den = audio_input_codec_ctx->sample_rate;
  
  audio_stream->time_base.num = 1;

  For the video pts, I increment the index by 1 once the frame is encoded and before sending the frame to the encoder I use rescale and also after receiving the frame,afterwards the packets are written via av_interleaved_write_frame().

  output_frame->pts = av_rescale_q(video_frame_index, video_output_codec_ctx->time_base, video_input_format_ctx->streams[0]->time_base);
  
  
  
  error = avcodec_send_frame(video_output_codec_ctx, output_frame);
  
  
  
  error = avcodec_receive_packet(video_output_codec_ctx, &output_packet);
  
  
  
  output_packet.stream_index = video_index;
  
  
  
  output_packet.pts = av_rescale_q_rnd(output_packet.pts, video_input_format_ctx->streams[0]->time_base, output_format_context->streams[video_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.dts = av_rescale_q_rnd(output_packet.dts, video_input_format_ctx->streams[0]->time_base, output_format_context->streams[video_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.duration = ((output_format_context->streams[0]->time_base.den / output_format_context->streams[0]->time_base.num) / video_output_codec_ctx->time_base.den);
  
  
  
  output_packet.pos = -1;  
  
  
  
  video_frame_index++;

  For the audio pts, I increment by frame->nb_samples once the frame is encoded I use rescale, afterwards the packets are written via av_interleaved_write_frame().

  frame->pts = aud_pts;
  
  aud_pts += frame->nb_samples;
  
  
  
  error = avcodec_send_frame(audio_output_codec_ctx, frame);
  
  
  
  error = avcodec_receive_packet(audio_output_codec_ctx, &output_packet);
  
  
  
  output_packet.stream_index = audio_index;
  
  
  
  output_packet.pts = av_rescale_q_rnd(output_packet.pts, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.dts = av_rescale_q_rnd(output_packet.dts, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.duration = av_rescale_q(output_packet.duration, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base);

  I am new to the FFmpeg C API, have tried various resources / posts over the internet but still haven’t been able to sync the audio and video in a robust way. Here are few question that I want to understand which will help me to resolve this. Any thoughts are really appreciated on this.

  1. Can the FFmpeg C API handle sync internally or this is something which needs to be handled from the caller side ?

  2. Am I setting up the PTS correctly both for audio and video in the first place ? I have noticed that when I use fps lower than 20, I get Invalid pts (66667) <= last (66667) Operation not permitted from the encoder. This must be something wrong with the way I am currently setting the video PTS. How can I set up the video PTS to handle lower fps ?

  3. I am also trying to adopt the idea of clocks sync from dranger’s tutorial, not sure whether this would be suitable for my use case, things like where to setup the audio and video clocks as he used only decoders, for the audio I am using fifo and not sure how to adjust the samples based on the clocks sync, also the way to call and setup the refresh timer ?

  4. Is there a better mechanism for creating a robust sync for my use case which can handle both audio and video if they go out of sync, would be great to have an idea about the samples and frames adjustment based on that ?

• FFmpeg C/C++ audio video streams (mic, webcam) sync to mp4

  21 février 2020, par NIAZ

  I am trying to capture audio and video using a microphone and a webcam into an mp4 file. The recorded file is playable but over the time the audio starts drifting away from video and for a longer period of time the gap increases. Both the audio and video is handled in separate threads, for the audio I am using audiofifo adapted from the transcode_acc.c example https://github.com/FFmpeg/FFmpeg/blob/master/doc/examples/transcode_aac.c
  Here is how I am setting up the streams
  Video :

  video_output_codec_ctx = video_stream->codec;
  
  video_output_codec_ctx->bit_rate = 2000000;
  
  video_output_codec_ctx->codec_id = AV_CODEC_ID_MPEG4;
  
  video_output_codec_ctx->width = 640;
  
  video_output_codec_ctx->height = 480;
  
  video_stream->time_base = (AVRational){1, fps};
  
  video_output_codec_ctx->time_base = video_stream->time_base;
  
  video_output_codec_ctx->pix_fmt = AV_PIX_FMT_YUV420P;
  
  video_output_codec_ctx->codec_type = AVMEDIA_TYPE_VIDEO;

  Audio :

  audio_output_codec_ctx->channels       = OUTPUT_CHANNELS; // 2
  
  audio_output_codec_ctx->channel_layout = av_get_default_channel_layout(OUTPUT_CHANNELS);
  
  audio_output_codec_ctx->sample_rate    = audio_input_codec_ctx->sample_rate;
  
  audio_output_codec_ctx->sample_fmt     = audio_output_codec->sample_fmts[0];
  
  audio_output_codec_ctx->bit_rate       = OUTPUT_BIT_RATE; // 96000
  
  audio_output_codec_ctx->strict_std_compliance = FF_COMPLIANCE_EXPERIMENTAL;
  
  
  
  /* Set the sample rate for the container. */
  
  audio_stream->time_base.den = audio_input_codec_ctx->sample_rate;
  
  audio_stream->time_base.num = 1;

  For the video pts, I increment the index by 1 once the frame is encoded and before sending the frame to the encoder I use rescale and also after receiving the frame,afterwards the packets are written via av_interleaved_write_frame().

  output_frame->pts = av_rescale_q(video_frame_index, video_output_codec_ctx->time_base, video_input_format_ctx->streams[0]->time_base);
  
  
  
  error = avcodec_send_frame(video_output_codec_ctx, output_frame);
  
  
  
  error = avcodec_receive_packet(video_output_codec_ctx, &output_packet);
  
  
  
  output_packet.stream_index = video_index;
  
  
  
  output_packet.pts = av_rescale_q_rnd(output_packet.pts, video_input_format_ctx->streams[0]->time_base, output_format_context->streams[video_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.dts = av_rescale_q_rnd(output_packet.dts, video_input_format_ctx->streams[0]->time_base, output_format_context->streams[video_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.duration = ((output_format_context->streams[0]->time_base.den / output_format_context->streams[0]->time_base.num) / video_output_codec_ctx->time_base.den);
  
  
  
  output_packet.pos = -1;  
  
  
  
  video_frame_index++;

  For the audio pts, I increment by frame->nb_samples once the frame is encoded I use rescale, afterwards the packets are written via av_interleaved_write_frame().

  frame->pts = aud_pts;
  
  aud_pts += frame->nb_samples;
  
  
  
  error = avcodec_send_frame(audio_output_codec_ctx, frame);
  
  
  
  error = avcodec_receive_packet(audio_output_codec_ctx, &output_packet);
  
  
  
  output_packet.stream_index = audio_index;
  
  
  
  output_packet.pts = av_rescale_q_rnd(output_packet.pts, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.dts = av_rescale_q_rnd(output_packet.dts, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.duration = av_rescale_q(output_packet.duration, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base);

  I am new to the FFmpeg C API, have tried various resources / posts over the internet but still haven’t been able to sync the audio and video in a robust way. Here are few question that I want to understand which will help me to resolve this. Any thoughts are really appreciated on this.

  1. Can the FFmpeg C API handle sync internally or this is something which needs to be handled from the caller side ?

  2. Am I setting up the PTS correctly both for audio and video in the first place ? I have noticed that when I use fps lower than 20, I get Invalid pts (66667) <= last (66667) Operation not permitted from the encoder. This must be something wrong with the way I am currently setting the video PTS. How can I set up the video PTS to handle lower fps ?

  3. I am also trying to adopt the idea of clocks sync from dranger’s tutorial, not sure whether this would be suitable for my use case, things like where to setup the audio and video clocks as he used only decoders, for the audio I am using fifo and not sure how to adjust the samples based on the clocks sync, also the way to call and setup the refresh timer ?

  4. Is there a better mechanism for creating a robust sync for my use case which can handle both audio and video if they go out of sync, would be great to have an idea about the samples and frames adjustment based on that ?

• ffmpeg to Youtube Live stops working. ffmpeg continues to run

  10 février 2020, par Robert Pringle

  I’m streaming a mobotix camera to Youtube Live.

  This works correctly for a period of time. Youtube then reports the stream as "noData".
  ffmpeg continues to run, but no video is being received by Youtube.

  The problem is intermittant.

  Restarting the ffmpeg process makes Youtube start receiving the stream again.

  ffmpeg is being run as a systemd service.

  ffmpeg is run with the following command :

  ffmpeg -nostdin  -f lavfi  -i anullsrc  -rtsp_transport tcp -thread_queue_size 512 -i "rtsp://@192.168.1.1:554/mobotix.h264"   -tune zerolatency  -vcodec libx264  -pix_fmt + -c:v copy  -c:a aac  -strict experimental  -f flv rtmp://a.rtmp.youtube.com/live2/

  Logs only show : ffmpeg[47357]: [48.0K blob data]

  Any ideas on how to solve this ? Or improve the logging.

  Edit :
  Logs from starting ffmpeg :

      Feb 07 17:57:00 localhost.localdomain ffmpeg[49525]: [48.0K blob data]
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:       encoder         : Lavc58.54.100 aac
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:     Metadata:
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:     Stream #0:1: Audio: aac (LC) ([10][0][0][0] / 0x000A), 44100 Hz, stereo, fltp, 128 kb/s
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:     Stream #0:0: Video: h264 (Constrained Baseline) ([7][0][0][0] / 0x0007), yuv420p(progressive), 1024x768, q=2-31, 12.50 tbr, 1k tbn, 90k tbc
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:     encoder         : Lavf58.29.100
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:   Metadata:
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]: Output #0, flv, to 'rtmp://a.rtmp.youtube.com/live2/':
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:   Stream #0:0 -> #0:1 (pcm_u8 (native) -> aac (native))
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]:   Stream #1:0 -> #0:0 (copy)
  
      Feb 07 17:52:47 localhost.localdomain ffmpeg[49525]: Stream mapping:
  
      Feb 07 17:52:46 localhost.localdomain ffmpeg[49525]:     Stream #1:0: Video: h264 (Constrained Baseline), yuv420p(progressive), 1024x768, 12.50 tbr, 90k tbn, 180k tbc
  
      Feb 07 17:52:46 localhost.localdomain ffmpeg[49525]:   Duration: N/A, start: 136.139322, bitrate: N/A
  
      Feb 07 17:52:46 localhost.localdomain ffmpeg[49525]:     comment         : mobotix.h264
  
      Feb 07 17:52:46 localhost.localdomain ffmpeg[49525]:     title           : Mobotix IP-Camera (H.264, unicast)
  
      Feb 07 17:52:46 localhost.localdomain ffmpeg[49525]:   Metadata:
  
      Feb 07 17:52:46 localhost.localdomain ffmpeg[49525]: Input #1, rtsp, from 'rtsp://@192.168.1.1:554/mobotix.h264':
  
      Feb 07 17:52:44 localhost.localdomain ffmpeg[49525]: [h264 @ 0x559ee7857940] concealing 2400 DC, 2400 AC, 2400 MV errors in P frame
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:     Stream #0:0: Audio: pcm_u8, 44100 Hz, stereo, u8, 705 kb/s
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   Duration: N/A, start: 0.000000, bitrate: 705 kb/s
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]: Input #0, lavfi, from 'anullsrc':
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libpostproc    55.  5.100 / 55.  5.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libswresample   3.  5.100 /  3.  5.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libswscale      5.  5.100 /  5.  5.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libavresample   4.  0.  0 /  4.  0.  0
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libavfilter     7. 57.100 /  7. 57.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libavdevice    58.  8.100 / 58.  8.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libavformat    58. 29.100 / 58. 29.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libavcodec     58. 54.100 / 58. 54.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   libavutil      56. 31.100 / 56. 31.100
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 >
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]:   built with gcc 8 (GCC)
  
      Feb 07 17:52:43 localhost.localdomain ffmpeg[49525]: ffmpeg version 4.2.1 Copyright (c) 2000-2019 the FFmpeg developers

  Running the command manually gave me logs and exited, whereas the systemd service keeps running.

      [flv @ 0x560541a8b3c0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
  
      av_interleaved_write_frame(): Broken pipekB time=03:35:18.84 bitrate=3131.8kbits/s speed=0.999x    
  
          Last message repeated 1 times
  
      [flv @ 0x560541a8b3c0] Failed to update header with correct duration.
  
      [flv @ 0x560541a8b3c0] Failed to update header with correct filesize.
  
      Error writing trailer of rtmp://a.rtmp.youtube.com/live2/: Broken pipe
  
      frame=134427 fps= 10 q=-1.0 Lsize= 4938869kB time=03:35:18.96 bitrate=3131.8kbits/s speed=0.997x    
  
      video:4923745kB audio:3296kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.240068%
  
      [aac @ 0x560541a6d680] Qavg: 65536.000
  
      Conversion failed!