Sur d’autres sites (9933)

• Why android video player that based on ffmpeg can't sync video' time and audio's time ?

  6 juillet 2021, par ZSpirytus

  Background

  


  Recently, I use ffmpeg to write my first Android video player. But video channel's time is faster than audio channel's time about 2 3 times.

  


  Question

  


  Why android video player audio and video is out of sync ? Video is about faster than audio 2 3 times. Thanks for your reading and answers.

  


  Code

  


  In short, I use PacketDispatcher to read AVPacket from http hlv source :

  


  void PacketDispatcher::RealDispatch() {
    while (GetStatus() != DISPATCHER_STOP) {
        ...

        AVPacket *av_packet = av_packet_alloc();
        int ret = av_read_frame(av_format_context, av_packet);

        // PacketDispatcher is who read the AVPacket from http hlv source 
        // and dispatch to decoder by its stream index.
        decoder_map[av_packet->stream_index]->Push(av_packet);
    }
}


  


  And then, Decoder written by Producer-Consumer Pattern, Decoder maintain a queue that store all the AVPacket received from PacketDispatcher. The code like this :

  


  // write to the queue
void BaseDecoder::Push(AVPacket *av_packet) {
    ...
    av_packet_queue.push(av_packet);
    ...
}

// real decode logic
void BaseDecoder::RealDecode() {
    ...
    while (true) {
        // read frame from codec
        AVFrame *av_frame = av_frame_alloc();
        ret = avcodec_receive_frame(av_codec_ctx, av_frame);

        void *decode_result = DecodeFrame(av_frame);
        // send to render
        m_render->Render(decode_result);
    }
}


  


  And then, I do rendering logic in Render. Render also written by Producer-Consumer Pattern, it maintain a queue that store AVFrame received from Decoder, the code like this :

  


  // write AVFrame
void BaseRender::Render(void *frame_data) {
    ...
    frame_queue.push(frame_data);
    ...
}

// render to surface or Open SL
void BaseRender::RealRender() {
    if (m_render_synchronizer && m_render_synchronizer->Sync(frame_data)) {
        continue;
    }
}


  


  Finally, the synchronizer will decide to sleep time or drop video frame according to the frame pts, frame pts is :

  


  frame_data->pts = av_frame->best_effort_timestamp * av_q2d(GetTimeBase());


  


  Also, video extra delay is :

  


  frame_data->video_extra_delay = av_frame->repeat_pict * 1.0 / fps * 2.0;


  


  RenderSynchronizer code like this :

  


  bool RenderSynchronizer::Sync(void *frame_data) {&#xA;    auto base_frame_data = static_cast<baseframedata>(frame_data);&#xA;    if (base_frame_data->media_type == AVMEDIA_TYPE_AUDIO) {&#xA;        audio_pts = pcm_data->pts;&#xA;        return false;&#xA;    } else if (base_frame_data->media_type == AVMEDIA_TYPE_VIDEO) {&#xA;        video_pts = rgba_data->pts;&#xA;        return ReceiveVideoFrame(static_cast<rgbadata>(frame_data));&#xA;    }&#xA;    return false;&#xA;}&#xA;&#xA;bool RenderSynchronizer::ReceiveVideoFrame(RGBAData *rgba_data) {&#xA;    if (audio_pts &lt;= 0 || video_pts &lt;= 0) {&#xA;        return false;&#xA;    }&#xA;&#xA;    double diff = video_pts - audio_pts;&#xA;    if (diff > 0) {&#xA;        if (diff > 1) {&#xA;            av_usleep((unsigned int) (rgba_data->extra_delay * 1000000.0));&#xA;        } else {&#xA;            av_usleep((unsigned int) ((diff &#x2B; rgba_data->extra_delay) * 1000000.0));&#xA;        }&#xA;        return false;&#xA;    } else if (diff &lt; 0) {&#xA;        LOGD(TAG, "drop video frame");&#xA;        return true;&#xA;    } else {&#xA;        return false;&#xA;    }&#xA;}&#xA;</rgbadata></baseframedata>

  


• Revision 1b556e1f9a : vp9_spatial_svc_encoder : Enable aq-mode for real-time mode. For real-time mode

  27 août 2015, par Marco

  Changed Paths :
   Modify /examples/vp9_spatial_svc_encoder.c

  
  
  vp9_spatial_svc_encoder : Enable aq-mode for real-time mode.

  For real-time mode (speeds >=5) enable aq-mode=3.

  Change-Id : Ib8b4ef7609bc30ac935742c8d27e8cd89933c6af

• ffmpeg can't stream to remote client

  4 septembre 2014, par KFL

  I’m building a simple ffmpeg command line on my laptop to stream from its camera. The command line reads (verbose output at the botton) :

  host1> ffmpeg -v verbose \
  
                -f dshow \
  
                -i video="Camera":audio="Microphone" \
  
                -r 30 -g 0 -vcodec h264 -acodec libmp3lame \
  
                -tune zerolatency \
  
                -preset ultrafast \
  
                -f mpegts udp://12.34.56.78:12345

  Firstly, it works locally. I.e., I can view the output by using ffplay on the same host :

  host1> ffplay -hide_banner -v udp://12.34.56.78:12345

  Now what is NOT working is when I do this from another machine in the same network. It shows a nan progress :

  host2> ffplay -hide_banner -v udp://12.34.56.78:12345
  
      nan    :  0.000 fd=   0 aq=    0KB vq=    0KB sq=    0B f=0/0   

  I used ncat to dump the raw content. But there’s no output :

  host2>\ncat\ncat -v -u 12.34.56.78 12345
  
  Ncat: Version 5.59BETA1 ( http://nmap.org/ncat )
  
  Ncat: Connected to 12.34.56.78:12345.
  
  (...and nothing happen...)

  Note that I can exclude firewall issues as I used ncat to communicate with each other across the wire using the same port and protocol (UDP). This works and they can chat to each other :

  host1> ncat -l -u -p 12345
  
  host2> ncat -u 12.34.56.78 12345

  Any hint ?

  I’m using Windows x64 with FFMPEG 64bit installed from here. Below is the Output of my ffmpeg command :

  C:\ffmpeg\bin>ffmpeg -v verbose -f dshow -i video="Integrated Camera":audio="Microphone (Realtek High Definition Audio)" -r 30 -g 0 -vcodec h264 -acodec libmp3lame -tune zerolatency -preset ultrafast -f mpegts udp://12.34.56.78:12345
  
  ffmpeg version N-66012-g97b8809 Copyright (c) 2000-2014 the FFmpeg developers
  
    built on Sep  1 2014 00:21:15 with gcc 4.8.3 (GCC)
  
    configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug -enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-decklink --enable-zlib
  
    libavutil      54.  7.100 / 54.  7.100
  
    libavcodec     56.  1.100 / 56.  1.100
  
    libavformat    56.  3.100 / 56.  3.100
  
    libavdevice    56.  0.100 / 56.  0.100
  
    libavfilter     5.  0.103 /  5.  0.103
  
    libswscale      3.  0.100 /  3.  0.100
  
    libswresample   1.  1.100 /  1.  1.100
  
    libpostproc    53.  0.100 / 53.  0.100
  
  Guessed Channel Layout for  Input Stream #0.1 : stereo
  
  Input #0, dshow, from 'video=Integrated Camera:audio=Microphone (Realtek High Definition Audio)':
  
    Duration: N/A, start: 171840.657000, bitrate: N/A
  
      Stream #0:0: Video: rawvideo, bgr24, 640x480, 30 fps, 30 tbr, 10000k tbn, 30 tbc
  
      Stream #0:1: Audio: pcm_s16le, 44100 Hz, 2 channels, s16, 1411 kb/s
  
  Matched encoder 'libx264' for codec 'h264'.
  
  [graph 0 input from stream 0:0 @ 0000000000470aa0] w:640 h:480 pixfmt:bgr24 tb:1/10000000 fr:10000000/333333 sar:0/1 sws_param:flags=2
  
  [auto-inserted scaler 0 @ 0000000004326d00] w:iw h:ih flags:'0x4' interl:0
  
  [format @ 0000000004325a00] auto-inserting filter 'auto-inserted scaler 0' between the filter 'Parsed_null_0' and the filter 'format'
  
  [auto-inserted scaler 0 @ 0000000004326d00] w:640 h:480 fmt:bgr24 sar:0/1 -> w:640 h:480 fmt:yuv444p sar:0/1 flags:0x4
  
  No pixel format specified, yuv444p for H.264 encoding chosen.
  
  Use -pix_fmt yuv420p for compatibility with outdated media players.
  
  [graph 1 input from stream 0:1 @ 0000000000460c20] tb:1/44100 samplefmt:s16 samplerate:44100 chlayout:0x3
  
  [audio format for output stream 0:1 @ 00000000004601a0] auto-inserting filter 'auto-inserted resampler 0' between the filter 'Parsed_anull_0' and the filter 'audio format for output stream 0:1'
  
  [auto-inserted resampler 0 @ 00000000004604a0] ch:2 chl:stereo fmt:s16 r:44100Hz -> ch:2 chl:stereo fmt:s16p r:44100Hz
  
  [libx264 @ 000000000081bb20] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
  
  [libx264 @ 000000000081bb20] profile High 4:4:4 Intra, level 3.0, 4:4:4 8-bit
  
  [mpegts @ 000000000081abe0] muxrate VBR, pcr every 3 pkts, sdt every 200, pat/pmt every 40 pkts
  
  Output #0, mpegts, to 'udp://12.34.56.78:12345':
  
    Metadata:
  
      encoder         : Lavf56.3.100
  
      Stream #0:0: Video: h264 (libx264), yuv444p, 640x480, q=-1--1, 30 fps, 90k tbn, 30 tbc
  
      Metadata:
  
        encoder         : Lavc56.1.100 libx264
  
      Stream #0:1: Audio: mp3 (libmp3lame), 44100 Hz, stereo, s16p
  
      Metadata:
  
        encoder         : Lavc56.1.100 libmp3lame
  
  Stream mapping:
  
    Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
  
    Stream #0:1 -> #0:1 (pcm_s16le (native) -> mp3 (libmp3lame))
  
  Press [q] to stop, [?] for help
  
  *** 1 dup!
  
  frame=  241 fps= 31 q=28.0 Lsize=    3439kB time=00:00:08.03 bitrate=3506.4kbits/s dup=1 drop=0
  
  video:3035kB audio:125kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 8.791966%
  
  Input file #0 (video=Integrated Camera:audio=Microphone (Realtek High Definition Audio)):
  
    Input stream #0:0 (video): 240 packets read (221184000 bytes); 240 frames decoded;
  
    Input stream #0:1 (audio): 16 packets read (1411200 bytes); 16 frames decoded (352800 samples);
  
    Total: 256 packets (222595200 bytes) demuxed
  
  Output file #0 (udp://12.34.56.78:12345):
  
    Output stream #0:0 (video): 241 frames encoded; 241 packets muxed (3108187 bytes);
  
    Output stream #0:1 (audio): 306 frames encoded (352512 samples); 307 packets muxed (128313 bytes);
  
    Total: 548 packets (3236500 bytes) muxed
  
  [libx264 @ 000000000081bb20] frame I:241   Avg QP:27.97  size: 12897
  
  [libx264 @ 000000000081bb20] mb I  I16..4: 100.0%  0.0%  0.0%
  
  [libx264 @ 000000000081bb20] coded y,u,v intra: 26.3% 0.5% 0.0%
  
  [libx264 @ 000000000081bb20] i16 v,h,dc,p: 19% 28% 21% 31%
  
  [libx264 @ 000000000081bb20] kb/s:3095.29
  
  [dshow @ 0000000000467720] real-time buffer[Integrated Camera] too full (90% of size: 3041280)! frame dropped!
  
  Received signal 2: terminating. (I pressed CTRL-C)