Recherche avancée

Sur d’autres sites (1720)

• LibVLC DXVA2 Windows C++ Hardware Acceleration

  15 novembre 2016, par Sergiu Vlad

  I want to decode a h264 RTSP stream using DXVA2 Accelerated decoder. In VLC it does work but in my code have the following output :

  avcodec generic warning: threaded frame decoding is not compatible with DXVA2, disabled
  
  avcodec generic debug: available hardware decoder output format 61 (dxva2_vld)
  
  avcodec generic debug: available hardware decoder output format 128 (d3d11va_vld)
  
  avcodec generic debug: available software decoder output format 12 (yuvj420p)
  
  core generic debug: looking for hw decoder module matching "any": 1 candidates
  
  core generic debug: no hw decoder modules matched

  Is this a problem with NV12 conversion ? Is there a way to force DXVA2 ?

  Thanks

  CODE :

      #include 
  
  #include 
  
  #include 
  
  #include 
  
  #include 
  
  #include <iostream>
  
  
  
  #include 
  
  #include <ctime>
  
  #include <vlc></vlc>vlc.h>
  
  
  
  // RTSP address
  
  const char* rtspAddress = "rtsp://admin:admin@192.168.0.245/live_st1";
  
  // Video resolution WxH
  
  cv::Size rtspRes(1920, 1080);
  
  
  
  struct VideoDataStruct
  
  {
  
      int param;
  
  };
  
  
  
  int done = 0;
  
  libvlc_media_player_t *mp;
  
  unsigned int videoBufferSize = 0;
  
  uint8_t *videoBuffer = 0;
  
  
  
  void cbVideoPrerender(void *p_video_data, uint8_t **pp_pixel_buffer, int size)
  
  {
  
      if (size > videoBufferSize || !videoBuffer)
  
      {
  
          printf("Reallocate raw video buffer %d bytes\n", size);
  
          free(videoBuffer);
  
          videoBuffer = (uint8_t *)malloc(size);
  
          videoBufferSize = size;
  
      }
  
  
  
      // videoBuffer = (uint8_t *)malloc(size);
  
      *pp_pixel_buffer = videoBuffer;
  
  }
  
  void cbVideoPostrender(void *p_video_data, uint8_t *p_pixel_buffer, int width, int height, int pixel_pitch, int size, int64_t pts)
  
  {
  
      // Unlocking
  
      //CloseHandle(hMutex);
  
  
  
  }
  
  
  
  //static void handleEvent(const libvlc_event_t* pEvt, void* pUserData)
  
  //{
  
  //    libvlc_time_t time;
  
  //    switch (pEvt->type)
  
  //    {
  
  //    case libvlc_MediaPlayerTimeChanged:
  
  //        time = libvlc_media_player_get_time(mp);
  
  //        printf("MediaPlayerTimeChanged %lld ms\n", (long long)time);
  
  //        break;
  
  //    case libvlc_MediaPlayerEndReached:
  
  //        printf("MediaPlayerEndReached\n");
  
  //        done = 1;
  
  //        break;
  
  //    default:
  
  //        printf("%s\n", libvlc_event_type_name(pEvt->type));
  
  //    }
  
  //}
  
  
  
  clock_t startTime[8];
  
  int framesReceivedThisSecond[8];
  
  
  
  int main()
  
  {
  
  
  
  
  
      // VLC pointers
  
      libvlc_instance_t *inst;
  
      libvlc_media_t *m;
  
      void *pUserData = 0;
  
  
  
      VideoDataStruct dataStruct;
  
  
  
      // VLC options
  
      char smem_options[1000];
  
  
  
      // RV24
  
      sprintf(smem_options
  
              , "#transcode{vcodec=RV24}:smem{"
  
                "video-prerender-callback=%lld,"
  
                "video-postrender-callback=%lld,"
  
                "video-data=%lld,"
  
                "no-time-sync},"
  
              , (long long int)(intptr_t)(void*)&amp;cbVideoPrerender
  
              , (long long int)(intptr_t)(void*)&amp;cbVideoPostrender
  
              , (long long int)(intptr_t)(void*)&amp;dataStruct
  
              );
  
  
  
      const char * const vlc_args[] = {
  
          "-I", "dummy",            // Don't use any interface
  
          "--ignore-config",        // Don't use VLC's config
  
          "--extraintf=logger",     // Log anything
  
          "--verbose=4",            // Be verbose
  
          //        "--ffmpeg-hw",
  
          //        "-vvv",
  
          "--ffmpeg-hw",
  
          "--avcodec-hw=any",
  
          "--sout", smem_options    // Stream to memory
  
  
  
      };
  
  
  
      // Launch VLC
  
      inst = libvlc_new(sizeof(vlc_args) / sizeof(vlc_args[0]), vlc_args);
  
  
  
      // Create a new item
  
      m = libvlc_media_new_location(inst, rtspAddress);
  
  
  
  
  
  
  
      // Create a media player playing environement
  
      mp = libvlc_media_player_new_from_media(m);
  
  
  
      //    libvlc_event_manager_t* eventManager = libvlc_media_player_event_manager(mp);
  
      //    libvlc_event_attach(eventManager, libvlc_MediaPlayerTimeChanged, handleEvent, pUserData);
  
      //    libvlc_event_attach(eventManager, libvlc_MediaPlayerEndReached, handleEvent, pUserData);
  
      //    libvlc_event_attach(eventManager, libvlc_MediaPlayerPositionChanged, handleEvent, pUserData);
  
  
  
      //libvlc_video_set_format(mp, "RV24", 240, 320, 240 * 3 );
  
  
  
      // play the media_player
  
      libvlc_media_player_play(mp);
  
  
  
  
  
      // Create a window for displaying the video
  
      std::string winName("Demo Video");
  
      cv::namedWindow(winName, cv::WINDOW_AUTOSIZE);
  
  
  
      cv::Mat frame;
  
      int key = 0;
  
  
  
      // Endless loop, press Esc to quit
  
      while (key != 27)
  
      {
  
  
  
          // Check for invalid input
  
          if (videoBuffer)
  
          {
  
              // CV_8UC3 = 8 bits, 3 chanels
  
              frame = cv::Mat(rtspRes, CV_8UC3, videoBuffer);
  
          }
  
  
  
  
  
          if (frame.rows == 0 || frame.cols == 0)
  
              continue;
  
  
  
          int stream_number=0;
  
          framesReceivedThisSecond[stream_number]++;
  
          long    duration = ( std::clock() - startTime[stream_number] );
  
          if(duration>1000)
  
          {
  
              std::cout&lt;&lt;"\n"&lt;&lt;"FPS "&lt;code></ctime></iostream>

  FULL LOG :

  core libvlc debug: Copyright © 1996-2016 the VideoLAN team
  
  core libvlc debug: revision 2.2.3-37-g888b7e89
  
  core libvlc debug: configured with ../extras/package/win32/../../../configure  '--enable-update-check' '--enable-lua' '--enable-faad' '--enable-flac' '--enable-theora' '--enable-twolame' '--enable-quicktime' '--enable-avcodec' '--enable-merge-ffmpeg' '--enable-dca' '--enable-mpc' '--enable-libass' '--enable-x264' '--enable-schroedinger' '--enable-realrtsp' '--enable-live555' '--enable-dvdread' '--enable-shout' '--enable-goom' '--enable-caca' '--disable-sdl' '--enable-qt' '--enable-skins2' '--enable-sse' '--enable-mmx' '--enable-libcddb' '--enable-zvbi' '--disable-telx' '--enable-nls' '--host=i686-w64-mingw32' '--build=x86_64-pc-linux-gnu' 'build_alias=x86_64-pc-linux-gnu' 'host_alias=i686-w64-mingw32' 'PKG_CONFIG_PATH=/opt/intel/opencl:' 'PKG_CONFIG_LIBDIR=/home/svlad/DEV/vlc/contrib/i686-w64-mingw32/lib/pkgconfig'
  
  core libvlc debug: using multimedia timers as clock source
  
  core libvlc debug:  min period: 1 ms, max period: 1000000 ms
  
  core libvlc debug: searching plug-in modules
  
  core libvlc debug: loading plugins cache file C:\DEV\VLCStreamProcessor\build-untitled-Desktop_Qt_5_5_1_MSVC2013_32bit-Debug\debug\plugins\plugins.dat
  
  core libvlc debug: recursively browsing `C:\DEV\VLCStreamProcessor\build-untitled-Desktop_Qt_5_5_1_MSVC2013_32bit-Debug\debug\plugins'
  
  core libvlc debug: saving plugins cache C:\DEV\VLCStreamProcessor\build-untitled-Desktop_Qt_5_5_1_MSVC2013_32bit-Debug\debug\plugins\plugins.dat
  
  core libvlc debug: plug-ins loaded: 418 modules
  
  core libvlc debug: translation test: code is "C"
  
  core libvlc debug: CPU has capabilities MMX MMXEXT SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 FPU 
  
  core playlist debug: creating audio output
  
  core audio output debug: looking for audio output module matching "any": 6 candidates
  
  directsound audio output debug: found 2 devices
  
  core audio output debug: using audio output module "directsound"
  
  core playlist debug: keeping audio output
  
  core interface debug: looking for interface module matching "logger,none": 16 candidates
  
  logger interface: VLC media player - 2.2.4 Weatherwax
  
  logger interface: Copyright © 1996-2016 the VideoLAN team
  
  logger interface: 
  
  Warning: if you cannot access the GUI anymore, open a command-line window, go to the directory where you installed VLC and run "vlc -I qt"
  
  logger interface: using logger.
  
  logger interface warning: no log filename provided, using `vlc-log.txt'
  
  logger interface debug: opening logfile `vlc-log.txt'
  
  core libvlc debug: VLC media player - 2.2.4 Weatherwax
  
  core libvlc debug: Copyright © 1996-2016 the VideoLAN team
  
  core libvlc debug: revision 2.2.3-37-g888b7e89
  
  core libvlc debug: configured with ../extras/package/win32/../../../configure  '--enable-update-check' '--enable-lua' '--enable-faad' '--enable-flac' '--enable-theora' '--enable-twolame' '--enable-quicktime' '--enable-avcodec' '--enable-merge-ffmpeg' '--enable-dca' '--enable-mpc' '--enable-libass' '--enable-x264' '--enable-schroedinger' '--enable-realrtsp' '--enable-live555' '--enable-dvdread' '--enable-shout' '--enable-goom' '--enable-caca' '--disable-sdl' '--enable-qt' '--enable-skins2' '--enable-sse' '--enable-mmx' '--enable-libcddb' '--enable-zvbi' '--disable-telx' '--enable-nls' '--host=i686-w64-mingw32' '--build=x86_64-pc-linux-gnu' 'build_alias=x86_64-pc-linux-gnu' 'host_alias=i686-w64-mingw32' 'PKG_CONFIG_PATH=/opt/intel/opencl:' 'PKG_CONFIG_LIBDIR=/home/svlad/DEV/vlc/contrib/i686-w64-mingw32/lib/pkgconfig'
  
  core interface debug: using interface module "logger"
  
  core generic debug: creating audio output
  
  core audio output debug: looking for audio output module matching "any": 6 candidates
  
  directsound audio output debug: found 2 devices
  
  core audio output debug: using audio output module "directsound"
  
  core generic debug: keeping audio output
  
  core input debug: Creating an input for 'rtsp://192.168.0.245/live_st1'
  
  core stream output debug: using sout chain=`transcode{vcodec=RV24}:smem{video-prerender-callback=15667245,video-postrender-callback=15667630,video-data=3471900,no-time-sync},'
  
  core stream output debug: stream=`smem'
  
  core stream out debug: looking for sout stream module matching "smem": 22 candidates
  
  core stream out debug: set config option: sout-smem-video-prerender-callback to 15667245
  
  core stream out debug: set config option: sout-smem-video-postrender-callback to 15667630
  
  core stream out debug: set config option: sout-smem-video-data to 3471900
  
  core stream out debug: set config option: sout-smem-time-sync to (null)
  
  core stream out debug: using sout stream module "stream_out_smem"
  
  core stream output debug: stream=`transcode'
  
  core stream out debug: looking for sout stream module matching "transcode": 22 candidates
  
  core stream out debug: set config option: sout-transcode-vcodec to RV24
  
  stream_out_transcode stream out debug: Checking video codec mapping for RV24 got RV24 
  
  stream_out_transcode stream out debug: codec video=RV24 0x0 scaling: 0.000000 0kb/s
  
  core stream out debug: using sout stream module "stream_out_transcode"
  
  core input debug: using timeshift granularity of 50 MiB, in path 'C:\Users\BCDVideo\AppData\Local\Temp'
  
  core input debug: `rtsp://admin:admin@192.168.0.245/live_st1' gives access `rtsp' demux `' path `admin:admin@192.168.0.245/live_st1'
  
  core input debug: specified demux `any'
  
  core input debug: creating demux: access='rtsp' demux='any' location='admin:admin@192.168.0.245/live_st1' file='\\admin:admin@192.168.0.245\live_st1'
  
  core demux debug: looking for access_demux module matching "rtsp": 12 candidates
  
  live555 demux debug: version 2016.01.12
  
  live555 demux debug: RTP subsession 'video/H264'
  
  core input debug: selecting program id=0
  
  live555 demux debug: setup start: 0.000000 stop:0.000000
  
  live555 demux debug: We have a timeout of 60 seconds
  
  live555 demux debug: spawned timeout thread
  
  live555 demux debug: play start: 0.000000 stop:0.000000
  
  core demux debug: using access_demux module "live555"
  
  core decoder debug: looking for packetizer module matching "any": 23 candidates
  
  packetizer_h264 decoder debug: found NAL_SPS (sps_id=0)
  
  packetizer_h264 decoder debug: found NAL_PPS (pps_id=0 sps_id=0)
  
  core decoder debug: using packetizer module "packetizer_h264"
  
  core input debug: starting in sync mode
  
  core demux meta debug: looking for meta reader module matching "any": 2 candidates
  
  lua demux meta debug: Trying Lua scripts in C:\Users\BCDVideo\AppData\Roaming\vlc\lua\meta\reader
  
  lua demux meta debug: Trying Lua scripts in C:\DEV\VLCStreamProcessor\build-untitled-Desktop_Qt_5_5_1_MSVC2013_32bit-Debug\debug\lua\meta\reader
  
  core demux meta debug: no meta reader modules matched
  
  core input debug: `rtsp://admin:admin@192.168.0.245/live_st1' successfully opened
  
  core input debug: switching to async mode
  
  core input debug: Buffering 0%
  
  core input debug: Buffering 3%
  
  core input debug: Buffering 10%
  
  core stream output debug: adding a new sout input (sout_input:00c21c50)
  
  core input debug: Buffering 13%
  
  stream_out_transcode stream out debug: creating video transcoding from fcc=`h264' to fcc=`RV24'
  
  core input debug: Buffering 16%
  
  core generic debug: looking for decoder module matching "any": 43 candidates
  
  core input debug: Buffering 20%
  
  core input debug: Buffering 23%
  
  avcodec generic debug: CPU flags: 0x0107d3db
  
  core input debug: Buffering 30%
  
  avcodec generic debug: trying to use direct rendering
  
  core input debug: Buffering 33%
  
  avcodec generic debug: allowing 4 thread(s) for decoding
  
  core input debug: Buffering 36%
  
  avcodec generic warning: threaded frame decoding is not compatible with DXVA2, disabled
  
  core input debug: Buffering 40%
  
  avcodec generic debug: avcodec codec (H264 - MPEG-4 AVC (part 10)) started
  
  core input debug: Buffering 43%
  
  avcodec generic debug: using slice thread mode with 4 threads
  
  core input debug: Buffering 50%
  
  core generic debug: using decoder module "avcodec"
  
  core input debug: Buffering 53%
  
  core encoder debug: looking for encoder module matching "any": 20 candidates
  
  core input debug: Buffering 56%
  
  core input debug: Buffering 60%
  
  avcodec encoder debug: CPU flags: 0x0107d3db
  
  core input debug: Buffering 63%
  
  avcodec encoder debug: Time base for probing setted to 1/1000000
  
  core input debug: Buffering 70%
  
  avcodec encoder debug: Time base set to 1/1000000
  
  core input debug: Buffering 73%
  
  avcodec encoder debug: found encoder Raw video
  
  core input debug: Buffering 76%
  
  core encoder debug: using encoder module "avcodec"
  
  core input debug: Buffering 80%
  
  core encoder debug: removing module "avcodec"
  
  core input debug: Buffering 83%
  
  core input debug: Buffering 90%
  
  core input debug: Buffering 93%
  
  core input debug: Buffering 96%
  
  core input debug: Stream buffering done (1002 ms in 834 ms)
  
  core input debug: Decoder wait done in 0 ms
  
  avcodec generic debug: available hardware decoder output format 61 (dxva2_vld)
  
  avcodec generic debug: available hardware decoder output format 128 (d3d11va_vld)
  
  avcodec generic debug: available software decoder output format 12 (yuvj420p)
  
  core generic debug: looking for hw decoder module matching "any": 1 candidates
  
  core generic debug: no hw decoder modules matched
  
  avcodec generic warning: plane 0 not aligned
  
  avcodec generic warning: disabling direct rendering
  
  stream_out_transcode stream out debug: decoder aspect is 1.761468:1
  
  stream_out_transcode stream out debug: source pixel aspect is 0.990826:1
  
  stream_out_transcode stream out debug: scaled pixel aspect is 0.990826:1
  
  stream_out_transcode stream out debug: source 1920x1080, destination 1920x1080
  
  stream_out_transcode stream out debug: source fps 30/1, destination 30/1
  
  stream_out_transcode stream out: input interval 33333 (base 1)
  
  stream_out_transcode stream out: output interval 33333 (base 1)
  
  stream_out_transcode stream out debug: encoder aspect is 1920:1090
  
  core filter debug: looking for video filter2 module matching "any": 58 candidates
  
  swscale filter debug: 1920x1080 (1920x1090) chroma: J420 -> 1920x1080 (1920x1090) chroma: RV24 with scaling using Bicubic (good quality)
  
  core filter debug: using video filter2 module "swscale"
  
  core stream out debug: Filter 'Swscale' (0089a2cc) appended to chain
  
  stream_out_transcode stream out debug: destination (after video filters) 1920x1090
  
  core encoder debug: looking for encoder module matching "any": 20 candidates
  
  avcodec encoder debug: CPU flags: 0x0107d3db
  
  avcodec encoder debug: Time base for probing setted to 1/30
  
  avcodec encoder debug: Time base set to 1/30
  
  avcodec encoder debug: found encoder Raw video
  
  core encoder debug: using encoder module "avcodec"
  
  stream_out_transcode stream out warning: Reseting video sync
  
  stream_out_transcode stream out warning: Reseting video sync
  
  stream_out_transcode stream out warning: Reseting video sync
  
  stream_out_transcode stream out warning: Reseting video sync
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518755, last: 1518769)
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518756, last: 1518769)
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518757, last: 1518769)
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518758, last: 1518769)
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518759, last: 1518769)
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518760, last: 1518769)
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518761, last: 1518769)
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518762, last: 1518769)
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518763, last: 1518769)
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518764, last: 1518769)
  
  stream_out_transcode stream out warning: Reseting video sync
  
  stream_out_transcode stream out warning: Reseting video sync
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518784, last: 1518795)
  
  live555 demux debug: tk->rtpSource->hasBeenSynchronizedUsingRTCP()
  
  core input error: ES_OUT_RESET_PCR called
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518785, last: 1518795)
  
  core input debug: Buffering 0%
  
  core input debug: Buffering 3%
  
  core input debug: Buffering 6%
  
  core input debug: Buffering 10%
  
  core input debug: Buffering 13%
  
  core input debug: Buffering 20%
  
  core input debug: Buffering 23%
  
  core input debug: Buffering 26%
  
  core input debug: Buffering 30%
  
  core input debug: Buffering 33%
  
  core input debug: Buffering 40%
  
  core input debug: Buffering 43%
  
  core input debug: Buffering 46%
  
  core input debug: Buffering 50%
  
  core input debug: Buffering 53%
  
  core input debug: Buffering 60%
  
  core input debug: Buffering 63%
  
  core input debug: Buffering 66%
  
  core input debug: Buffering 70%
  
  core input debug: Buffering 73%
  
  core input debug: Buffering 80%
  
  core input debug: Buffering 83%
  
  core input debug: Buffering 86%
  
  core input debug: Buffering 90%
  
  core input debug: Buffering 93%
  
  core input debug: Stream buffering done (1002 ms in 817 ms)
  
  core input debug: Decoder wait done in 0 ms
  
  stream_out_transcode stream out warning: Reseting video sync
  
  stream_out_transcode stream out warning: Reseting video sync
  
  stream_out_transcode stream out warning: Reseting video sync
  
  stream_out_transcode stream out warning: Reseting video sync
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518834, last: 1518837)
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518835, last: 1518837)
  
  avcodec encoder warning: almost fed libavcodec with a frame in the past (current: 1518836, last: 1518837)
  
  avcodec encoder warning: almost fed libavcodec with two frames with the same PTS (1518837)
  
  stream_out_transcode stream out warning: Reseting video sync
  
  stream_out_transcode stream out warning: Reseting video sync

• FFMPEG Audio/video out of sync after cutting and concatonating even after transcoding

  4 mai 2020, par Ham789

  I am attempting to take cuts from a set of videos and concatonate them together with the concat demuxer.

  



  However, the audio is out of sync of the video in the output. The audio seems to drift further out of sync as the video progresses. Interestingly, if I click to seek another time in the video with the progress bar on the player, the audio becomes synced up with the video but then gradually drifts out of sync again. Seeking to a new time in the player seems to reset the audio/video. It is like they are being played back at different rates or something. I get this behaviour in both Quicktime and VLC players.

  



  For each video, I decode it, trim a clip from it and then encode it to 4k resolution at 25 fps with its audio :

  



  ffmpeg -ss 0.5 -t 0.5 -i input_video1.mp4 -r 25 -vf scale=3840:2160 output_video1.mp4

  



  I then take each of these videos and concatonate them together with the concat demuxer :

  



  ffmpeg -f concat -safe 0 -i cut_videos.txt -c copy -y output.mp4

  



  I am taking short cuts of each video (approximately 0.5s)

  



  I am using Python's subprocess to automate the cutting and concatonating of the videos.

  



  I am not sure if this happens because of the trimming or concatenation steps but when I play back the intermediate cut video files (output_video1.mp4 in the above command), there seems to be some silence before the audio comes in at the start of the video.

  



  When I concatonate the videos, I sometimes get a lot of these warnings however the audio still becomes out of sync even when I do not get them :

  



  [mp4 @ 0000021a252ce080] Non-monotonous DTS in output stream 0:1; previous: 51792, current: 50009; changing to 51793. This may result in incorrect timestamps in the output file.

  



  From this post, it seems to be a problem with cutting the videos and their timestamps. The solution proposed in the post is to decode, cut and then encode the video however I am already doing that.

  



  How can I ensure the audio and video are in sync ? Am I transcoding incorrectly ? This seems to be the only solution I can find online however it does not seem to work.

  



  UPDATE :

  



  I took inspiration from this post and seperated the audio and video from output_video1.mp4 using :

  



  ffmpeg -i output_video1.mp4 -acodec copy -vn video.mp4

  



  and

  



  ffmpeg -i output_video1.mp4 -vcodec copy -an audio.mp4

  



  I then compared the durations of video.mp4 and audio.mp4 and got 0.57s and 0.52s respectively. Since the video is longer, this explains why there is a period of silence in the videos. The post then suggests transcoding is the solution however as you can see from the code above that does not work for me.

  



  Sample Output Log for the Trim Command

  



    built with Apple LLVM version 10.0.0 (clang-1000.11.45.5)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags='-I/Library/Java/JavaVirtualMachines/adoptopenjdk-13.0.1.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-13.0.1.jdk/Contents/Home/include/darwin' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input_video1.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.29.100
  Duration: 00:00:04.06, start: 0.000000, bitrate: 14266 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 3840x2160, 14268 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
    Metadata:
      handler_name    : Core Media Video
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, mono, fltp, 94 kb/s (default)
    Metadata:
      handler_name    : Core Media Audio
File 'output_video1.mp4' already exists. Overwrite ? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[libx264 @ 0x7fcae4001e00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x7fcae4001e00] profile High, level 5.1
[libx264 @ 0x7fcae4001e00] 264 - core 155 r2917 0a84d98 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'output_video1.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.29.100
    Stream #0:0(und): Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 3840x2160, q=-1--1, 25 fps, 12800 tbn, 25 tbc (default)
    Metadata:
      handler_name    : Core Media Video
      encoder         : Lavc58.54.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, mono, fltp, 69 kb/s (default)
    Metadata:
      handler_name    : Core Media Audio
      encoder         : Lavc58.54.100 aac
frame=   14 fps=7.0 q=-1.0 Lsize=     928kB time=00:00:00.51 bitrate=14884.2kbits/s dup=0 drop=1 speed=0.255x    
video:922kB audio:5kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.194501%
[libx264 @ 0x7fcae4001e00] frame I:1     Avg QP:21.06  size:228519
[libx264 @ 0x7fcae4001e00] frame P:4     Avg QP:22.03  size: 85228
[libx264 @ 0x7fcae4001e00] frame B:9     Avg QP:22.88  size: 41537
[libx264 @ 0x7fcae4001e00] consecutive B-frames: 14.3%  0.0%  0.0% 85.7%
[libx264 @ 0x7fcae4001e00] mb I  I16..4: 27.6% 64.3%  8.1%
[libx264 @ 0x7fcae4001e00] mb P  I16..4:  9.1% 10.7%  0.2%  P16..4: 48.5%  7.3%  3.9%  0.0%  0.0%    skip:20.2%
[libx264 @ 0x7fcae4001e00] mb B  I16..4:  1.1%  1.0%  0.0%  B16..8: 44.5%  2.9%  0.2%  direct: 8.3%  skip:42.0%  L0:45.6% L1:53.2% BI: 1.2%
[libx264 @ 0x7fcae4001e00] 8x8 transform intra:58.2% inter:93.4%
[libx264 @ 0x7fcae4001e00] coded y,uvDC,uvAC intra: 31.4% 62.2% 5.2% inter: 11.4% 30.9% 0.0%
[libx264 @ 0x7fcae4001e00] i16 v,h,dc,p: 15% 52% 12% 21%
[libx264 @ 0x7fcae4001e00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 19% 33% 32%  2%  2%  2%  4%  2%  4%
[libx264 @ 0x7fcae4001e00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 39%  9%  3%  4%  4% 12%  3%  4%
[libx264 @ 0x7fcae4001e00] i8c dc,h,v,p: 43% 36% 18%  3%
[libx264 @ 0x7fcae4001e00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fcae4001e00] ref P L0: 69.3%  8.0% 14.8%  7.9%
[libx264 @ 0x7fcae4001e00] ref B L0: 88.1%  9.2%  2.6%
[libx264 @ 0x7fcae4001e00] ref B L1: 90.2%  9.8%
[libx264 @ 0x7fcae4001e00] kb/s:13475.29
[aac @ 0x7fcae4012400] Qavg: 125.000```


  


• My python script using ffmpeg captures video content, but the captured content freezes in the middle and jumps frames

  11 novembre 2022, par Supriyo Mitra

  I am new to ffmpeg and I am trying to use it through a python script. The python functions that captures the video content is given below. The problem I am facing is that the captured content freezes at (uneven) intervals and skips a few frames every time it happens.

  


  `    def capturelivestream(self, argslist):
        streamurl, outnum, feedid, outfilename = argslist[0], argslist[1], argslist[2], argslist[3]
        try:
            info = ffmpeg.probe(streamurl, select_streams='a')
            streams = info.get('streams', [])
        except:
            streams = []
        if len(streams) == 0:
            print('There are no streams available')
            stream = {}
        else:
            stream = streams[0]
            for stream in streams:
                if stream.get('codec_type') != 'audio':
                    continue
                else:
                    break
        if 'channels' in stream.keys():
            channels = stream['channels']
            samplerate = float(stream['sample_rate'])
        else:
            channels = None
            samplerate = 44100
        process = ffmpeg.input(streamurl).output('pipe:', pix_fmt='yuv420p', format='avi', vcodec='libx264', acodec='pcm_s16le', ac=channels, ar=samplerate, vsync=0, loglevel='quiet').run_async(pipe_stdout=True)
        fpath = os.path.dirname(outfilename)
        fnamefext = os.path.basename(outfilename)
        fname = fnamefext.split(".")[0]
        read_size = 320 * 180 * 3 # This is width * height * 3
        lastcaptured = time.time()
        maxtries = 12
        ntries = 0
        while True:
            if process:
                inbytes = process.stdout.read(read_size)
                if inbytes is not None and inbytes.__len__() > 0:
                    try:
                        frame = (np.frombuffer(inbytes, np.uint8).reshape([180, 320, 3]))
                    except:
                        print("Failed to reshape frame: %s"%sys.exc_info()[1].__str__())
                        continue # This could be an issue if there is a continuous supply of frames that cannot be reshaped
                    self.processq.put([outnum, frame])
                    lastcaptured = time.time()
                    ntries = 0
                else:
                    if self.DEBUG:
                        print("Could not read frame for feed ID %s"%feedid)
                    t = time.time()
                    if t - lastcaptured > 30: # If the frames can't be read for more than 30 seconds...
                        print("Reopening feed identified by feed ID %s"%feedid)
                        process = ffmpeg.input(streamurl).output('pipe:', pix_fmt='yuv420p', format='avi', vcodec='libx264', acodec='pcm_s16le', ac=channels, ar=samplerate, vsync=0, loglevel='quiet').run_async(pipe_stdout=True)
                        ntries += 1
                    if ntries > maxtries:
                        if self.DEBUG:
                            print("Stream %s is no longer available."%streamurl)
                        # DB statements removed here
                        
                        break # Break out of infinite loop.
                    continue
        
        return None`



  


  The function that captures the frames is as follows :

  


  
`    def framewriter(self, outlist):
        isempty = False
        endofrun = False
        while True:
            frame = None
            try:
                args = self.processq.get()
            except: # Sometimes, the program crashes at this point due to lack of memory...
                print("Error in framewriter while reading from queue: %s"%sys.exc_info()[1].__str__())
                continue
            outnum = args[0]
            frame = args[1]
            if outlist.__len__() > outnum:
                out = outlist[outnum]
            else:
                if self.DEBUG == 2:
                    print("Could not get writer %s"%outnum)
                continue
            if frame is not None and out is not None:
                out.write(frame)
                isempty = False
                endofrun = False
            else:
                if self.processq.empty() and not isempty:
                    isempty = True
                elif self.processq.empty() and isempty: # processq queue is empty now and was empty last time
                    print("processq is empty")
                    endofrun = True
                elif endofrun and isempty:
                    print("Could not find any frames to process. Quitting")
                    break
        print("Done writing feeds. Quitting.")
        return None`


  


  The scenario is as follows : There are multiple video streams from a certain website at any time during the day, and the program containing these functions has to capture them as they get streamed. The memory available to this program is 6GB and there could be upto 3 streams running at any instant. Given below is the relevant main section of the script that uses the functions given above.

  


  


  


  `itftennis = VideoBot(siteurl)
outlist = []
t = Thread(target=itftennis.framewriter, args=(outlist,))
t.daemon = True
t.start()
tp = Thread(target=handleprocesstermination, args=())
tp.daemon = True
tp.start()
# Create a database connection and as associated cursor object. We will handle database operations from main thread only.
# DB statements removed from here...
feedidlist = []
vidsdict = {}
streampattern = re.compile("\?vid=(\d+)$")
while True:
    streampageurls = itftennis.checkforlivestream()
    if itftennis.DEBUG:
        print("Checking for new urls...")
        print(streampageurls.__len__())
    if streampageurls.__len__() > 0:
        argslist = []
        newurlscount = 0
        for streampageurl in streampageurls:
            newstream = False
            sps = re.search(streampattern, streampageurl)
            if sps:
                streamnum = sps.groups()[0]
                if streamnum not in vidsdict.keys(): # Check if this stream has already been processed.
                    vidsdict[streamnum] = 1
                    newstream = True
                else:
                    continue
            else:
                continue
            print("Detected new live stream... Getting it.")
            streamurl = itftennis.getstreamurlfrompage(streampageurl)
            print("Adding %s to list..."%streamurl)
            if streamurl is not None:
                # Now, get feed metadata...
                metadata = itftennis.getfeedmetadata(streampageurl)
                if metadata is None:
                    continue
                # lines to get matchescounter omitted here...
                if matchescounter >= itftennis.__class__.MAX_CONCURRENT_MATCHES:
                    break
                if newstream is True:
                    newurlscount += 1
                outfilename = time.strftime("./videodump/" + "%Y%m%d%H%M%S",time.localtime())+".avi"
                out = open(outfilename, "wb")
                outlist.append(out) # Save it in the list and take down the number for usage in framewriter
                outnum = outlist.__len__() - 1
                # Save metadata in DB
                # lines omitted here....
                argslist.append([streamurl, outnum, feedid, outfilename])   
            else:
                print("Couldn't get the stream url from page")
        if newurlscount > 0:
            for args in argslist:
                try:
                    p = Process(target=itftennis.capturelivestream, args=(args,))
                    p.start()
                    processeslist.append(p)
                    if itftennis.DEBUG:
                        print("Started process with args %s"%args)
                except:
                    print("Could not start process due to error: %s"%sys.exc_info()[1].__str__())
            print("Created processes, continuing now...")
            continue
    time.sleep(itftennis.livestreamcheckinterval)
t.join()
tp.join()
for out in outlist:
    out.close()`


  


  


  


  Please accept my apologies for swamping with this amount of code. I wanted to provide maximum context to my problem. I have removed the absolutely irrelevant DB statements, but apart from that this is what the code looks like.

  


  If you need to know anything else about the code, please let me know. What I would really like to know is if I am using the ffmpeg streams capturing statements correctly. The stream contains both video and audio components and I need to capture both. Hence I am making the following call :

  


  process = ffmpeg.input(streamurl).output('pipe:', pix_fmt='yuv420p', format='avi', vcodec='libx264', acodec='pcm_s16le', ac=channels, ar=samplerate, vsync=0, loglevel='quiet').run_async(pipe_stdout=True)


  


  Is this how it is supposed to be done ? More importantly, why do I keep getting the freezes in the output video. I have monitored the streams manually, and they are quite consistent. Frame losses do not happen when I view them on the website (at least it is not obviously noticeable). Also, I have run 'top' command on the host running the program. The CPU usage sometimes go over 100% (which, I came to understand from some answers on SO, is to be expected when running ffmpeg) but the memory usage usually remain below 30%. So what is the issue here. What do I need to do in order to fix this problem (other than learn more about how ffmpeg works).

  


  Thanks

  


  I have tried using various ffmpeg options (while trying to find similar issues that others encountered). I also tried running ffmpeg from command line for a limited period of time (11 mins), using the same options as used in the python code, and the captured content came out quite well. No freezes. No jumps in frames. But I need to use it in an automated way and there would be multiple streams at any time. Also, when I try playing the captured content using ffplay, I sometimes get the message "co located POCs unavailable" when these freezes happen. What does it mean ?