Recherche avancée

Sur d’autres sites (3586)

• How to extract motion vectors from h264 without a full decode on the CPU

  25 septembre 2020, par Adrian May

  I'm trying to use my nose as a pointing device. The plan is to encode the video stream from a webcam pointed at my face as h264 or the like, get the motion vectors, cook the numbers a bit and chuck them into /dev/uinput to make the mouse pointer move about. The uinput bit was easy.

  


  This has to work with zero discernable latency. This, for instance :

  


  #!/bin/bash
[ -p pipe.mkv ] || mkfifo pipe.mkv
ffmpeg -y -rtbufsize 1M -s 640x360 -vcodec mjpeg -i /dev/video0 -c h264_nvenc pipe.mkv &
ffplay -flags2 +export_mvs -vf codecview=mv=pf+bf+bb pipe.mkv


  


  shows that the vectors are there but with a latency of several seconds which is unusable in a mouse. I know that the first ffmpeg step is working very fast by using the GPU, so either the pipe or the h264 decode in the second step is introducing the latency.

  


  I tried MV Tractus (same as mpegflow I think) in a similar pipe arrangement and it was also very slow. They do a full h264 decode on the CPU and I think that's the problem cos I can see them imposing a lot of load on one CPU. If the pipe had caused the delay by buffering badly then the CPU wouldn't have been loaded. I guess ffplay also did the decoding on the CPU and I couldn't persuade it not to, but it only wants to draw arrows which are no use to me.

  


  I think there are several approaches, and I'd like advice on which would be best, or if there's something even better I don't know about. I could :

  


  

  • Decode in hardware and get the motion vectors. So far this has failed. I tried combining ffmpeg's extract_mvs.c and hw_decode.c samples but no motion vectors turn up. vdpau is the only decoder I got working on my linux box. I have a nvidia gpu.

  


  • Do a minimal parse of the h264 to fish out the motion vectors only, ignoring all the other data. I think this would mean putting some kind of "motion only" option in libav's parser, but I'm not at all familiar with that code.

  


  • Find some other h264 parsing library that has said option and also unpacks the container.

  


  • Forget about hardware accelerated encoding and use a stripped down encoder to make only the motion vectors on either CPU or GPU. I suspect this would be slow cos I think calculating the motion vectors is the hardest part of the algorithm.

  


  


  I'm tending towards the second option but I need some help figuring out where in the libav code to do it.

  


• Playing RTSP in WPF application with low latency using FFMPEG / FFMediaElement (FFME)

  22 mars 2019, par Paboka

  I’m trying to use FFMediaElement (FFME, WPF MediaElement replacement based on FFmpeg) component to play RSTP live video in my WPF application.

  I have a good connection to my camera and I want to play it with minimum available latency.

  I’ve reduced the latency by changing ProbeSize to its minimal value :

  private void Media_OnMediaInitializing(object Sender, MediaInitializingRoutedEventArgs e)
  
  {
  
    e.Configuration.GlobalOptions.ProbeSize = 32;
  
  }

  But I still have about 1 second of latency since the very beginning of the stream. I mean, when I start playing, I have to wait for 1 second till the video appears and then I have 1s of latency.

  I’ve also tried to change following parameters :

  e.Configuration.GlobalOptions.EnableReducedBuffering = true;
  
  e.Configuration.GlobalOptions.FlagNoBuffer = true;
  
  e.Configuration.GlobalOptions.MaxAnalyzeDuration = TimeSpan.Zero;

  but it gave no result.

  I measured time-interval between FFmpeg output lines (the number in the first column is the time, elapsed from the previous line, ms)

   ----     OpenCommand: Entered
  
     39     FFInterop.Initialize: FFmpeg v4.0
  
      0     EVENT START: MediaInitializing
  
      0     EVENT DONE : MediaInitializing
  
    379     EVENT START: MediaOpening
  
      0     EVENT DONE : MediaOpening
  
      0     COMP VIDEO: Start Offset:      0,000; Duration:        N/A
  
     41     SYNC-BUFFER: Started.
  
    609     SYNC-BUFFER: Finished. Clock set to 1534932751,634
  
      0     EVENT START: MediaOpened
  
      0     EVENT DONE : MediaOpened
  
      0     EVENT START: BufferingStarted
  
      0     EVENT DONE : BufferingStarted
  
      0     OpenCommand: Completed
  
      0     V BLK: 1534932751,634 | CLK: 1534932751,634 | DFT:    0 | IX:   0 | PQ:     0,0k | TQ:     0,0k
  
      0     Command Queue (1 commands): Before ProcessNext
  
      0        Play - ID: 404 Canceled: False; Completed: False; Status: WaitingForActivation; State: 
  
     94     V BLK: 1534932751,675 | CLK: 1534932751,699 | DFT:   24 | IX:   1 | PQ:     0,0k | TQ:     0,0k

  So, the most the process of "sync-buffering" takes the most of the time.

  Is there any parameter of FFmpeg which allows reducing a size of the buffer ?

• Use ffmpeg libraries to convert stream formats

  17 septembre 2021, par Syrinx

  I'm attempting to write a small program and link it to a minimal set of ffmpeg libraries, like libavformat and whatever other libraries I need.

  


  I am looking for documentation to get me started, or maybe a quick fix to the example program I am using.

  


  I know ffmpeg (the project) provides example programs to help developers get started. I'm using the transcoding example, as it's close to my end goal, but it exits during init with an error about an audio issue.

  


  Here I am using the transcoding example program that come with ffmpeg v4.4, on Ubuntu 18.04. My input source has one video channel (h264) and one audio channel (pcm_mulaw).

  


  $ LD_LIBRARY_PATH=../../dist/lib ./transcoding rtsp://ip-camera/stream out.flv
...
  Stream #0:0: Video: h264, yuv420p, 1280x720, q=2-31, 20 tbn
  Stream #0:1: Audio: pcm_mulaw, 8000 Hz, 0 channels, s16
auto_resampler_0 @ 0x55da787fa140] [SWR @ 0x55da787fa5c0] Rematrix is needed between mono and 0 channels but there is not enough information to do it
[auto_resampler_0 @ 0x55da787fa140] Failed to configure output pad on auto_resampler_0


  


  In libswresample/swresample.c :

  


  320      if ((!s->out_ch_layout || !s->in_ch_layout) && s->used_ch_count != s->out.ch_count && !s->rematrix_custom) {
321          av_log(s, AV_LOG_ERROR, "Rematrix is needed between %s and %s "
322                 "but there is not enough information to do it\n", l1, l2);
323          ret = AVERROR(EINVAL);
324          goto fail;
325      }


  


  I'd really like it if I could make the transcoding example program work (fix it, or maybe use it appropriately if I am misunderstanding something). But short of that, where should I look for documentation about using the ffmpeg libraries ?

  


  I don't even care about the audio. If I can just disable the audio, I would be happy with that solution. I tried tracking the "-an" option to ffmpeg (the program) to see how it does that in source code, but the options handling is a mess and I can't distinguish the parts of the code that I need from all the noise.

  


  ffmpeg has web pages like this that aren't useful at all. There is documentation in the source code that looks like it should be viewed as HTML, but I don't see it exported anywhere. "make doc" generates a very small set of man pages that are insufficient to get me started.