Recherche avancée

Sur d’autres sites (8482)

• FFmpeg h264_v4l2m2m to rtmp

  6 juillet 2021, par KnightRex

  On an up to date Raspberry Pi 4 B+ 4GB with the Raspberry OS 32bit I'm attempting to stream to an rtmp service, like Youtube or Twitch with FFmpeg using the hardware encoder h264_v4l2m2m but it fails.

  


  Issue

  


  When streaming the streaming service it ingests the input, bitrate and goes live but no output is shown. When connecting to Twitch the stream inspector correctly detects video resolution but fails to detect the codec or fps.

  


  When I replace the stream output to for example a file, test.flv, it records a viewable video with the correct codec (h264/aac) codecs in vlc and ffplay.

  


  When I replace the h264_v4l2m2m encoder with h264_omx it streams the video correctly.

  


  Steps

  


  


  • Installed a clean version of the latest Raspi OS 32bit

    


  


  • Recompiled FFmpeg to 4.3.2 (to the solve green screen)

    


  


  • Record video, creates viewable video :

    


    ffmpeg \
-f v4l2 -input_format mjpeg -video_size 1280x720 -framerate 30 -i /dev/video0 \
-c:v h264_v4l2m2m -g 60 -pix_fmt yuv420p -b:v 3000k -minrate 3000k -maxrate 3000k -r 30 -an \
-t 15 test.flv


    


  


  • Start stream, unable to view :

    


    ffmpeg \&#xA;-f v4l2 -input_format mjpeg -video_size 1280x720 -framerate 30 -i /dev/video0 \&#xA;-c:v h264_v4l2m2m -g 60 -pix_fmt yuv420p -b:v 3000k -minrate 3000k -maxrate 3000k -r 30 -an \&#xA;-f flv "<destination uri="uri">"&#xA;</destination>

    


  


  • Start stream, unable to view (Extended based on comments below) :

    


    ffmpeg \&#xA; -re -f lavfi -i anullsrc \&#xA; -f v4l2 -input_format mjpeg -video_size 1280x720 -framerate 30 -i /dev/video0 \&#xA; -c:v h264_v4l2m2m -g 60 -pix_fmt yuv420p -b:v 3000k -minrate 3000k -maxrate 3000k -r 30 \&#xA; -c:a aac \&#xA; -f flv "<destination uri="uri">"&#xA;</destination>

    


  


  • Start stream, with omx works :

    


    ffmpeg \&#xA;-f v4l2 -input_format mjpeg -video_size 1280x720 -framerate 30 -i /dev/video0 \&#xA;-c:v h264_omx -preset ultrafast -pix_fmt yuv420p -g 60 -b:v 3000k -minrate 3000k -maxrate 3000k -r 30 -an \&#xA;-f flv "<destination uri="uri">"&#xA;</destination>

    


  


  


  Output

  


  With h264_v4l2m2m (invalid output - loglevel verbose) :

  


  $ ffmpeg -re -f lavfi -i anullsrc  -f v4l2 -input_format mjpeg -video_size 1280x720 -framerate 30 -i /dev/video0  -c:v h264_v4l2m2m -g 60 -pix_fmt yuv420p -b:v 3000k -minrate 3000k -maxrate 3000k -r 30 -c:a aac -f flv <destination url="url">&#xA;ffmpeg version n4.3.2 Copyright (c) 2000-2021 the FFmpeg developers&#xA;  built with gcc 8 (Raspbian 8.3.0-6&#x2B;rpi1)&#xA;  configuration: --prefix=/home/pi/ffmpeg_build --bindir=/home/pi/bin --extra-cflags=-I/usr/local/include --extra-ldflags=-L/usr/local/lib --extra-libs=&#x27;-lpthread -lm -latomic&#x27; --enable-omx --enable-encoder=h264_omx&#xA;  libavutil      56. 51.100 / 56. 51.100&#xA;  libavcodec     58. 91.100 / 58. 91.100&#xA;  libavformat    58. 45.100 / 58. 45.100&#xA;  libavdevice    58. 10.100 / 58. 10.100&#xA;  libavfilter     7. 85.100 /  7. 85.100&#xA;  libswscale      5.  7.100 /  5.  7.100&#xA;  libswresample   3.  7.100 /  3.  7.100&#xA;[Parsed_anullsrc_0 @ 0x7215c0] sample_rate:44100 channel_layout:&#x27;stereo&#x27; nb_samples:1024&#xA;Input #0, lavfi, from &#x27;anullsrc&#x27;:&#xA;  Duration: N/A, start: 0.000000, bitrate: 705 kb/s&#xA;    Stream #0:0: Audio: pcm_u8, 44100 Hz, stereo, u8, 705 kb/s&#xA;[video4linux2,v4l2 @ 0x725350] fd:3 capabilities:84a00001&#xA;[mjpeg @ 0x1e10210] EOI missing, emulating&#xA;Input #1, video4linux2,v4l2, from &#x27;/dev/video0&#x27;:&#xA;  Duration: N/A, start: 42396.344658, bitrate: N/A&#xA;    Stream #1:0: Video: mjpeg (Baseline), yuvj422p(pc, bt470bg/unknown/unknown), 1280x720, 30 fps, 30 tbr, 1000k tbn, 1000k tbc&#xA;[tcp @ 0x72cac0] Starting connection attempt to <destination ip="ip"> port 1935&#xA;[tcp @ 0x72cac0] Successfully connected to <destination ip="ip"> port 1935&#xA;Stream mapping:&#xA;  Stream #1:0 -> #0:0 (mjpeg (native) -> h264 (h264_v4l2m2m))&#xA;  Stream #0:0 -> #0:1 (pcm_u8 (native) -> aac (native))&#xA;Press [q] to stop, [?] for help&#xA;[graph 0 input from stream 1:0 @ 0x73de00] w:1280 h:720 pixfmt:yuvj422p tb:1/1000000 fr:30/1 sar:0/1 sws_param:flags=2&#xA;[auto_scaler_0 @ 0x745400] w:iw h:ih flags:&#x27;bicubic&#x27; interl:0&#xA;[format @ 0x73e750] auto-inserting filter &#x27;auto_scaler_0&#x27; between the filter &#x27;Parsed_null_0&#x27; and the filter &#x27;format&#x27;&#xA;[swscaler @ 0x1e300b0] deprecated pixel format used, make sure you did set range correctly&#xA;[auto_scaler_0 @ 0x745400] w:1280 h:720 fmt:yuvj422p sar:0/1 -> w:1280 h:720 fmt:yuv420p sar:0/1 flags:0x4&#xA;[h264_v4l2m2m @ 0x72a500] driver &#x27;bcm2835-codec&#x27; on card &#x27;bcm2835-codec-isp&#x27;&#xA;[h264_v4l2m2m @ 0x72a500] driver &#x27;uvcvideo&#x27; on card &#x27;USB Video: USB Video&#x27;&#xA;    Last message repeated 1 times&#xA;[h264_v4l2m2m @ 0x72a500] driver &#x27;bcm2835-codec&#x27; on card &#x27;bcm2835-codec-encode&#x27;&#xA;[h264_v4l2m2m @ 0x72a500] Using device /dev/video11&#xA;[h264_v4l2m2m @ 0x72a500] driver &#x27;bcm2835-codec&#x27; on card &#x27;bcm2835-codec-encode&#x27;&#xA;[h264_v4l2m2m @ 0x72a500] Failed to set number of B-frames&#xA;    Last message repeated 1 times&#xA;[h264_v4l2m2m @ 0x72a500] Failed to set gop size&#xA;[h264_v4l2m2m @ 0x72a500] h264 profile not found&#xA;[h264_v4l2m2m @ 0x72a500] Encoder adjusted: qmin (0), qmax (51)&#xA;[h264_v4l2m2m @ 0x72a500] Failed to set minimum video quantizer scale&#xA;[h264_v4l2m2m @ 0x72a500] Failed to set maximum video quantizer scale&#xA;[graph_1_in_0_0 @ 0x786f90] tb:1/44100 samplefmt:u8 samplerate:44100 chlayout:0x3&#xA;[format_out_0_1 @ 0x7871d0] auto-inserting filter &#x27;auto_resampler_0&#x27; between the filter &#x27;Parsed_anull_0&#x27; and the filter &#x27;format_out_0_1&#x27;&#xA;[auto_resampler_0 @ 0x7887f0] ch:2 chl:stereo fmt:u8 r:44100Hz -> ch:2 chl:stereo fmt:fltp r:44100Hz&#xA;Output #0, flv, to &#x27;<destination url="url">&#x27;:&#xA;  Metadata:&#xA;    encoder         : Lavf58.45.100&#xA;    Stream #0:0: Video: h264 (h264_v4l2m2m) ([7][0][0][0] / 0x0007), yuv420p(progressive), 1280x720, q=-1--1, 3000 kb/s, 30 fps, 1k tbn, 30 tbc&#xA;    Metadata:&#xA;      encoder         : Lavc58.91.100 h264_v4l2m2m&#xA;    Stream #0:1: Audio: aac (LC) ([10][0][0][0] / 0x000A), 44100 Hz, stereo, fltp, 128 kb/s&#xA;    Metadata:&#xA;      encoder         : Lavc58.91.100 aac&#xA;[video4linux2,v4l2 @ 0x1e0f410] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)&#xA;frame= 1966 fps= 24 q=-0.0 size=   13016kB time=00:01:23.60 bitrate=1275.4kbits/s speed=   1x&#xA;</destination></destination></destination></destination>

  


  

  • deprecated pixel format yuvy422p : video0 has mjpeg and raw output in pixel format yuvy422p.

  


  


  With h264 omx (works)

  


    libavutil      56. 22.100 / 56. 22.100&#xA;  libavcodec     58. 35.100 / 58. 35.100&#xA;  libavformat    58. 20.100 / 58. 20.100&#xA;  libavdevice    58.  5.100 / 58.  5.100&#xA;  libavfilter     7. 40.101 /  7. 40.101&#xA;  libavresample   4.  0.  0 /  4.  0.  0&#xA;  libswscale      5.  3.100 /  5.  3.100&#xA;  libswresample   3.  3.100 /  3.  3.100&#xA;  libpostproc    55.  3.100 / 55.  3.100&#xA;Input #0, lavfi, from &#x27;anullsrc&#x27;:&#xA;  Duration: N/A, start: 0.000000, bitrate: 705 kb/s&#xA;    Stream #0:0: Audio: pcm_u8, 44100 Hz, stereo, u8, 705 kb/s&#xA;[mjpeg @ 0x7c7320] EOI missing, emulating&#xA;Input #0, video4linux2,v4l2, from &#x27;/dev/video0&#x27;:&#xA;  Duration: N/A, start: 71.378330, bitrate: N/A&#xA;    Stream #0:0: Video: mjpeg, yuvj422p(pc, bt470bg/unknown/unknown), 1280x720, 30 fps, 30 tbr, 1000k tbn, 1000k tbc&#xA;Codec AVOption preset (Configuration preset) specified for output file #0 (<destination url="url">) has not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video streams) or that it is a private option of some encoder which was not actually used for any stream.&#xA;Stream mapping:&#xA;  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (h264_omx))&#xA;  Stream #0:0 -> #0:1 (pcm_u8 (native) -> aac (native))&#xA;Press [q] to stop, [?] for help&#xA;[swscaler @ 0x7e5180] deprecated pixel format used, make sure you did set range correctly&#xA;[h264_omx @ 0x7cb800] Using OMX.broadcom.video_encode&#xA;Output #0, flv, to &#x27;<destination url="url">&#x27;:&#xA;  Metadata:&#xA;    encoder         : Lavf58.20.100&#xA;    Stream #0:0: Video: h264 (h264_omx) ([7][0][0][0] / 0x0007), yuv420p(progressive), 1280x720, q=2-31, 3000 kb/s, 30 fps, 1k tbn, 30 tbc&#xA;    Metadata:&#xA;      encoder         : Lavc58.35.100 h264_omx&#xA;    Stream #0:1: Audio: aac (LC) ([10][0][0][0] / 0x000A), 44100 Hz, stereo, fltp, 128 kb/s&#xA;    Metadata:&#xA;      encoder         : Lavc58.35.100 aac&#xA;[flv @ 0x7ca190] Failed to update header with correct duration.rate=1438.3kbits/s speed=   1x&#xA;[flv @ 0x7ca190] Failed to update header with correct filesize.&#xA;frame= 2190 fps= 24 q=-0.0 Lsize=   16424kB time=00:01:33.20 bitrate=1443.6kbits/s speed=   1x&#xA;</destination></destination>

  


• videotoolbox transcoding from mjpeg to h264 slower that libx264

  16 juin 2021, par SolskGaer

  here are the command that I used to do the transcoding, however the videotoolbox one is slower than libx264, why is that happening ? Is there any mistake in my videotoolbox command ?

  


  ffmpeg -loglevel quiet -f mjpeg -r 20 -i pipe:0 -vf scale=500:-2 -vcodec libx264 -tune zerolatency -profile:v baseline -preset medium -r 20 -f h264 -

  


  ffmpeg -loglevel quiet -f mjpeg -r 20 -i pipe:0 -vf scale=500:-2 -vcodec h264_videotoolbox -profile:v baseline -realtime true -r 20 -f h264 -

  


  and although libx264 is a little bit faster, it is still not in realtime and there is notable latency, how can I do the transcoding in real time ? And I prefer h264_videotoolbox since it is CPU friendly

  


  the following is the stderr of ffmpeg

  


  for videotoolbox

  


  ffmpeg version 4.4 Copyright (c) 2000-2021 the FFmpeg developers
  built with Apple clang version 12.0.0 (clang-1200.0.32.29)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.4_1 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Input #0, mjpeg, from 'pipe:0':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 750x1334 [SAR 72:72 DAR 375:667], 20 tbr, 1200k tbn, 20 tbc
Please use -profile:a or -profile:v, -profile is ambiguous
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (h264_videotoolbox))
[swscaler @ 0x7fddd15f3000] deprecated pixel format used, make sure you did set range correctly
[h264_videotoolbox @ 0x7fddd200b600] Cannot use B-frames with baseline profile. Output will not contain B-frames.
[h264_videotoolbox @ 0x7fddd200b600] CABAC entropy requires 'main' or 'high' profile, but baseline was requested. Encode will not use CABAC entropy.
Output #0, h264, to 'pipe:':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #0:0: Video: h264, yuv420p(tv, bt470bg/unknown/unknown, progressive), 400x712 [SAR 1335:1334 DAR 375:667], q=2-31, 3000 kb/s, 20 fps, 20 tbn
    Metadata:
      encoder         : Lavc58.134.100 h264_videotoolbox
frame=    1 fps=0.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   0x    
frame=    7 fps=0.0 q=-0.0 size=     148kB time=00:00:00.20 bitrate=6057.2kbits/s speed=0.369x    
frame=   12 fps= 11 q=-0.0 size=     236kB time=00:00:00.45 bitrate=4305.3kbits/s speed=0.426x    
frame=   17 fps= 11 q=-0.0 size=     363kB time=00:00:00.70 bitrate=4243.6kbits/s speed=0.442x    
frame=   22 fps= 10 q=-0.0 size=     438kB time=00:00:00.95 bitrate=3778.8kbits/s speed=0.451x    
frame=   27 fps= 10 q=-0.0 size=     558kB time=00:00:01.20 bitrate=3809.1kbits/s speed=0.454x    
frame=   32 fps= 10 q=-0.0 size=     615kB time=00:00:01.45 bitrate=3476.4kbits/s speed=0.458x    
frame=   37 fps= 10 q=-0.0 size=     685kB time=00:00:01.70 bitrate=3300.9kbits/s speed=0.46x    
frame=   42 fps= 10 q=-0.0 size=     796kB time=00:00:01.95 bitrate=3343.7kbits/s speed=0.462x    
frame=   47 fps=9.9 q=-0.0 size=     878kB time=00:00:02.20 bitrate=3268.8kbits/s speed=0.464x    
frame=   52 fps=9.9 q=-0.0 size=    1002kB time=00:00:02.45 bitrate=3351.7kbits/s speed=0.465x    
frame=   57 fps=9.9 q=-0.0 size=    1076kB time=00:00:02.70 bitrate=3263.1kbits/s speed=0.467x    
frame=   62 fps=9.8 q=-0.0 size=    1168kB time=00:00:02.95 bitrate=3242.7kbits/s speed=0.468x    
frame=   67 fps=9.8 q=-0.0 size=    1281kB time=00:00:03.20 bitrate=3279.4kbits/s speed=0.468x    
frame=   72 fps=9.8 q=-0.0 size=    1366kB time=00:00:03.45 bitrate=3243.8kbits/s speed=0.469x    
frame=   77 fps=9.8 q=-0.0 size=    1488kB time=00:00:03.70 bitrate=3294.9kbits/s speed=0.47x    
frame=   82 fps=9.8 q=-0.0 size=    1560kB time=00:00:03.95 bitrate=3235.9kbits/s speed=0.47x    
frame=   87 fps=9.8 q=-0.0 size=    1686kB time=00:00:04.20 bitrate=3288.8kbits/s speed=0.471x    
frame=   92 fps=9.7 q=-0.0 size=    1750kB time=00:00:04.45 bitrate=3221.4kbits/s speed=0.471x    
frame=   97 fps=9.7 q=-0.0 size=    1825kB time=00:00:04.70 bitrate=3180.2kbits/s speed=0.472x    


  


  for libx264

  


  ffmpeg version 4.4 Copyright (c) 2000-2021 the FFmpeg developers
  built with Apple clang version 12.0.0 (clang-1200.0.32.29)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.4_1 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Input #0, mjpeg, from 'pipe:0':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 750x1334 [SAR 72:72 DAR 375:667], 20 tbr, 1200k tbn, 20 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
[swscaler @ 0x7fd1864fd000] deprecated pixel format used, make sure you did set range correctly
[libx264 @ 0x7fd187009400] using SAR=1335/1334
[libx264 @ 0x7fd187009400] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x7fd187009400] profile Constrained Baseline, level 3.0, 4:2:0, 8-bit
Output #0, h264, to 'pipe:':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #0:0: Video: h264, yuvj420p(pc, bt470bg/unknown/unknown, progressive), 400x712 [SAR 1335:1334 DAR 375:667], q=2-31, 20 fps, 20 tbn
    Metadata:
      encoder         : Lavc58.134.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame=    1 fps=0.0 q=20.0 size=      62kB time=00:00:00.05 bitrate=10091.5kbits/s speed=2.5e+04x    
frame=    7 fps=0.0 q=29.0 size=     168kB time=00:00:00.35 bitrate=3941.8kbits/s speed=0.644x    
frame=   12 fps= 11 q=29.0 size=     244kB time=00:00:00.60 bitrate=3329.8kbits/s speed=0.562x    
frame=   17 fps= 11 q=29.0 size=     303kB time=00:00:00.85 bitrate=2923.6kbits/s speed=0.535x    
frame=   22 fps= 10 q=29.0 size=     359kB time=00:00:01.10 bitrate=2677.2kbits/s speed=0.52x    
frame=   27 fps= 10 q=29.0 size=     433kB time=00:00:01.35 bitrate=2625.4kbits/s speed=0.511x    
frame=   32 fps= 10 q=29.0 size=     498kB time=00:00:01.60 bitrate=2552.0kbits/s speed=0.506x    
frame=   37 fps= 10 q=29.0 size=     570kB time=00:00:01.85 bitrate=2525.9kbits/s speed=0.502x    
frame=   42 fps= 10 q=29.0 size=     647kB time=00:00:02.10 bitrate=2522.2kbits/s speed=0.498x    
frame=   47 fps=9.9 q=29.0 size=     702kB time=00:00:02.35 bitrate=2447.2kbits/s speed=0.497x    
frame=   52 fps=9.9 q=28.0 size=     756kB time=00:00:02.60 bitrate=2381.6kbits/s speed=0.495x    
frame=   57 fps=9.9 q=29.0 size=     818kB time=00:00:02.85 bitrate=2350.2kbits/s speed=0.494x    
frame=   62 fps=9.8 q=28.0 size=     872kB time=00:00:03.10 bitrate=2304.9kbits/s speed=0.491x    
frame=   67 fps=9.8 q=29.0 size=     930kB time=00:00:03.35 bitrate=2275.0kbits/s speed=0.491x    
frame=   72 fps=9.8 q=29.0 size=     990kB time=00:00:03.60 bitrate=2253.4kbits/s speed=0.489x    
frame=   77 fps=9.8 q=28.0 size=    1051kB time=00:00:03.85 bitrate=2235.9kbits/s speed=0.488x    
frame=   82 fps=9.8 q=28.0 size=    1103kB time=00:00:04.10 bitrate=2204.4kbits/s speed=0.488x    
frame=   87 fps=9.7 q=28.0 size=    1153kB time=00:00:04.35 bitrate=2171.6kbits/s speed=0.487x    
frame=   92 fps=9.7 q=29.0 size=    1209kB time=00:00:04.60 bitrate=2153.8kbits/s speed=0.486x    
frame=   97 fps=9.7 q=29.0 size=    1279kB time=00:00:04.85 bitrate=2160.2kbits/s speed=0.485x    
frame=  102 fps=9.7 q=29.0 size=    1351kB time=00:00:05.10 bitrate=2170.8kbits/s speed=0.485x 


  


• FFMPEG Issue : Video breaks a lot hence the real time video gets distorted and gets a little delayed while streaming on a webpage from drone

  26 mars 2022, par ashiyaa nunhuck

  ****I am trying to detect a face from my drone camera in real time.The video streams successfully but it is delayed and breaks a lot. Is there any solution to this problem ? How can i have a smooth video streaming with little delay and no video breaking in order to succeed in detecting a face ? Your help will be much appreciated.
Also, this is printed while my code is running: :

  


  


  INFO:werkzeug:127.0.0.1 - - [27/May/2021 15:16:14] "GET
/video/streaming HTTP/1.1" 200 -
INFO:drone_face_recognition_and_tracking.controllers.server :'action' :
'command', 'cmd' : 'takeOff'
INFO:drone_face_recognition_and_tracking.models.manage_drone :'action' :
'send_command', 'command' : 'takeoff' [h264 @ 0x55aa924a2e40] error
while decoding MB 45 38, bytestream -6 [h264 @ 0x55aa924a2e40]
concealing 424 DC, 424 AC, 424 MV errors in I frame [h264 @
0x55aa922a9a00] concealing 687 DC, 687 AC, 687 MV errors in P frame
[h264 @ 0x55aa923f79c0] left block unavailable for requested intra
mode [h264 @ 0x55aa923f79c0] error while decoding MB 0 34, bytestream
1347 [h264 @ 0x55aa923f79c0] concealing 709 DC, 709 AC, 709 MV errors
in P frame INFO:werkzeug:127.0.0.1 - - [27/May/2021 15:16:17] "POST
/api/command/ HTTP/1.1" 200 - pipe:0 : corrupt decoded frame in stream
0
Last message repeated 2 times [h264 @ 0x55aa922a9a00] error while decoding MB 49 30, bytestream -6 [h264 @ 0x55aa922a9a00] concealing
900 DC, 900 AC, 900 MV errors in P frame pipe:0 : corrupt decoded frame
in stream 0
INFO:drone_face_recognition_and_tracking.models.manage_drone :'action' :
'receive_response', 'response' : b'ok'
INFO:drone_face_recognition_and_tracking.controllers.server :'action' :
'command', 'cmd' : 'faceDetectAndTrack' INFO:werkzeug:127.0.0.1 - -
[27/May/2021 15:16:21] "POST /api/command/ HTTP/1.1" 200 - [h264 @
0x55aa924a2e40] left block unavailable for requested intra4x4 mode -1
[h264 @ 0x55aa924a2e40] error while decoding MB 0 30, bytestream 1624
[h264 @ 0x55aa924a2e40] concealing 949 DC, 949 AC, 949 MV errors in I
frame pipe:0 : corrupt decoded frame in stream 0 [h264 @
0x55aa9244d400] left block unavailable for requested intra mode [h264
@ 0x55aa9244d400] error while decoding MB 0 12, bytestream 2936 [h264
@ 0x55aa9244d400] concealing 2029 DC, 2029 AC, 2029 MV errors in I
frame pipe:0 : corrupt decoded frame in stream 0 [h264 @
0x55aa924bf700] concealing 1632 DC, 1632 AC, 1632 MV errors in P frame
pipe:0 : corrupt decoded frame in stream 0 [h264 @ 0x55aa92414280]
concealing 1571 DC, 1571 AC, 1571 MV errors in P frame

  


  


  Here is my code :****

  


  import logging
import contextlib
import os
import socket
import subprocess
import threading
import time
import cv2 as cv
import numpy as np

from drone_face_recognition_and_tracking.models.base import Singleton

logger = logging.getLogger(__name__)

DEFAULT_DISTANCE = 0.30
DEFAULT_SPEED = 10
DEFAULT_DEGREE = 10

FRAME_X = int(320)  # try 640
FRAME_Y = int(240)  # try 480
FRAME_AREA = FRAME_X * FRAME_Y

FRAME_SIZE = FRAME_AREA * 3
FRAME_CENTER_X = FRAME_X / 2
FRAME_CENTER_Y = FRAME_Y / 2

CMD_FFMPEG = (f'ffmpeg -probesize 32 -hwaccel auto -hwaccel_device opencl -i pipe:0 '
              f'-pix_fmt bgr24 -s {FRAME_X}x{FRAME_Y} -f rawvideo pipe:1')

FACE_DETECT_XML_FILE = './drone_face_recognition_and_tracking/models/haarcascade_frontalface_default.xml'


def receive_video(stop_event, pipe_in, host_ip, video_port):
    with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock_video:
        sock_video.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        sock_video.settimeout(.5)
        sock_video.bind((host_ip, video_port))
        data = bytearray(2048)
        while not stop_event.is_set():
            try:
                size, addr = sock_video.recvfrom_into(data)
                # logger.info({'action': 'receive_video', 'data': data})
            except socket.timeout as ex:
                logger.warning({'action': 'receive_video', 'ex': ex})
                time.sleep(0.5)
                continue
            except socket.error as ex:
                logger.error({'action': 'receive_video', 'ex': ex})
                break

            try:
                pipe_in.write(data[:size])
                pipe_in.flush()
            except Exception as ex:
                logger.error({'action': 'receive_video', 'ex': ex})
                break


class Tello_Drone(metaclass=Singleton):
    def __init__(self, host_ip='192.168.10.2', host_port=8889,
                 drone_ip='192.168.10.1', drone_port=8889,
                 is_imperial=False, speed=DEFAULT_SPEED):
        self.host_ip = host_ip
        self.host_port = host_port
        self.drone_ip = drone_ip
        self.drone_port = drone_port
        self.drone_address = (drone_ip, drone_port)
        self.is_imperial = is_imperial
        self.speed = speed
        self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        self.socket.bind((self.host_ip, self.host_port))

        self.response = None
        self.stop_event = threading.Event()
        self._response_thread = threading.Thread(target=self.receive_response, args=(self.stop_event, ))
        self._response_thread.start()

        self.proc = subprocess.Popen(CMD_FFMPEG.split(' '),
                                     stdin=subprocess.PIPE,
                                     stdout=subprocess.PIPE)
        self.proc_stdin = self.proc.stdin
        self.proc_stdout = self.proc.stdout

        self.video_port = 11111

        self._receive_video_thread = threading.Thread(
            target=receive_video,
            args=(self.stop_event, self.proc_stdin,
                  self.host_ip, self.video_port,))
        self._receive_video_thread.start()

        self.face_cascade = cv.CascadeClassifier(FACE_DETECT_XML_FILE)
        self._is_enable_face_detect = False

        self.send_command('command')
        self.send_command('streamon')
        self.set_speed(self.speed)

    def receive_response(self, stop_event):
        while not stop_event.is_set():
            try:
                self.response, ip = self.socket.recvfrom(3000)
                logger.info({'action': 'receive_response',
                             'response': self.response})
            except socket.error as ex:
                logger.error({'action': 'receive_response',
                             'ex': ex})
                break

    def __dell__(self):
        self.stop()

    def stop(self):
        self.stop_event.set()
        retry = 0
        while self._response_thread.is_alive():
            time.sleep(0.3)
            if retry > 30:
                break
            retry += 1
        self.socket.close()
        os.kill(self.proc.pid, 9)

    def send_command(self, command):
        logger.info({'action': 'send_command', 'command': command})
        self.socket.sendto(command.encode('utf-8'), self.drone_address)

        retry = 0
        while self.response is None:
            time.sleep(0.3)
            if retry > 3:
                break
            retry += 1

        if self.response is None:
            response = None
        else:
            response = self.response.decode('utf-8')
        self.response = None
        return response

    def takeoff(self):
        return self.send_command('takeoff')

    def land(self):
        return self.send_command('land')

    def move(self, direction, distance):
        distance = float(distance)
        if self.is_imperial:
            distance = int(round(distance * 30.48))
        else:
            distance = int(round(distance * 100))
        return self.send_command(f'{direction} {distance}')

    def up(self, distance=DEFAULT_DISTANCE):
        return self.move('up', distance)

    def down(self, distance=DEFAULT_DISTANCE):
        return self.move('down', distance)

    def left(self, distance=DEFAULT_DISTANCE):
        return self.move('left', distance)

    def right(self, distance=DEFAULT_DISTANCE):
        return self.move('right', distance)

    def forward(self, distance=DEFAULT_DISTANCE):
        return self.move('forward', distance)

    def back(self, distance=DEFAULT_DISTANCE):
        return self.move('back', distance)

    def set_speed(self, speed):
        return self.send_command(f'speed {speed}')

    def clockwise(self, degree=DEFAULT_DEGREE):
        return self.send_command(f'cw {degree}')

    def counter_clockwise(self, degree=DEFAULT_DEGREE):
        return self.send_command(f'ccw {degree}')

    def video_binary_generator(self):
        while True:
            try:
                frame = self.proc_stdout.read(FRAME_SIZE)
            except Exception as ex:
                logger.error({'action': 'video_binary_generator', 'ex': ex})
                continue

            if not frame:
                continue

            frame = np.fromstring(frame, np.uint8).reshape(FRAME_Y, FRAME_X, 3)
            yield frame

    def enable_face_detect(self):
        self._is_enable_face_detect = True

    def disable_face_detect(self):
        self._is_enable_face_detect = False

    def video_jpeg_generator(self):
        for frame in self.video_binary_generator():
            if self._is_enable_face_detect:
                gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)
                faces = self.face_cascade.detectMultiScale(gray, 1.2, 4)
                for (x, y, w, h) in faces:
                    cv.rectangle(frame, (x, y), (x+w, y+h), (0, 0, 255), 2)
                    break

            _, jpeg = cv.imencode('.jpg', frame)
            jpeg_binary = jpeg.tobytes()
            yield jpeg_binary