Recherche avancée

Sur d’autres sites (5268)

• FFmpeg - RTMP streaming from NodeJS, stream is faster than realtime

  6 avril 2023, par Weston

  My goal is to render a canvas in Node, and stream that canvas to an RTMP server (Twitch ultimately, but for now I'm testing on a local RTMP server). The standard way to stream to RTMP seems to be ffmpeg, so I'm using that, spawned as a child process from within NodeJS.

  


  I've tried a bunch of different combinations of techniques and ffmpeg params to get a consistent framerate and a stream at "realtime" speed, but can't figure it out. Here's the paths I've gone down so far

  


  Render canvas and send input in continuous interval

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  '-re',
  '-framerate', String(.fps),
  '-r', String(fps),

  '-i', '-',
  
  '-vcodec', 'libx264',
  '-r', String(fps),
  '-s', '1920x1080',
  '-g:v', String(2*fps),
  '-c:a', 'aac',
  '-f', 'flv', 'rtmp://127.0.0.1/live'
]);
ffmpeg.stdout.pipe(process.stdout)
ffmpeg.stderr.pipe(process.stderr)


const send = () => {
  ctx.fillStyle = 'red'
  ctx.fillRect(0, 0, 1920, 1080);
  ctx.font = '100px Arial';
  ctx.fillStyle = 'black'
  ctx.fillText(new Date().toLocaleString(), 500, 500);
  ffmpeg.stdin.write(canvas.toBuffer())
  setImmediate(() => send())
}
send()


  


  Observations

  


  

  • Took about 35 seconds for the stream to actually start (I think because of ffmpeg needing some amount of time to analyze the input ?)

  


  • Frame rate extremely below what I set it to, and "speed" also very low, although I'm not 100% sure what this means. example log Frame=  906 fps=3.9 q=29.0 size= 311kB time=00:00:27.83 bitrate= 91.5kbits/s speed=0.119x

  


  • Stream behavior

    

    • Takes about a minute to load once opened in VLC

    


    • Timer on the stream starts about 1 minute behind real time, stays stuck on a single second for 30+ seconds, then shoots up a few seconds quickly, and gets stuck again

    


    


  


  


  I had a hunch here that at least some of the reason for the strange behavior was that rendering the canvas in the same loop that I send input to ffmpeg in was too slow to achieve 30 FPS.

  


  Render canvas in separate interval from ffmpeg input interval

  


  Only render canvas FPS-times per second

  


  Continue sending input to ffmpeg as fast as possible

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

let buffer = canvas.toBuffer();

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  ...same as before
]);

const render = () => {
  ctx.fillStyle = 'red'
  ctx.fillRect(0, 0, 1920, 1080);
  ctx.font = '100px Arial';
  ctx.fillStyle = 'black'
  ctx.fillText(new Date().toLocaleString(), 500, 500);
  buffer = canvas.toBuffer();
  setTimeout(() => render(), 1/fps)
}
render();

const send = () => {
  ffmpeg.stdin.write(buffer)
  setImmediate(() => send())
}
send()


  


  Observations

  


  

  • ffmpeg starts streaming almost immediately

  


  • fps starts out around 16, takes a couple seconds to hit 28, and then 30 more seconds to hit 30fps. speed much closer to 1x, but not quite all the way. example log frame=15421 fps= 30 q=29.0 size=    4502kB time=00:08:31.66 bitrate=  72.1kbits/s speed=0.994x      

  


  • Stream behavior

    

    • Takes about 5 seconds to load once opened in VLC

    


    • Timer stays stuck on the same second for multiple minutes

    


    


  


  


  My hunch here for the stream being stuck on 1 timestamp is that while ffmpeg is sending frames out at 30 frames per second, I'm sending it frames much quicker than that. So in the first 1st of a second of streaming

  


  

  1. Canvas renders with timestamp T 30 times

  


  2. send runs N times where N is likely way higher than 30, sending ffmpeg N frames with the current timestamp

  


  3. ffmpeg now has N frames with timestamp T on them, but can only send them out 30 per second, so it takes more than 1 second for the timestamp on the screen to change

  


  


  Only send ffmpeg a frame every 1/FPS second

  


  Same as before, but instead of sending ffmpeg frames as quickly as possible, only send it FPS frames every second.

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

let buffer = canvas.toBuffer();

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  ...same as before
]);

const render = () => {
  ...same as before
}
render();

const send = () => {
  ffmpeg.stdin.write(buffer)
  setTimeout(() => send(), 1/fps)
}
send()


  


  Observations

  


  

  • ffmpeg takes a few seconds to start streaming

  


  • fps starts out high, around 28, and over the next minute or so drops down to 16. Speed drops along with it. example log frame= 1329 fps= 16 q=29.0 size=     463kB time=00:00:41.93 bitrate=  90.5kbits/s speed= 0.5x

  


  • Stream behavior

    

    • Takes about 10 seconds to load once opened in VLC

    


    • Timer increases about twice as fast as expected, then gets hung on one second for a bit, and then starts increasing again at same rate

    


    


  


  


  ---

  


  I'll stop there, but tl ;dr I've tried a whole bunch of different combinations of -re, -framerate, -fps_mode, -r ffmpeg args, and some other techniques in the code like continuing to use setImmediate to send frames, but use a date comparison to actually send a frame at an FPS rate. I'm sure there's probably some fundamental video streaming knowledge I'm missing, so I'm just looking for any sort of guidance on how I can get my canvas to stream at a "realtime" speed, or whatever I could be missing here.

  


• Inconsistent frame number with ffmpeg

  21 avril 2023, par Ricky Bobby

  I'm having regularly issue with hvc1 videos getting an inconsistent number of frames between ffprobe info and FFmpeg info, and I would like to know what could be the reason for this issue and how if it's possible to solve it without re-encoding the video.

  


  I wrote the following sample script with a test video I have

  


  I split the video into 5-sec segments and I get ffprobe giving the expected video length but FFmpeg gave 3 frames less than expected on every segment but the first one.

  


  The issue is exactly the same if I split by 10 seconds or any split, I always lose 3 frames.

  


  I noted that the first segment is always 3 frames smaller (on ffprobe) than the other ones and it's the only consistent one.

  


  Here is an example script I wrote to test this issue :

  


  # get total video frame number using ffprobe or ffmpeg
total_num_frames=$(ffprobe -v quiet -show_entries stream=nb_read_packets -count_packets -select_streams v:0 -print_format json test_video.mp4 | jq '.streams[0].nb_read_packets' | tr -d '"')
echo $total_num_frames
ffmpeg -hwaccel cuda -i test_video.mp4 -vsync 2 -f null -

# Check ffprobe of each segment is consistent 
rm -rf clips && mkdir clips && \
ffmpeg -i test_video.mp4 -acodec copy -f segment -vcodec copy -reset_timestamps 1 -segment_time 5 -map 0 clips/part_%d.mp4
count_frames=0
for i in {0..5}
do
    num_packets=$(ffprobe -v quiet -show_entries stream=nb_read_packets -count_packets -select_streams v:0 -print_format json clips/part_$i.mp4 | jq '.streams[0].nb_read_packets' | tr -d '"')
    count_frames=$(($count_frames+$num_packets))
    echo $num_packets $count_frames $total_num_frames
done


  


  Output is the following

  


  3597
ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test_video.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf58.29.100
  Duration: 00:00:59.95, start: 0.035000, bitrate: 11797 kb/s
    Stream #0:0(und): Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv, bt709), 1920x1080, 11692 kb/s, 60.01 fps, 60 tbr, 19200 tbn, 19200 tbc (default)
    Metadata:
      handler_name    : Core Media Video
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, mono, fltp, 91 kb/s (default)
    Metadata:
      handler_name    : Core Media Audio
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf58.29.100
    Stream #0:0(und): Video: wrapped_avframe, nv12, 1920x1080, q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc (default)
    Metadata:
      handler_name    : Core Media Video
      encoder         : Lavc58.54.100 wrapped_avframe
    Stream #0:1(und): Audio: pcm_s16le, 44100 Hz, mono, s16, 705 kb/s (default)
    Metadata:
      handler_name    : Core Media Audio
      encoder         : Lavc58.54.100 pcm_s16le
frame= 3597 fps=788 q=-0.0 Lsize=N/A time=00:00:59.95 bitrate=N/A speed=13.1x    
video:1883kB audio:5162kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown


  


  then

  


  297 297 3597
300 597 3597
300 897 3597
300 1197 3597
300 1497 3597
300 1797 3597 <--- output are consistent based on ffprobe


  


  But then if i check segment size with ffmpeg with the following command

  


  ffmpeg -hwaccel cuda -i clips/part_$i.mp4 -vsync 2 -f null - 


  


  for part 0 its ok

  


  frame=  297 fps=0.0 q=-0.0 Lsize=N/A time=00:00:04.95 bitrate=N/A speed=12.5x    
video:155kB audio:424kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown


  


  for all other parts it's inconsistent and should be 300

  


  frame=  297 fps=0.0 q=-0.0 Lsize=N/A time=00:00:04.95 bitrate=N/A speed=12.3x    
video:155kB audio:423kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown


  


  The issue is exactly the same with any other interval size, e.g with 10 seconds I would get the following video size :

  


  ffprobe 597 - 600 ...
ffmpeg 597 597 ...


  


  I thought it could be related to source vfr or cfr but I tried to convert the input to cfr and nothing changed.

  


  Moreover, I tried to force the keyframe every second to check if it was a keyframe issue with the following arg : -force_key_frames "expr:gte(t,n_forced*1)", but the problem is exactly the same.

  


  What am I doing wrong ? it happens to me a lot with files in hvc1 and I really don't know how to deal with that.

  


• Scale with scale_vaapi and than pad, got error. Seems ok without "pad"

  24 mai 2023, par huangda1982

  My computer : customized Linux, haswell
I am trying to scale 1280x720 video to 640x450 in 640x480 frame with black borders. To do so, I run this command :

  


  ffmpeg -v warning -hide_banner -init_hw_device vaapi=vaapi0:/dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device vaapi0 -filter_hw_device vaapi0 -i 'my_video.mp4' -vf 'scale_vaapi=640:450,pad=640:480:0:14' -c:v h264_vaapi -q:v 18 -profile:v main -level 3.1 -c:a aac -q:a 3 -f mp4 out.mp4 -loglevel verbose


  


  And I got this error :

  


  Impossible to convert between the formats supported by the filter &#x27;Parsed_scale_vaapi_0&#x27; and the filter &#x27;auto_scaler_0&#x27;<br />&#xA;Error reinitializing filters!&#xA;Failed to inject frame into filter network: Function not implemented&#xA;Error while processing the decoded data for stream #0:0&#xA;

  


  When I removed the ",pad=640:480:0:14" part, the command continued. I pressed ctrl-c, and did not check the result.

  


  "-q:v 18", "-level 3.1" and "-q:a" options seem not work.

  


  Full message :

  


  [h264 @ 0x55dfeb529340] Reinit context to 1280x720, pix_fmt: yuv420p
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'my_video.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: mp41
    creation_time   : 1970-01-01T00:00:00.000000Z
    encoder         : PotEncoder
  Duration: 00:22:45.01, start: 0.000000, bitrate: 3264 kb/s
  Stream #0:0(und): Video: h264 (Constrained Baseline), 1 reference frame (avc1 / 0x31637661), yuv420p(left), 1280x720 [SAR 1:1 DAR 16:9], 3068 kb/s, 23.98 fps, 23.98 tbr, 1199 tbn, 47.96 tbc (default)
    Metadata:
      creation_time   : 1970-01-01T00:00:00.000000Z
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1(und): Audio: mp3 (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 192 kb/s (default)
    Metadata:
      creation_time   : 1970-01-01T00:00:00.000000Z
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
File 'out.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_vaapi))
  Stream #0:1 -> #0:1 (mp3 (mp3float) -> aac (native))
Press [q] to stop, [?] for help
[h264 @ 0x55dfeb553e40] Reinit context to 1280x720, pix_fmt: vaapi_vld
[graph_1_in_0_1 @ 0x55dfeb6b7580] tb:1/44100 samplefmt:fltp samplerate:44100 chlayout:0x3
[graph 0 input from stream 0:0 @ 0x55dfebb4d000] w:1280 h:720 pixfmt:vaapi_vld tb:1/1199 fr:1199/50 sar:1/1
[auto_scaler_0 @ 0x55dfebb4f940] w:iw h:ih flags:'bicubic' interl:0
[Parsed_pad_1 @ 0x55dfebb4c700] auto-inserting filter 'auto_scaler_0' between the filter 'Parsed_scale_vaapi_0' and the filter 'Parsed_pad_1'
Impossible to convert between the formats supported by the filter 'Parsed_scale_vaapi_0' and the filter 'auto_scaler_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
[AVIOContext @ 0x55dfeb5522c0] Statistics: 0 seeks, 0 writeouts
[aac @ 0x55dfeb551600] Qavg: 354.000
[aac @ 0x55dfeb551600] 2 frames left in the queue on closing
[AVIOContext @ 0x55dfeb530500] Statistics: 1144787 bytes read, 2 seeks
Conversion failed!