Recherche avancée

Sur d’autres sites (5361)

• FFmpeg - RTMP streaming from NodeJS, stream is faster than realtime

  6 avril 2023, par Weston

  My goal is to render a canvas in Node, and stream that canvas to an RTMP server (Twitch ultimately, but for now I'm testing on a local RTMP server). The standard way to stream to RTMP seems to be ffmpeg, so I'm using that, spawned as a child process from within NodeJS.

  


  I've tried a bunch of different combinations of techniques and ffmpeg params to get a consistent framerate and a stream at "realtime" speed, but can't figure it out. Here's the paths I've gone down so far

  


  Render canvas and send input in continuous interval

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  '-re',
  '-framerate', String(.fps),
  '-r', String(fps),

  '-i', '-',
  
  '-vcodec', 'libx264',
  '-r', String(fps),
  '-s', '1920x1080',
  '-g:v', String(2*fps),
  '-c:a', 'aac',
  '-f', 'flv', 'rtmp://127.0.0.1/live'
]);
ffmpeg.stdout.pipe(process.stdout)
ffmpeg.stderr.pipe(process.stderr)


const send = () => {
  ctx.fillStyle = 'red'
  ctx.fillRect(0, 0, 1920, 1080);
  ctx.font = '100px Arial';
  ctx.fillStyle = 'black'
  ctx.fillText(new Date().toLocaleString(), 500, 500);
  ffmpeg.stdin.write(canvas.toBuffer())
  setImmediate(() => send())
}
send()


  


  Observations

  


  

  • Took about 35 seconds for the stream to actually start (I think because of ffmpeg needing some amount of time to analyze the input ?)

  


  • Frame rate extremely below what I set it to, and "speed" also very low, although I'm not 100% sure what this means. example log Frame=  906 fps=3.9 q=29.0 size= 311kB time=00:00:27.83 bitrate= 91.5kbits/s speed=0.119x

  


  • Stream behavior

    

    • Takes about a minute to load once opened in VLC

    


    • Timer on the stream starts about 1 minute behind real time, stays stuck on a single second for 30+ seconds, then shoots up a few seconds quickly, and gets stuck again

    


    


  


  


  I had a hunch here that at least some of the reason for the strange behavior was that rendering the canvas in the same loop that I send input to ffmpeg in was too slow to achieve 30 FPS.

  


  Render canvas in separate interval from ffmpeg input interval

  


  Only render canvas FPS-times per second

  


  Continue sending input to ffmpeg as fast as possible

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

let buffer = canvas.toBuffer();

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  ...same as before
]);

const render = () => {
  ctx.fillStyle = 'red'
  ctx.fillRect(0, 0, 1920, 1080);
  ctx.font = '100px Arial';
  ctx.fillStyle = 'black'
  ctx.fillText(new Date().toLocaleString(), 500, 500);
  buffer = canvas.toBuffer();
  setTimeout(() => render(), 1/fps)
}
render();

const send = () => {
  ffmpeg.stdin.write(buffer)
  setImmediate(() => send())
}
send()


  


  Observations

  


  

  • ffmpeg starts streaming almost immediately

  


  • fps starts out around 16, takes a couple seconds to hit 28, and then 30 more seconds to hit 30fps. speed much closer to 1x, but not quite all the way. example log frame=15421 fps= 30 q=29.0 size=    4502kB time=00:08:31.66 bitrate=  72.1kbits/s speed=0.994x      

  


  • Stream behavior

    

    • Takes about 5 seconds to load once opened in VLC

    


    • Timer stays stuck on the same second for multiple minutes

    


    


  


  


  My hunch here for the stream being stuck on 1 timestamp is that while ffmpeg is sending frames out at 30 frames per second, I'm sending it frames much quicker than that. So in the first 1st of a second of streaming

  


  

  1. Canvas renders with timestamp T 30 times

  


  2. send runs N times where N is likely way higher than 30, sending ffmpeg N frames with the current timestamp

  


  3. ffmpeg now has N frames with timestamp T on them, but can only send them out 30 per second, so it takes more than 1 second for the timestamp on the screen to change

  


  


  Only send ffmpeg a frame every 1/FPS second

  


  Same as before, but instead of sending ffmpeg frames as quickly as possible, only send it FPS frames every second.

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

let buffer = canvas.toBuffer();

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  ...same as before
]);

const render = () => {
  ...same as before
}
render();

const send = () => {
  ffmpeg.stdin.write(buffer)
  setTimeout(() => send(), 1/fps)
}
send()


  


  Observations

  


  

  • ffmpeg takes a few seconds to start streaming

  


  • fps starts out high, around 28, and over the next minute or so drops down to 16. Speed drops along with it. example log frame= 1329 fps= 16 q=29.0 size=     463kB time=00:00:41.93 bitrate=  90.5kbits/s speed= 0.5x

  


  • Stream behavior

    

    • Takes about 10 seconds to load once opened in VLC

    


    • Timer increases about twice as fast as expected, then gets hung on one second for a bit, and then starts increasing again at same rate

    


    


  


  


  ---

  


  I'll stop there, but tl ;dr I've tried a whole bunch of different combinations of -re, -framerate, -fps_mode, -r ffmpeg args, and some other techniques in the code like continuing to use setImmediate to send frames, but use a date comparison to actually send a frame at an FPS rate. I'm sure there's probably some fundamental video streaming knowledge I'm missing, so I'm just looking for any sort of guidance on how I can get my canvas to stream at a "realtime" speed, or whatever I could be missing here.

  


• FFmpeg - RTMP streaming from Node, stream is faster than realtime

  31 mars 2023, par Weston

  My goal is to render a canvas in Node, and stream that canvas to an RTMP server (Twitch ultimately, but testing on a local RTMP server). The standard way to stream to RTMP seems to be ffmpeg, so I'm using that, spawned as a child process from within node. I've tried a bunch of different combinations of techniques and ffmpeg params to get a consistent framerate and a stream at "realtime" speed, but can't figure it out. Here's the paths I've gone down so far

  


  Render canvas and send input in continuous interval

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  '-re',
  '-framerate', String(.fps),
  '-r', String(fps),

  '-i', '-',
  
  '-vcodec', 'libx264',
  '-r', String(fps),
  '-s', '1920x1080',
  '-g:v', String(2*fps),
  '-c:a', 'aac',
  '-f', 'flv', 'rtmp://127.0.0.1/live'
]);
ffmpeg.stdout.pipe(process.stdout)
ffmpeg.stderr.pipe(process.stderr)


const send = () => {
  ctx.fillStyle = 'red'
  ctx.fillRect(0, 0, 1920, 1080);
  ctx.font = '100px Arial';
  ctx.fillStyle = 'black'
  ctx.fillText(new Date().toLocaleString(), 500, 500);
  ffmpeg.stdin.write(canvas.toBuffer())
  setImmediate(() => send())
}
send()


  


  Observations

  


  

  • Took about 35 seconds for the stream to actually start (I think because of ffmpeg needing some amount of time to analyze the input ?)

  


  • Frame rate extremely below what I set it to, and "speed" also very low, although I'm not 100% sure what this means. example log Frame=  906 fps=3.9 q=29.0 size= 311kB time=00:00:27.83 bitrate= 91.5kbits/s speed=0.119x

  


  • Stream behavior

    

    • Takes about a minute to load once opened in VLC

    


    • Timer on the stream starts about 1 minute behind real time, stays stuck on a single second for 30+ seconds, then shoots up a few seconds quickly, and gets stuck again

    


    


  


  


  I had a hunch here that at least some of the reason for the strange behavior was that rendering the canvas in the same loop that I send input to ffmpeg in was too slow to achieve 30 FPS.

  


  Render canvas in separate interval from ffmpeg input interval

  


  Only render canvas FPS-times per second

  


  Continue sending input to ffmpeg as fast as possible

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

let buffer = canvas.toBuffer();

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  ...same as before
]);

const render = () => {
  ctx.fillStyle = 'red'
  ctx.fillRect(0, 0, 1920, 1080);
  ctx.font = '100px Arial';
  ctx.fillStyle = 'black'
  ctx.fillText(new Date().toLocaleString(), 500, 500);
  buffer = canvas.toBuffer();
  setTimeout(() => render(), 1/fps)
}
render();

const send = () => {
  ffmpeg.stdin.write(buffer)
  setImmediate(() => send())
}
send()


  


  Observations

  


  

  • ffmpeg starts streaming almost immediately

  


  • fps starts out around 16, takes a couple seconds to hit 28, and then 30 more seconds to hit 30fps. speed much closer to 1x, but not quite all the way. example log frame=15421 fps= 30 q=29.0 size=    4502kB time=00:08:31.66 bitrate=  72.1kbits/s speed=0.994x      

  


  • Stream behavior

    

    • Takes about 5 seconds to load once opened in VLC

    


    • Timer stays stuck on the same second for multiple minutes

    


    


  


  


  My hunch here for the stream being stuck on 1 timestamp is that while ffmpeg is sending frames out at 30 frames per second, I'm sending it frames much quicker than that. So in the first 1st of a second of streaming

  


  

  1. Canvas renders with timestamp T 30 times

  


  2. send runs N times where N is likely way higher than 30, sending ffmpeg N frames with the current timestamp

  


  3. ffmpeg now has N frames with timestamp T on them, but can only send them out 30 per second, so it takes more than 1 second for the timestamp on the screen to change

  


  


  Only send ffmpeg a frame every 1/FPS second

  


  Same as before, but instead of sending ffmpeg frames as quickly as possible, only send it FPS frames every second.

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

let buffer = canvas.toBuffer();

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  ...same as before
]);

const render = () => {
  ...same as before
}
render();

const send = () => {
  ffmpeg.stdin.write(buffer)
  setTimeout(() => send(), 1/fps)
}
send()


  


  Observations

  


  

  • ffmpeg takes a few seconds to start streaming

  


  • fps starts out high, around 28, and over the next minute or so drops down to 16. Speed drops along with it. example log frame= 1329 fps= 16 q=29.0 size=     463kB time=00:00:41.93 bitrate=  90.5kbits/s speed= 0.5x

  


  • Stream behavior

    

    • Takes about 10 seconds to load once opened in VLC

    


    • Timer increases about twice as fast as expected, then gets hung on one second for a bit, and then starts increasing again at same rate

    


    


  


  


  ---

  


  I'll stop there, but tl ;dr I've tried a whole bunch of different combinations of -re, -framerate, -fps_mode, -r ffmpeg args, and some other techniques in the code like continuing to use setImmediate to send frames, but use a date comparison to actually send a frame at an FPS rate. I'm sure there's probably some fundamental video streaming knowledge I'm missing, so I'm just looking for any sort of guidance on how I can get my canvas to stream at a "realtime" speed, or whatever I could be missing here.

  


• Flash block/timeout handling improvements. Cleaner CSS, updated demo and main demos include handling now

  20 mars 2010, par Scott Schiller

  m demo/360-player/360player.css m demo/360-player/canvas-visualization-basic.html m demo/360-player/canvas-visualization.html m demo/360-player/index.html m demo/api/index.html + demo/flashblock/flashblock.css m demo/flashblock/index.html m demo/index.css m (...)