Recherche avancée

Sur d’autres sites (7182)

• FFmpeg - RTMP streaming from NodeJS, stream is faster than realtime

  6 avril 2023, par Weston

  My goal is to render a canvas in Node, and stream that canvas to an RTMP server (Twitch ultimately, but for now I'm testing on a local RTMP server). The standard way to stream to RTMP seems to be ffmpeg, so I'm using that, spawned as a child process from within NodeJS.

  


  I've tried a bunch of different combinations of techniques and ffmpeg params to get a consistent framerate and a stream at "realtime" speed, but can't figure it out. Here's the paths I've gone down so far

  


  Render canvas and send input in continuous interval

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  '-re',
  '-framerate', String(.fps),
  '-r', String(fps),

  '-i', '-',
  
  '-vcodec', 'libx264',
  '-r', String(fps),
  '-s', '1920x1080',
  '-g:v', String(2*fps),
  '-c:a', 'aac',
  '-f', 'flv', 'rtmp://127.0.0.1/live'
]);
ffmpeg.stdout.pipe(process.stdout)
ffmpeg.stderr.pipe(process.stderr)


const send = () => {
  ctx.fillStyle = 'red'
  ctx.fillRect(0, 0, 1920, 1080);
  ctx.font = '100px Arial';
  ctx.fillStyle = 'black'
  ctx.fillText(new Date().toLocaleString(), 500, 500);
  ffmpeg.stdin.write(canvas.toBuffer())
  setImmediate(() => send())
}
send()


  


  Observations

  


  

  • Took about 35 seconds for the stream to actually start (I think because of ffmpeg needing some amount of time to analyze the input ?)

  


  • Frame rate extremely below what I set it to, and "speed" also very low, although I'm not 100% sure what this means. example log Frame=  906 fps=3.9 q=29.0 size= 311kB time=00:00:27.83 bitrate= 91.5kbits/s speed=0.119x

  


  • Stream behavior

    

    • Takes about a minute to load once opened in VLC

    


    • Timer on the stream starts about 1 minute behind real time, stays stuck on a single second for 30+ seconds, then shoots up a few seconds quickly, and gets stuck again

    


    


  


  


  I had a hunch here that at least some of the reason for the strange behavior was that rendering the canvas in the same loop that I send input to ffmpeg in was too slow to achieve 30 FPS.

  


  Render canvas in separate interval from ffmpeg input interval

  


  Only render canvas FPS-times per second

  


  Continue sending input to ffmpeg as fast as possible

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

let buffer = canvas.toBuffer();

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  ...same as before
]);

const render = () => {
  ctx.fillStyle = 'red'
  ctx.fillRect(0, 0, 1920, 1080);
  ctx.font = '100px Arial';
  ctx.fillStyle = 'black'
  ctx.fillText(new Date().toLocaleString(), 500, 500);
  buffer = canvas.toBuffer();
  setTimeout(() => render(), 1/fps)
}
render();

const send = () => {
  ffmpeg.stdin.write(buffer)
  setImmediate(() => send())
}
send()


  


  Observations

  


  

  • ffmpeg starts streaming almost immediately

  


  • fps starts out around 16, takes a couple seconds to hit 28, and then 30 more seconds to hit 30fps. speed much closer to 1x, but not quite all the way. example log frame=15421 fps= 30 q=29.0 size=    4502kB time=00:08:31.66 bitrate=  72.1kbits/s speed=0.994x      

  


  • Stream behavior

    

    • Takes about 5 seconds to load once opened in VLC

    


    • Timer stays stuck on the same second for multiple minutes

    


    


  


  


  My hunch here for the stream being stuck on 1 timestamp is that while ffmpeg is sending frames out at 30 frames per second, I'm sending it frames much quicker than that. So in the first 1st of a second of streaming

  


  

  1. Canvas renders with timestamp T 30 times

  


  2. send runs N times where N is likely way higher than 30, sending ffmpeg N frames with the current timestamp

  


  3. ffmpeg now has N frames with timestamp T on them, but can only send them out 30 per second, so it takes more than 1 second for the timestamp on the screen to change

  


  


  Only send ffmpeg a frame every 1/FPS second

  


  Same as before, but instead of sending ffmpeg frames as quickly as possible, only send it FPS frames every second.

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

let buffer = canvas.toBuffer();

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  ...same as before
]);

const render = () => {
  ...same as before
}
render();

const send = () => {
  ffmpeg.stdin.write(buffer)
  setTimeout(() => send(), 1/fps)
}
send()


  


  Observations

  


  

  • ffmpeg takes a few seconds to start streaming

  


  • fps starts out high, around 28, and over the next minute or so drops down to 16. Speed drops along with it. example log frame= 1329 fps= 16 q=29.0 size=     463kB time=00:00:41.93 bitrate=  90.5kbits/s speed= 0.5x

  


  • Stream behavior

    

    • Takes about 10 seconds to load once opened in VLC

    


    • Timer increases about twice as fast as expected, then gets hung on one second for a bit, and then starts increasing again at same rate

    


    


  


  


  ---

  


  I'll stop there, but tl ;dr I've tried a whole bunch of different combinations of -re, -framerate, -fps_mode, -r ffmpeg args, and some other techniques in the code like continuing to use setImmediate to send frames, but use a date comparison to actually send a frame at an FPS rate. I'm sure there's probably some fundamental video streaming knowledge I'm missing, so I'm just looking for any sort of guidance on how I can get my canvas to stream at a "realtime" speed, or whatever I could be missing here.

  


• FFmpeg - RTMP streaming from Node, stream is faster than realtime

  31 mars 2023, par Weston

  My goal is to render a canvas in Node, and stream that canvas to an RTMP server (Twitch ultimately, but testing on a local RTMP server). The standard way to stream to RTMP seems to be ffmpeg, so I'm using that, spawned as a child process from within node. I've tried a bunch of different combinations of techniques and ffmpeg params to get a consistent framerate and a stream at "realtime" speed, but can't figure it out. Here's the paths I've gone down so far

  


  Render canvas and send input in continuous interval

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  '-re',
  '-framerate', String(.fps),
  '-r', String(fps),

  '-i', '-',
  
  '-vcodec', 'libx264',
  '-r', String(fps),
  '-s', '1920x1080',
  '-g:v', String(2*fps),
  '-c:a', 'aac',
  '-f', 'flv', 'rtmp://127.0.0.1/live'
]);
ffmpeg.stdout.pipe(process.stdout)
ffmpeg.stderr.pipe(process.stderr)


const send = () => {
  ctx.fillStyle = 'red'
  ctx.fillRect(0, 0, 1920, 1080);
  ctx.font = '100px Arial';
  ctx.fillStyle = 'black'
  ctx.fillText(new Date().toLocaleString(), 500, 500);
  ffmpeg.stdin.write(canvas.toBuffer())
  setImmediate(() => send())
}
send()


  


  Observations

  


  

  • Took about 35 seconds for the stream to actually start (I think because of ffmpeg needing some amount of time to analyze the input ?)

  


  • Frame rate extremely below what I set it to, and "speed" also very low, although I'm not 100% sure what this means. example log Frame=  906 fps=3.9 q=29.0 size= 311kB time=00:00:27.83 bitrate= 91.5kbits/s speed=0.119x

  


  • Stream behavior

    

    • Takes about a minute to load once opened in VLC

    


    • Timer on the stream starts about 1 minute behind real time, stays stuck on a single second for 30+ seconds, then shoots up a few seconds quickly, and gets stuck again

    


    


  


  


  I had a hunch here that at least some of the reason for the strange behavior was that rendering the canvas in the same loop that I send input to ffmpeg in was too slow to achieve 30 FPS.

  


  Render canvas in separate interval from ffmpeg input interval

  


  Only render canvas FPS-times per second

  


  Continue sending input to ffmpeg as fast as possible

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

let buffer = canvas.toBuffer();

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  ...same as before
]);

const render = () => {
  ctx.fillStyle = 'red'
  ctx.fillRect(0, 0, 1920, 1080);
  ctx.font = '100px Arial';
  ctx.fillStyle = 'black'
  ctx.fillText(new Date().toLocaleString(), 500, 500);
  buffer = canvas.toBuffer();
  setTimeout(() => render(), 1/fps)
}
render();

const send = () => {
  ffmpeg.stdin.write(buffer)
  setImmediate(() => send())
}
send()


  


  Observations

  


  

  • ffmpeg starts streaming almost immediately

  


  • fps starts out around 16, takes a couple seconds to hit 28, and then 30 more seconds to hit 30fps. speed much closer to 1x, but not quite all the way. example log frame=15421 fps= 30 q=29.0 size=    4502kB time=00:08:31.66 bitrate=  72.1kbits/s speed=0.994x      

  


  • Stream behavior

    

    • Takes about 5 seconds to load once opened in VLC

    


    • Timer stays stuck on the same second for multiple minutes

    


    


  


  


  My hunch here for the stream being stuck on 1 timestamp is that while ffmpeg is sending frames out at 30 frames per second, I'm sending it frames much quicker than that. So in the first 1st of a second of streaming

  


  

  1. Canvas renders with timestamp T 30 times

  


  2. send runs N times where N is likely way higher than 30, sending ffmpeg N frames with the current timestamp

  


  3. ffmpeg now has N frames with timestamp T on them, but can only send them out 30 per second, so it takes more than 1 second for the timestamp on the screen to change

  


  


  Only send ffmpeg a frame every 1/FPS second

  


  Same as before, but instead of sending ffmpeg frames as quickly as possible, only send it FPS frames every second.

  


  import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

let buffer = canvas.toBuffer();

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  ...same as before
]);

const render = () => {
  ...same as before
}
render();

const send = () => {
  ffmpeg.stdin.write(buffer)
  setTimeout(() => send(), 1/fps)
}
send()


  


  Observations

  


  

  • ffmpeg takes a few seconds to start streaming

  


  • fps starts out high, around 28, and over the next minute or so drops down to 16. Speed drops along with it. example log frame= 1329 fps= 16 q=29.0 size=     463kB time=00:00:41.93 bitrate=  90.5kbits/s speed= 0.5x

  


  • Stream behavior

    

    • Takes about 10 seconds to load once opened in VLC

    


    • Timer increases about twice as fast as expected, then gets hung on one second for a bit, and then starts increasing again at same rate

    


    


  


  


  ---

  


  I'll stop there, but tl ;dr I've tried a whole bunch of different combinations of -re, -framerate, -fps_mode, -r ffmpeg args, and some other techniques in the code like continuing to use setImmediate to send frames, but use a date comparison to actually send a frame at an FPS rate. I'm sure there's probably some fundamental video streaming knowledge I'm missing, so I'm just looking for any sort of guidance on how I can get my canvas to stream at a "realtime" speed, or whatever I could be missing here.

  


• FFmpeg and QuickTime Player dimensions of video do not match [duplicate]

  31 mai 2020, par GRS

  I have a video.mp4, which I scaled using FFmpeg to get out.mp4. The new video has the following probe :

  



  {'streams': [{'index': 0,
   'codec_name': 'h264',
   'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10',
   'profile': 'High',
   'codec_type': 'video',
   'codec_time_base': '1/50',
   'codec_tag_string': 'avc1',
   'codec_tag': '0x31637661',
   'width': 886,
   'height': 1920,
   'coded_width': 896,
   'coded_height': 1920,
   'has_b_frames': 2,
   'sample_aspect_ratio': '5120:1329',
   'display_aspect_ratio': '16:9',
   'pix_fmt': 'yuv420p',
   'level': 40,
   'chroma_location': 'left',
   'refs': 1,
   'is_avc': 'true',
   'nal_length_size': '4',
   'r_frame_rate': '25/1',
   'avg_frame_rate': '25/1',
   'time_base': '1/12800',
   'start_pts': 0,
   'start_time': '0.000000',
   'duration_ts': 384512,
   'duration': '30.040000',
   'bit_rate': '490832',
   'bits_per_raw_sample': '8',
   'nb_frames': '751',
   'disposition': {'default': 1,
    'dub': 0,
    'original': 0,
    'comment': 0,
    'lyrics': 0,
    'karaoke': 0,
    'forced': 0,
    'hearing_impaired': 0,
    'visual_impaired': 0,
    'clean_effects': 0,
    'attached_pic': 0,
    'timed_thumbnails': 0},
   'tags': {'language': 'und', 'handler_name': 'VideoHandler'}}],
 'format': {'filename': 'out.mp4',
  'nb_streams': 1,
  'nb_programs': 0,
  'format_name': 'mov,mp4,m4a,3gp,3g2,mj2',
  'format_long_name': 'QuickTime / MOV',
  'start_time': '0.000000',
  'duration': '30.040000',
  'size': '1852948',
  'bit_rate': '493461',
  'probe_score': 100,
  'tags': {'major_brand': 'isom',
   'minor_version': '512',
   'compatible_brands': 'isomiso2avc1mp41',
   'encoder': 'Lavf58.20.100'}}}


  



  I am expecting my video player to say the video is 886 x 1920, however, it is 3413:1920.


  



  What could be the error ? I am using pix_frmt=yuva420p and codec=libx264 to create the video.

  



  The error is that my ascpect ratio remains 16:9, which is the original input video, when it should automatically change to 9:19. E.g. 3413 = 1920 * 16 / 9. So why isn't the ascpect ratio changed when the video is scaled ?