Recherche avancée

Sur d’autres sites (9326)

• ffmpeg : crop video into two grayscale sub-videos ; guarantee monotonical frames ; and get timestamps

  13 mars 2021, par lurix66

  The need

  


  Hello, I need to extract two regions of a .h264 video file via the crop filter into two files. The output videos need to be monochrome and extension .mp4. The encoding (or format ?) should guarantee that video frames are organized monotonically. Finally, I need to get the timestamps for both files (which I'd bet are the same timestamps that I would get from the input file, see below).

  


  In the end I will be happy to do everything in one command via an elegant one liner (via a complex filter I guess), but I start doing it in multiple steps to break it down in simpler problems.

  


  In this path I get into many difficulties and despite having searched in many places I don't seem to find solutions that work. Unfortunately I'm no expert of ffmpeg or video conversion, so the more I search, the more details I discover, the less I solve problems.

  


  Below you find some of my attempts to work with the following options :

  


  

  • -filter:v "crop=400:ih:260:0,format=gray" to do the crop and the monochrome conversion

  


  • -vf showinfo possibly combined with -vsync 0 or -copyts to get the timestamps via stderr redirection &> filename

  


  • -c:v mjpeg to force monotony of frames (are there other ways ?)

  


  


  1. cropping each region and obtaining monochrome videos

  


  $ ffmpeg -y -hide_banner -i inVideo.h264 -filter:v "crop=400:ih:260:0,format=gray" outL.mp4
$ ffmpeg -y -hide_banner -i inVideo.h264 -filter:v "crop=400:ih:1280:0,format=gray" outR.mp4


  


  The issue here is that in the output files the frames are not organized monotonically (I don't understand why ; how come would that make sense in any video format ? I can't say if that comes from the input file).

  


  EDIT. Maybe it is not frames, but packets, as returned by av .demux() method that are not monotonic (see below "instructions to reproduce...")

  


  I have got the advice to do a ffmpeg -i outL.mp4 outL.mjpeg after, but this produces two videos that look very pixellated (at least playing them with ffplay) despite being surprisingly 4x bigger than the input. Needless to say, I need both monotonic frames and lossless conversion.

  


  EDIT. I acknowledge the advice to specify -q:v 1 ; this fixes the pixellation effect but produces a file even bigger, 12x in size. Is it necessary ? (see below "instructions to reproduce...")

  


  2. getting the timestamps

  


  I found this piece of advice, but I don't want to generate hundreds of image files, so I tried the following :

  


  $ ffmpeg -y -hide_banner -i outL.mp4 -vf showinfo -vsync 0 &>tsL.txt
$ ffmpeg -y -hide_banner -i outR.mp4 -vf showinfo -vsync 0 &>tsR.txt


  


  The issue here is that I don't get any output because ffmpeg claims it needs an output file.

  


  The need to produce an output file, and the doubt that the timestamps could be lost in the previous conversions, leads me back to making a first attempt of a one liner, where I am testing also the -copyts option, and the forcing the encoding with -c:v mjpeg option as per the advice mentioned above (don't know if in the right position though)

  


  ffmpeg -y -hide_banner -i testTex2.h264 -copyts -filter:v "crop=400:ih:1280:0,format=gray" -vf showinfo -c:v mjpeg eyeL.mp4 &>tsL.txt


  


  This does not work because surprisingly the output .mp4 I get is the same as the input. If instead I put the -vf showinfo option just before the stderr redirection, I get no redirected output

  


  ffmpeg -y -hide_banner -i testTex2.h264 -copyts -filter:v "crop=400:ih:260:0,format=gray" -c:v mjpeg outR.mp4 -vf showinfo dummy.mp4 &>tsR.txt


  


  In this case I get the desired timestamps output (too much : I will need some solution to grab only the pts and pts_time data out of it) but I have to produce a big dummy file. The worst thing is anyway, that the mjpeg encoding produces a low resolution very pixellated video again

  


  I admit that the logic how to place the options and the output files on the command line is obscure to me. Possible combinations are many, and the more options I try the more complicated it gets, and I am not getting much closer to the solution.

  


  3. [EDIT] instructions how to reproduce this

  


  

  • get a .h264 video

  


  • turn it into .mp by ffmpeg command $ ffmpeg -i inVideo.h264 out.mp4

  


  • run the following python cell in a jupyter-notebook

  


  • see that the packets timestamps have diffs greater and less than zero

  


  


  %matplotlib inline
import av
import numpy as np
import matplotlib.pyplot as mpl

fname, ext="outL.direct", "mp4"

cont=av.open(f"{fname}.{ext}")
pk_pts=np.array([p.pts for p in cont.demux(video=0) if p.pts is not None])

cont=av.open(f"{fname}.{ext}")
fm_pts=np.array([f.pts for f in cont.decode(video=0) if f.pts is not None])

print(pk_pts.shape,fm_pts.shape)

mpl.subplot(211)
mpl.plot(np.diff(pk_pts))

mpl.subplot(212)
mpl.plot(np.diff(fm_pts))


  


  

  • finally create also the mjpeg encoded files in various ways, and check packets monotony with the same script (see also file size)

  


  


  $ ffmpeg -i inVideo.h264 out.mjpeg
$ ffmpeg -i inVideo.h264 -c:v mjpeg out.c_mjpeg.mp4
$ ffmpeg -i inVideo.h264 -c:v mjpeg -q:v 1 out.c_mjpeg_q1.mp4


  


  Finally, the question

  


  What is a working way / the right way to do it ?

  


  Any hints, even about single steps and how to rightly combine them will be appreciated. Also, I am not limited tio the command line, and I would be able to try some more programmatic solution in python (jupyter notebook) instead of the command line if someone points me in that direction.

  


• How do you use Node.js to stream an MP4 file with ffmpeg ?

  2 novembre 2016, par LaserJesus

  I’ve been trying to solve this problem for several days now and would really appreciate any help on the subject.

  I’m able to successfully stream an mp4 audio file stored on a Node.js server using fluent-ffmpeg by passing the location of the file as a string and transcoding it to mp3. If I create a file stream from the same file and pass that to fluent-ffmpeg instead it works for an mp3 input file, but not a mp4 file. In the case of the mp4 file no error is thrown and it claims the stream completed successfully, but nothing is playing in the browser. I’m guessing this has to do with the meta data being stored at the end of an mp4 file, but I don’t know how to code around this. This is the exact same file that works correctly when it’s location is passed to ffmpeg, rather than the stream. When I try and pass a stream to the mp4 file on s3, again no error is thrown, but nothing streams to the browser. This isn’t surprising as ffmpeg won’t work with the file locally as stream, so expecting it to handle the stream from s3 is wishful thinking.

  How can I stream the mp4 file from s3, without storing it locally as a file first ? How do I get ffmpeg to do this without transcoding the file too ? The following is the code I have at the moment which isn’t working. Note that it attempts to pass the s3 file as a stream to ffmpeg and it’s also transcoding it into an mp3, which I’d prefer not to do.

  .get(function(req,res) {
  
      aws.s3(s3Bucket).getFile(s3Path, function (err, result) {
  
          if (err) {
  
              return next(err);
  
          }
  
          var proc = new ffmpeg(result)
  
              .withAudioCodec('libmp3lame')
  
              .format('mp3')
  
              .on('error', function (err, stdout, stderr) {
  
                  console.log('an error happened: ' + err.message);
  
                  console.log('ffmpeg stdout: ' + stdout);
  
                  console.log('ffmpeg stderr: ' + stderr);
  
              })
  
              .on('end', function () {
  
                  console.log('Processing finished !');
  
              })
  
              .on('progress', function (progress) {
  
                  console.log('Processing: ' + progress.percent + '% done');
  
              })
  
              .pipe(res, {end: true});
  
      });
  
  });

  This is using the knox library when it calls aws.s3... I’ve also tried writing it using the standard aws sdk for Node.js, as shown below, but I get the same outcome as above.

  var AWS = require('aws-sdk');
  
  
  
  var s3 = new AWS.S3({
  
      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
  
      secretAccessKey: process.env.AWS_SECRET_KEY,
  
      region: process.env.AWS_REGION_ID
  
  });
  
  var fileStream = s3.getObject({
  
          Bucket: s3Bucket,
  
          Key: s3Key
  
      }).createReadStream();
  
  var proc = new ffmpeg(fileStream)
  
      .withAudioCodec('libmp3lame')
  
      .format('mp3')
  
      .on('error', function (err, stdout, stderr) {
  
          console.log('an error happened: ' + err.message);
  
          console.log('ffmpeg stdout: ' + stdout);
  
          console.log('ffmpeg stderr: ' + stderr);
  
      })
  
      .on('end', function () {
  
          console.log('Processing finished !');
  
      })
  
      .on('progress', function (progress) {
  
          console.log('Processing: ' + progress.percent + '% done');
  
      })
  
      .pipe(res, {end: true});

  =====================================

  Updated

  I placed an mp3 file in the same s3 bucket and the code I have here worked and was able to stream the file through to the browser without storing a local copy. So the streaming issues I face have something to do with the mp4/aac container/encoder format.

  I’m still interested in a way to bring the m4a file down from s3 to the Node.js server in it’s entirety, then pass it to ffmpeg for streaming without actually storing the file in the local file system.

  =====================================

  Updated Again

  I’ve managed to get the server streaming the file, as mp4, straight to the browser. This half answers my original question. My only issue now is that I have to download the file to a local store first, before I can stream it. I’d still like to find a way to stream from s3 without needing the temporary file.

  aws.s3(s3Bucket).getFile(s3Path, function(err, result){
  
      result.pipe(fs.createWriteStream(file_location));
  
      result.on('end', function() {
  
          console.log('File Downloaded!');
  
          var proc = new ffmpeg(file_location)
  
              .outputOptions(['-movflags isml+frag_keyframe'])
  
              .toFormat('mp4')
  
              .withAudioCodec('copy')
  
              .seekInput(offset)
  
              .on('error', function(err,stdout,stderr) {
  
                  console.log('an error happened: ' + err.message);
  
                  console.log('ffmpeg stdout: ' + stdout);
  
                  console.log('ffmpeg stderr: ' + stderr);
  
              })
  
              .on('end', function() {
  
                  console.log('Processing finished !');
  
              })
  
              .on('progress', function(progress) {
  
                  console.log('Processing: ' + progress.percent + '% done');
  
              })
  
              .pipe(res, {end: true});
  
      });
  
  });

  On the receiving side I just have the following javascript in an empty html page :

  window.AudioContext = window.AudioContext || window.webkitAudioContext;
  
  context = new AudioContext();
  
  
  
  function process(Data) {
  
      source = context.createBufferSource(); // Create Sound Source
  
      context.decodeAudioData(Data, function(buffer){
  
          source.buffer = buffer;
  
          source.connect(context.destination);
  
          source.start(context.currentTime);
  
      });
  
  };
  
  
  
  function loadSound() {
  
      var request = new XMLHttpRequest();
  
      request.open("GET", "/stream/", true);
  
      request.responseType = "arraybuffer";
  
  
  
      request.onload = function() {
  
          var Data = request.response;
  
          process(Data);
  
      };
  
  
  
      request.send();
  
  };
  
  
  
  loadSound()

  =====================================

  The Answer

  The code above under the title ’updated again’ will stream an mp4 file, from s3, via a Node.js server to a browser without using flash. It does require that the file be stored temporarily on the Node.js server so that the meta data in the file is moved from the end of the file to the front. In order to stream without storing the temporary file, you need to actual modify the file on S3 first and make this meta data change. If you have changed the file in this way on S3 then you can modify the code under the title ’updated again’ so that the result from S3 is piped straight into the ffmpeg constructor, rather than into a file stream on the Node.js server, then providing that file location to ffmepg, as the code does now. You can change the final ’pipe’ command to ’save(location)’ to get a version of the mp4 file locally with the meta data moved to the front. You can then upload that new version of the file to S3 and try out the end to end streaming. Personally I’m now going to create a task that modifies the files in this way as they are uploaded to s3 in the first place. This allows me to record and stream in mp4 without transcoding or storing a temp file on the Node.js server.

• How do you use Node.js to stream an MP4 file with ffmpeg ?

  27 avril 2023, par LaserJesus

  I've been trying to solve this problem for several days now and would really appreciate any help on the subject.

  



  I'm able to successfully stream an mp4 audio file stored on a Node.js server using fluent-ffmpeg by passing the location of the file as a string and transcoding it to mp3. If I create a file stream from the same file and pass that to fluent-ffmpeg instead it works for an mp3 input file, but not a mp4 file. In the case of the mp4 file no error is thrown and it claims the stream completed successfully, but nothing is playing in the browser. I'm guessing this has to do with the meta data being stored at the end of an mp4 file, but I don't know how to code around this. This is the exact same file that works correctly when it's location is passed to ffmpeg, rather than the stream. When I try and pass a stream to the mp4 file on s3, again no error is thrown, but nothing streams to the browser. This isn't surprising as ffmpeg won't work with the file locally as stream, so expecting it to handle the stream from s3 is wishful thinking.

  



  How can I stream the mp4 file from s3, without storing it locally as a file first ? How do I get ffmpeg to do this without transcoding the file too ? The following is the code I have at the moment which isn't working. Note that it attempts to pass the s3 file as a stream to ffmpeg and it's also transcoding it into an mp3, which I'd prefer not to do.

  



  .get(function(req,res) {
    aws.s3(s3Bucket).getFile(s3Path, function (err, result) {
        if (err) {
            return next(err);
        }
        var proc = new ffmpeg(result)
            .withAudioCodec('libmp3lame')
            .format('mp3')
            .on('error', function (err, stdout, stderr) {
                console.log('an error happened: ' + err.message);
                console.log('ffmpeg stdout: ' + stdout);
                console.log('ffmpeg stderr: ' + stderr);
            })
            .on('end', function () {
                console.log('Processing finished !');
            })
            .on('progress', function (progress) {
                console.log('Processing: ' + progress.percent + '% done');
            })
            .pipe(res, {end: true});
    });
});


  



  This is using the knox library when it calls aws.s3... I've also tried writing it using the standard aws sdk for Node.js, as shown below, but I get the same outcome as above.

  



  var AWS = require('aws-sdk');

var s3 = new AWS.S3({
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_KEY,
    region: process.env.AWS_REGION_ID
});
var fileStream = s3.getObject({
        Bucket: s3Bucket,
        Key: s3Key
    }).createReadStream();
var proc = new ffmpeg(fileStream)
    .withAudioCodec('libmp3lame')
    .format('mp3')
    .on('error', function (err, stdout, stderr) {
        console.log('an error happened: ' + err.message);
        console.log('ffmpeg stdout: ' + stdout);
        console.log('ffmpeg stderr: ' + stderr);
    })
    .on('end', function () {
        console.log('Processing finished !');
    })
    .on('progress', function (progress) {
        console.log('Processing: ' + progress.percent + '% done');
    })
    .pipe(res, {end: true});


  



  =====================================

  



  Updated

  



  I placed an mp3 file in the same s3 bucket and the code I have here worked and was able to stream the file through to the browser without storing a local copy. So the streaming issues I face have something to do with the mp4/aac container/encoder format.

  



  I'm still interested in a way to bring the m4a file down from s3 to the Node.js server in it's entirety, then pass it to ffmpeg for streaming without actually storing the file in the local file system.

  



  =====================================

  



  Updated Again

  



  I've managed to get the server streaming the file, as mp4, straight to the browser. This half answers my original question. My only issue now is that I have to download the file to a local store first, before I can stream it. I'd still like to find a way to stream from s3 without needing the temporary file.

  



  aws.s3(s3Bucket).getFile(s3Path, function(err, result){
    result.pipe(fs.createWriteStream(file_location));
    result.on('end', function() {
        console.log('File Downloaded!');
        var proc = new ffmpeg(file_location)
            .outputOptions(['-movflags isml+frag_keyframe'])
            .toFormat('mp4')
            .withAudioCodec('copy')
            .seekInput(offset)
            .on('error', function(err,stdout,stderr) {
                console.log('an error happened: ' + err.message);
                console.log('ffmpeg stdout: ' + stdout);
                console.log('ffmpeg stderr: ' + stderr);
            })
            .on('end', function() {
                console.log('Processing finished !');
            })
            .on('progress', function(progress) {
                console.log('Processing: ' + progress.percent + '% done');
            })
            .pipe(res, {end: true});
    });
});


  



  On the receiving side I just have the following javascript in an empty html page :

  



  window.AudioContext = window.AudioContext || window.webkitAudioContext;
context = new AudioContext();

function process(Data) {
    source = context.createBufferSource(); // Create Sound Source
    context.decodeAudioData(Data, function(buffer){
        source.buffer = buffer;
        source.connect(context.destination);
        source.start(context.currentTime);
    });
};

function loadSound() {
    var request = new XMLHttpRequest();
    request.open("GET", "/stream/", true);
    request.responseType = "arraybuffer";

    request.onload = function() {
        var Data = request.response;
        process(Data);
    };

    request.send();
};

loadSound()


  



  =====================================

  



  The Answer

  



  The code above under the title 'updated again' will stream an mp4 file, from s3, via a Node.js server to a browser without using flash. It does require that the file be stored temporarily on the Node.js server so that the meta data in the file is moved from the end of the file to the front. In order to stream without storing the temporary file, you need to actual modify the file on S3 first and make this meta data change. If you have changed the file in this way on S3 then you can modify the code under the title 'updated again' so that the result from S3 is piped straight into the ffmpeg constructor, rather than into a file stream on the Node.js server, then providing that file location to ffmepg, as the code does now. You can change the final 'pipe' command to 'save(location)' to get a version of the mp4 file locally with the meta data moved to the front. You can then upload that new version of the file to S3 and try out the end to end streaming. Personally I'm now going to create a task that modifies the files in this way as they are uploaded to s3 in the first place. This allows me to record and stream in mp4 without transcoding or storing a temp file on the Node.js server.