Recherche avancée

Sur d’autres sites (6648)

• FFmpeg filtergraph memory leak

  5 juillet 2017, par Leif Andersen

  I have an FFmpeg program that :

  1. Demuxes and decodes a video file.
  2. Passes it through a filtergraph
  3. encodes and muxes the new video.

  The filtergraph itself is rather complex, and can be run directly from the command line as such :

  ffmpeg -i demo.mp4 -filter_complex \
  
   "[audio3]atrim=end=30:start=10[audio2];\
  
    [video5]trim=end=30:start=10[video4];[audio2]anull[audio6];\
  
    [video4]scale=width=1920:height=1080[video7];[audio6]anull[audio8];\
  
    [video7]fps=fps=30[video9];[audio8]anull[audio10];\
  
    [video9]format=pix_fmts=yuv420p[video11];\
  
    [audio10]asetpts=expr=PTS-STARTPTS[audio12];\
  
    [video11]setpts=expr=PTS-STARTPTS[video13];\
  
    [audio15]concat=v=0:a=1:n=1[audio14];\
  
    [video17]concat=v=1:a=0:n=1[video16];\
  
    [audio12]afifo[audio15];[video13]fifo[video17];\
  
    [audio14]afifo[audio18];[video16]fifo[video19];\
  
    [audio18]anull[audio20];\
  
    [video19]pad=width=1920:height=1080[video21];\
  
    [audio20]anull[audio22];[video21]fps=fps=25[video23];\
  
    [audio22]aformat=sample_fmts=fltp:sample_rates=44100:channel_layouts=stereo[fa];\
  
    [video23]format=pix_fmts=yuv420p[fv];[0:a]afifo[audio3];\
  
    [0:v]fifo[video5]" \
  
  -map "[fv]" -map "[fa]" out.mp4

  I realize this is a massive filtergraph with a lot of no-op filters, it was autogenerated rather than being hand written. Here is a more cleaner version of the graph. (Its a graphviz file, you can run it in the command line or here.)

  Anyway, when I run the program that uses this filtergraph my memory usage spikes. I end up using about 7 GB of RAM for a 30 second clip. However, when I run the program using the ffmpeg command above, it peaks out at about 600 MB of RAM. This causes me to believe that the problem is not the ungodly size of the filtergraph, but a problem with how my program is using it.

  The program sets up the filtergraph (using av_filter_parse_ptr, giving the filtergraph string shown above), encoder, muxer, decoder, and demuxer, then spawns two threads, one that sends frames into the filtergraph, and one that receives them. The frame that sends them looks something like :

  void decode () {
  
      while(... more_frames ...) {
  
          AVFrame *frame = av_frame_alloc();
  
          ... fill next frame of stream ...
  
          av_buffersrc_write_frame(ctx, frame);
  
          av_frame_free(&frame);
  
      }
  
  }

  (I have elided the av_send_packet/av_receive_frame functions as they don’t seem to be leaking memory. I have also elided the process of flushing the buffersrc as that won’t happen until the end, and the memory spikes long before that.)

  And the encoder thread looks similar :

  void encode() {
  
      while(... nodes_in_graph ...) {
  
          AVFrame *frame = av_frame_alloc();
  
          av_buffersink_get_frame(ctx, frame);
  
          ... ensure frame actually was filled ...
  
          ... send frame to encoder ...
  
          av_frame_free(&frame);
  
      }
  
  }

  As with the decoder, I have elided the send_frame/receive_packet combo as they don’t seem to be leaking memory. Additionally I have elided the details of ensuring that the frame actually was filled. The code loops until the frame eventually does get filled.

  Every frame I allocate I fairly quickly deallocate. I additionally handled all of the error cases that the ffmpeg can give (Elided in the example).

  I have also tried having only one frame for the encoder and one for the decoder (and calling av_frame_unref in each iteration of the loop).

  Am I forgetting to free something, or am I just using the calls to libavfilter incorrectly such that it has to buffer all of the data ? I don’t think the leak is caused by the memory graph because running it from the command line doesn’t seem to cause the same memory explosion.

  FWIW, the actual code is here, although its written in Racket. I do have a minimal example that also seems to duplicate this behavior (modified from the doc/example/filtering_video.c file from the ffmpeg code :

  #include 
  
  
  
  #include <libavcodec></libavcodec>avcodec.h>
  
  #include <libavformat></libavformat>avformat.h>
  
  #include <libavfilter></libavfilter>avfiltergraph.h>
  
  #include <libavfilter></libavfilter>buffersink.h>
  
  #include <libavfilter></libavfilter>buffersrc.h>
  
  #include <libavutil></libavutil>opt.h>
  
  
  
  const char *filter_descr = "trim=start=10:end=30,scale=78:24,transpose=cclock";
  
  
  
  static AVFormatContext *fmt_ctx;
  
  static AVCodecContext *dec_ctx;
  
  AVFilterContext *buffersink_ctx;
  
  AVFilterContext *buffersrc_ctx;
  
  AVFilterGraph *filter_graph;
  
  static int video_stream_index = -1;
  
  static int64_t last_pts = AV_NOPTS_VALUE;
  
  
  
  static int open_input_file(const char *filename)
  
  {
  
      int ret;
  
      AVCodec *dec;
  
  
  
      if ((ret = avformat_open_input(&amp;fmt_ctx, filename, NULL, NULL)) &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot open input file\n");
  
          return ret;
  
      }
  
  
  
      if ((ret = avformat_find_stream_info(fmt_ctx, NULL)) &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot find stream information\n");
  
          return ret;
  
      }
  
  
  
      /* select the video stream */
  
      ret = av_find_best_stream(fmt_ctx, AVMEDIA_TYPE_VIDEO, -1, -1, &amp;dec, 0);
  
      if (ret &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot find a video stream in the input file\n");
  
          return ret;
  
      }
  
      video_stream_index = ret;
  
  
  
      /* create decoding context */
  
      dec_ctx = avcodec_alloc_context3(dec);
  
      if (!dec_ctx)
  
          return AVERROR(ENOMEM);
  
      avcodec_parameters_to_context(dec_ctx, fmt_ctx->streams[video_stream_index]->codecpar);
  
      av_opt_set_int(dec_ctx, "refcounted_frames", 1, 0);
  
  
  
      /* init the video decoder */
  
      if ((ret = avcodec_open2(dec_ctx, dec, NULL)) &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot open video decoder\n");
  
          return ret;
  
      }
  
  
  
      return 0;
  
  }
  
  
  
  static int init_filters(const char *filters_descr)
  
  {
  
      char args[512];
  
      int ret = 0;
  
      AVFilter *buffersrc  = avfilter_get_by_name("buffer");
  
      AVFilter *buffersink = avfilter_get_by_name("buffersink");
  
      AVFilterInOut *outputs = avfilter_inout_alloc();
  
      AVFilterInOut *inputs  = avfilter_inout_alloc();
  
      AVRational time_base = fmt_ctx->streams[video_stream_index]->time_base;
  
      enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_GRAY8, AV_PIX_FMT_NONE };
  
  
  
      filter_graph = avfilter_graph_alloc();
  
      if (!outputs || !inputs || !filter_graph) {
  
          ret = AVERROR(ENOMEM);
  
          goto end;
  
      }
  
  
  
      /* buffer video source: the decoded frames from the decoder will be inserted here. */
  
      snprintf(args, sizeof(args),
  
              "video_size=%dx%d:pix_fmt=%d:time_base=%d/%d:pixel_aspect=%d/%d",
  
              dec_ctx->width, dec_ctx->height, dec_ctx->pix_fmt,
  
              time_base.num, time_base.den,
  
              dec_ctx->sample_aspect_ratio.num, dec_ctx->sample_aspect_ratio.den);
  
  
  
      ret = avfilter_graph_create_filter(&amp;buffersrc_ctx, buffersrc, "in",
  
                                     args, NULL, filter_graph);
  
      if (ret &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot create buffer source\n");
  
          goto end;
  
      }
  
  
  
      /* buffer video sink: to terminate the filter chain. */
  
      ret = avfilter_graph_create_filter(&amp;buffersink_ctx, buffersink, "out",
  
                                         NULL, NULL, filter_graph);
  
      if (ret &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot create buffer sink\n");
  
          goto end;
  
      }
  
  
  
      ret = av_opt_set_int_list(buffersink_ctx, "pix_fmts", pix_fmts,
  
                                AV_PIX_FMT_NONE, AV_OPT_SEARCH_CHILDREN);
  
      if (ret &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot set output pixel format\n");
  
          goto end;
  
      }
  
  
  
      outputs->name       = av_strdup("in");
  
      outputs->filter_ctx = buffersrc_ctx;
  
      outputs->pad_idx    = 0;
  
      outputs->next       = NULL;
  
      inputs->name       = av_strdup("out");
  
      inputs->filter_ctx = buffersink_ctx;
  
      inputs->pad_idx    = 0;
  
      inputs->next       = NULL;
  
  
  
      if ((ret = avfilter_graph_parse_ptr(filter_graph, filters_descr,
  
                                      &amp;inputs, &amp;outputs, NULL)) &lt; 0)
  
          goto end;
  
  
  
      if ((ret = avfilter_graph_config(filter_graph, NULL)) &lt; 0)
  
          goto end;
  
  
  
  end:
  
      avfilter_inout_free(&amp;inputs);
  
      avfilter_inout_free(&amp;outputs);
  
  
  
      return ret;
  
  }
  
  
  
  int main(int argc, char **argv)
  
  {
  
      int ret;
  
      AVPacket packet;
  
      AVFrame *frame = av_frame_alloc();
  
      AVFrame *filt_frame = av_frame_alloc();
  
  
  
      if (!frame || !filt_frame) {
  
          perror("Could not allocate frame");
  
          exit(1);
  
      }
  
      if (argc != 2) {
  
          fprintf(stderr, "Usage: %s file\n", argv[0]);
  
          exit(1);
  
      }
  
  
  
      av_register_all();
  
      avfilter_register_all();
  
  
  
      if ((ret = open_input_file(argv[1])) &lt; 0)
  
          goto end;
  
      if ((ret = init_filters(filter_descr)) &lt; 0)
  
          goto end;
  
  
  
      /* read all packets */
  
      while (1) {
  
          if ((ret = av_read_frame(fmt_ctx, &amp;packet)) &lt; 0)
  
              break;
  
  
  
          if (packet.stream_index == video_stream_index) {
  
              ret = avcodec_send_packet(dec_ctx, &amp;packet);
  
              if (ret &lt; 0) {
  
                  av_log(NULL, AV_LOG_ERROR, "Error while sending a packet to the decoder\n");
  
                  break;
  
              }
  
  
  
              while (ret >= 0) {
  
                  ret = avcodec_receive_frame(dec_ctx, frame);
  
                  if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF) {
  
                      break;
  
                  } else if (ret &lt; 0) {
  
                      av_log(NULL, AV_LOG_ERROR, "Error while receiving a frame from the decoder\n");
  
                      goto end;
  
                  }
  
  
  
                  if (ret >= 0) {
  
                       frame->pts = av_frame_get_best_effort_timestamp(frame);
  
  
  
                      /* push the decoded frame into the filtergraph */
  
                      if (av_buffersrc_add_frame_flags(buffersrc_ctx, frame, AV_BUFFERSRC_FLAG_KEEP_REF) &lt; 0) {
  
                          av_log(NULL, AV_LOG_ERROR, "Error while feeding the filtergraph\n");
  
                          break;
  
                      }
  
  
  
                      /* pull filtered frames from the filtergraph */
  
                      while (1) {
  
                          ret = av_buffersink_get_frame(buffersink_ctx, filt_frame);
  
                          if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
  
                              break;
  
                          if (ret &lt; 0)
  
                              goto end;
  
                          av_frame_unref(filt_frame);
  
                      }
  
                      av_frame_unref(frame);
  
                  }
  
              }
  
          }
  
          av_packet_unref(&amp;packet);
  
      }
  
  end:
  
      avfilter_graph_free(&amp;filter_graph);
  
      avcodec_free_context(&amp;dec_ctx);
  
      avformat_close_input(&amp;fmt_ctx);
  
      av_frame_free(&amp;frame);
  
      av_frame_free(&amp;filt_frame);
  
  
  
      return ret;
  
  }

• ffmpeg 1FPS extraction call spits out infinite images until disk is bricked on poorly-encoded movie

  12 juillet 2017, par John Allard

  This is one of the oddest things I’ve ever seen while using ffmpeg.

  This is the scenario - we are getting video footage from a IONODE ION-E100 encoder, the footage is encoded in h264. I can view the footage fine in VLC, and I can copy it over to another video using ffmpeg and -c:v copy. The issue arises when I try and extract one FPS from the movie using the following command :

  ffmpeg -i testmovie.mp4 -r 1 -vf 640:-2 -q:v 18 -loglevel error /tmp/tmp/extraction_%04d.jpeg

  What this should do is go through the movie and extract one frame per second, then dump the frames to the files /tmp/tmp/extraction_0001.jpeg, /tmp/tmp/extraction_0002.jpeg, ..., /tmp/tmp/extraction_000X.jpeg. It should end when it reaches the end of the video.

  Now the odd thing that happens when I run this command is that it will run in a tight loop, writing tens or hundreds of thousands of images to the disk until the disk is full at which point the command fails. All of the extracted images are the exact same as one another.

  Here is the ffmpeg stats output on the video

  ffmpeg version 3.2.2 Copyright (c) 2000-2016 the FFmpeg developers
  
    built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
  
    configuration: --prefix=/usr/local/Cellar/ffmpeg/3.2.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-libmp3lame --enable-libx264 --enable-libxvid --enable-opencl --disable-lzma --enable-vda
  
    libavutil      55. 34.100 / 55. 34.100
  
    libavcodec     57. 64.101 / 57. 64.101
  
    libavformat    57. 56.100 / 57. 56.100
  
    libavdevice    57.  1.100 / 57.  1.100
  
    libavfilter     6. 65.100 /  6. 65.100
  
    libavresample   3.  1.  0 /  3.  1.  0
  
    libswscale      4.  2.100 /  4.  2.100
  
    libswresample   2.  3.100 /  2.  3.100
  
    libpostproc    54.  1.100 / 54.  1.100
  
  Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/john/ionode/test2.mp4':
  
    Metadata:
  
      major_brand     : isom
  
      minor_version   : 512
  
      compatible_brands: isomiso2avc1mp41
  
      title           : /videoinput_1:0/h264_1/media.stm
  
      encoder         : Lavf57.56.100
  
    Duration: 00:00:16.52, start: -9791.427750, bitrate: 2165 kb/s
  
      Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 2119 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
  
      Metadata:
  
        handler_name    : VideoHandler
  
      Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 8000 Hz, mono, fltp, 236 kb/s (default)
  
      Metadata:
  
        handler_name    : SoundHandler

  (notice the start: -9791.427750 value, what is this ?)

  Here are the encoding details on the footage :

  {
  
      "programs": [
  
  
  
      ],
  
      "streams": [
  
          {
  
              "index": 0,
  
              "codec_name": "h264",
  
              "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
  
              "profile": "High",
  
              "codec_type": "video",
  
              "codec_time_base": "1/50",
  
              "codec_tag_string": "avc1",
  
              "codec_tag": "0x31637661",
  
              "width": 1920,
  
              "height": 1080,
  
              "coded_width": 1920,
  
              "coded_height": 1080,
  
              "has_b_frames": 2,
  
              "sample_aspect_ratio": "1:1",
  
              "display_aspect_ratio": "16:9",
  
              "pix_fmt": "yuv420p",
  
              "level": 40,
  
              "chroma_location": "left",
  
              "refs": 1,
  
              "is_avc": "true",
  
              "nal_length_size": "4",
  
              "r_frame_rate": "25/1",
  
              "avg_frame_rate": "25/1",
  
              "time_base": "1/12800",
  
              "start_pts": 0,
  
              "start_time": "0.000000",
  
              "duration_ts": 211456,
  
              "duration": "16.520000",
  
              "bit_rate": "2119813",
  
              "bits_per_raw_sample": "8",
  
              "nb_frames": "413",
  
              "disposition": {
  
                  "default": 1,
  
                  "dub": 0,
  
                  "original": 0,
  
                  "comment": 0,
  
                  "lyrics": 0,
  
                  "karaoke": 0,
  
                  "forced": 0,
  
                  "hearing_impaired": 0,
  
                  "visual_impaired": 0,
  
                  "clean_effects": 0,
  
                  "attached_pic": 0,
  
                  "timed_thumbnails": 0
  
              },
  
              "tags": {
  
                  "language": "und",
  
                  "handler_name": "VideoHandler"
  
              }
  
          }
  
      ]
  
  }

  Here a link to the sample video to use : http://s000.tinyupload.com/?file_id=03169189167771012515
  (it’s a 5MB video file)

  The only clues I have as to what is actually going on is the error output
  from the command that grabs the video from the encoder :

  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823640, current: -630275919; changing to -629823639. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823639, current: -630274895; changing to -629823638. This may result in incorrect timestamps in the output file.
  
  frame=  124 fps= 21 q=28.0 size=      41kB time=00:00:02.60 bitrate= 128.8kbits/s dup=26 drop=4 spee[mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823638, current: -630273871; changing to -629823637. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823637, current: -630272847; changing to -629823636. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823636, current: -630271824; changing to -629823635. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823635, current: -630270800; changing to -629823634. This may result in incorrect timestamps in the output file.
  
  frame=  124 fps= 19 q=28.0 size=      43kB time=00:00:02.60 bitrate= 137.0kbits/s dup=26 drop=4 spee[mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823634, current: -630269776; changing to -629823633. This may result in incorrect timestamps in the output file.
  
  [aac @ 0x7faa1a832000] Queue input is backward in time
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823633, current: -630268752; changing to -629823632. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823632, current: -630267728; changing to -629823631. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823631, current: -630513833; changing to -629823630. This may result in incorrect timestamps in the output file.
  
  frame=  124 fps= 18 q=28.0 size=      46kB time=00:00:02.60 bitrate= 145.4kbits/s dup=26 drop=4 spee[mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823630, current: -630512810; changing to -629823629. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823629, current: -630511786; changing to -629823628. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823628, current: -630510762; changing to -629823627. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823627, current: -630509738; changing to -629823626. This may result in incorrect timestamps in the output file.
  
  frame=  124 fps= 17 q=28.0 size=      49kB time=00:00:02.60 bitrate= 153.6kbits/s dup=26 drop=4 spee[mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823626, current: -630508714; changing to -629823625. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823625, current: -630507690; changing to -629823624. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823624, current: -630506667; changing to -629823623. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823623, current: -630505643; changing to -629823622. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823622, current: -630504619; changing to -629823621. This may result in incorrect timestamps in the output file.
  
  frame=  124 fps= 16 q=28.0 size=      52kB time=00:00:02.60 bitrate= 163.5kbits/s dup=26 drop=4 spee[mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823621, current: -630503595; changing to -629823620. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823620, current: -630502571; changing to -629823619. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823619, current: -630501548; changing to -629823618. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823618, current: -630500524; changing to -629823617. This may result in incorrect timestamps in the output file.
  
  frame=  124 fps= 15 q=28.0 size=      54kB time=00:00:02.60 bitrate= 171.4kbits/s dup=26 drop=4 spee[mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823617, current: -630499500; changing to -629823616. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823616, current: -630498476; changing to -629823615. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823615, current: -630497452; changing to -629823614. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823614, current: -630496428; changing to -629823613. This may result in incorrect timestamps in the output file.
  
  frame=  124 fps= 14 q=28.0 size=      57kB time=00:00:02.60 bitrate= 179.3kbits/s dup=26 drop=4 spee[mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823613, current: -630495405; changing to -629823612. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823612, current: -630494381; changing to -629823611. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823611, current: -630493357; changing to -629823610. This may result in incorrect timestamps in the output file.
  
  frame=  124 fps= 13 q=28.0 size=      59kB time=00:00:02.60 bitrate= 185.6kbits/s dup=26 drop=4 spee[aac @ 0x7faa1a832000] Queue input is backward in time
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823610, current: -630492333; changing to -629823609. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823609, current: -630491309; changing to -629823608. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823608, current: -630736398; changing to -629823607. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823607, current: -630735375; changing to -629823606. This may result in incorrect timestamps in the output file.
  
  frame=  124 fps= 12 q=28.0 size=      61kB time=00:00:02.60 bitrate= 193.5kbits/s dup=26 drop=4 spee[mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823606, current: -630734351; changing to -629823605. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823605, current: -630733327; changing to -629823604. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823604, current: -630732303; changing to -629823603. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823603, current: -630731279; changing to -629823602. This may result in incorrect timestamps in the output file.
  
  [mp4 @ 0x7faa1a83ba00] Non-monotonous DTS in output stream 0:1; previous: -629823602, current: -630730256; changing to -629823601. This may result in incorrect timestamps in the output file.
  
  frame=  124 fps= 12 q=-1.0 Lsize=     507kB time=00:00:04.84 bitrate= 858.7kbits/s dup=26 drop=4 speed=0.453x

  So my question is, why does this video break ffmpeg ? Why does ffmpeg get caught up in a tight loop extracting images without ever making any progres towards the end of the video ?

• Video rotating to left by 90 degree when converted using ffmpeg

  15 juillet 2017, par Herdesh Verma

  I developed a below code :

  extern "C"
  
  {
  
  #include <libavutil></libavutil>imgutils.h>
  
  #include <libavutil></libavutil>opt.h>
  
  #include <libavcodec></libavcodec>avcodec.h>
  
  #include <libavutil></libavutil>mathematics.h>
  
  #include <libavutil></libavutil>samplefmt.h>
  
  #include <libavutil></libavutil>timestamp.h>
  
  #include <libavformat></libavformat>avformat.h>
  
  #include <libavfilter></libavfilter>avfiltergraph.h>
  
  #include <libswscale></libswscale>swscale.h>
  
  }
  
  #include 
  
  static AVFormatContext *fmt_ctx = NULL;
  
  
  
  static int frame_index = 0;
  
  
  
  static int j = 0, nbytes=0;
  
  uint8_t *video_outbuf = NULL;
  
  static AVPacket *pAVPacket=NULL;
  
  static int value=0;
  
  static AVFrame *pAVFrame=NULL;
  
  static AVFrame *outFrame=NULL;
  
  static AVStream *video_st=NULL;
  
  static AVFormatContext *outAVFormatContext=NULL;
  
  static AVCodec *outAVCodec=NULL;
  
  static AVOutputFormat *output_format=NULL;
  
  static AVCodecContext *video_dec_ctx = NULL, *audio_dec_ctx;
  
  static AVCodecContext *outAVCodecContext=NULL;
  
  static int width, height;
  
  static enum AVPixelFormat pix_fmt;
  
  static AVStream *video_stream = NULL, *audio_stream = NULL;
  
  static const char *src_filename = NULL;
  
  static const char *video_dst_filename = NULL;
  
  static const char *audio_dst_filename = NULL;
  
  static FILE *video_dst_file = NULL;
  
  static FILE *audio_dst_file = NULL;
  
  static uint8_t *video_dst_data[4] = {NULL};
  
  static int      video_dst_linesize[4];
  
  static int video_dst_bufsize;
  
  static int video_stream_idx = -1, audio_stream_idx = -1;
  
  static AVPacket *pkt=NULL;
  
  static AVPacket *pkt1=NULL;
  
  static AVFrame *frame = NULL;
  
  //static AVPacket pkt;
  
  static int video_frame_count = 0;
  
  static int audio_frame_count = 0;
  
  static int refcount = 0;
  
  AVCodec *codec;
  
  static struct SwsContext *sws_ctx;
  
  AVCodecContext *c= NULL;
  
   int i, out_size, size, x, y, outbuf_size;
  
   AVFrame *picture;
  
   uint8_t *outbuf, *picture_buf;
  
  int video_outbuf_size;
  
  int w, h;
  
  AVPixelFormat pixFmt;
  
  uint8_t *data[4];
  
  int linesize[4];
  
  
  
  static int open_codec_context(int *stream_idx,
  
                            AVCodecContext **dec_ctx, AVFormatContext 
  
  *fmt_ctx, enum AVMediaType type)
  
  {
  
  int ret, stream_index;
  
  AVStream *st;
  
  AVCodec *dec = NULL;
  
  AVDictionary *opts = NULL;
  
  ret = av_find_best_stream(fmt_ctx, type, -1, -1, NULL, 0);
  
  if (ret &lt; 0) {
  
      printf("Could not find %s stream in input file '%s'\n",
  
              av_get_media_type_string(type), src_filename);
  
      return ret;
  
  } else {
  
      stream_index = ret;
  
      st = fmt_ctx->streams[stream_index];
  
      /* find decoder for the stream */
  
      dec = avcodec_find_decoder(st->codecpar->codec_id);
  
      if (!dec) {
  
          printf("Failed to find %s codec\n",
  
                  av_get_media_type_string(type));
  
          return AVERROR(EINVAL);
  
      }
  
      /* Allocate a codec context for the decoder */
  
      *dec_ctx = avcodec_alloc_context3(dec);
  
      if (!*dec_ctx) {
  
          printf("Failed to allocate the %s codec context\n",
  
                  av_get_media_type_string(type));
  
          return AVERROR(ENOMEM);
  
      }
  
      /* Copy codec parameters from input stream to output codec context */
  
      if ((ret = avcodec_parameters_to_context(*dec_ctx, st->codecpar)) &lt; 0) {
  
          printf("Failed to copy %s codec parameters to decoder context\n",
  
                  av_get_media_type_string(type));
  
          return ret;
  
      }
  
      /* Init the decoders, with or without reference counting */
  
      av_dict_set(&amp;opts, "refcounted_frames", refcount ? "1" : "0", 0);
  
      if ((ret = avcodec_open2(*dec_ctx, dec, &amp;opts)) &lt; 0) {
  
          printf("Failed to open %s codec\n",
  
                  av_get_media_type_string(type));
  
          return ret;
  
      }
  
      *stream_idx = stream_index;
  
  }
  
  return 0;
  
  }
  
  
  
  
  
  
  
  int main (int argc, char **argv)
  
  {
  
   int ret = 0, got_frame;
  
  src_filename = argv[1];
  
  video_dst_filename = argv[2];
  
  audio_dst_filename = argv[3];
  
  av_register_all();
  
  avcodec_register_all();
  
  printf("Registered all\n");
  
  
  
  /* open input file, and allocate format context */
  
  if (avformat_open_input(&amp;fmt_ctx, src_filename, NULL, NULL) &lt; 0) {
  
      printf("Could not open source file %s\n", src_filename);
  
      exit(1);
  
  }
  
  
  
  /* retrieve stream information */
  
  if (avformat_find_stream_info(fmt_ctx, NULL) &lt; 0) {
  
      printf("Could not find stream information\n");
  
      exit(1);
  
  }
  
  
  
  if (open_codec_context(&amp;video_stream_idx, &amp;video_dec_ctx, fmt_ctx, 
  
  AVMEDIA_TYPE_VIDEO) >= 0) {
  
      video_stream = fmt_ctx->streams[video_stream_idx];
  
      avformat_alloc_output_context2(&amp;outAVFormatContext, NULL, NULL, 
  
  video_dst_filename);
  
      if (!outAVFormatContext)
  
      {
  
              printf("\n\nError : avformat_alloc_output_context2()");
  
              return -1;
  
      }
  
  }
  
  
  
  if (open_codec_context(&amp;audio_stream_idx, &amp;audio_dec_ctx, fmt_ctx, 
  
  AVMEDIA_TYPE_AUDIO) >= 0) {
  
      audio_stream = fmt_ctx->streams[audio_stream_idx];
  
      audio_dst_file = fopen(audio_dst_filename, "wb");
  
      if (!audio_dst_file) {
  
          printf("Could not open destination file %s\n", audio_dst_filename);
  
          ret = 1;
  
          goto end;
  
      }
  
  }
  
  /* dump input information to stderr */
  
  av_dump_format(fmt_ctx, 0, src_filename, 0);
  
  
  
  if (!audio_stream &amp;&amp; !video_stream) {
  
      printf("Could not find audio or video stream in the input, aborting\n");
  
      ret = 1;
  
      goto end;
  
  }
  
  
  
      output_format = av_guess_format(NULL, video_dst_filename, NULL);
  
      if( !output_format )
  
      {
  
       printf("\n\nError : av_guess_format()");
  
       return -1;
  
      }
  
  
  
      video_st = avformat_new_stream(outAVFormatContext ,NULL);
  
      if( !video_st )
  
      {
  
              printf("\n\nError : avformat_new_stream()");
  
        return -1;
  
      }
  
  
  
      outAVCodecContext = avcodec_alloc_context3(outAVCodec);
  
      if( !outAVCodecContext )
  
      {
  
        printf("\n\nError : avcodec_alloc_context3()");
  
        return -1;
  
      }
  
  
  
  
  
      outAVCodecContext = video_st->codec;
  
      outAVCodecContext->codec_id = AV_CODEC_ID_MPEG4;// AV_CODEC_ID_MPEG4; // 
  
   AV_CODEC_ID_H264 // AV_CODEC_ID_MPEG1VIDEO
  
      outAVCodecContext->codec_type = AVMEDIA_TYPE_VIDEO;
  
      outAVCodecContext->pix_fmt  = AV_PIX_FMT_YUV420P;
  
      outAVCodecContext->bit_rate = 400000; // 2500000
  
      outAVCodecContext->width = 1920;
  
      //outAVCodecContext->width = 500;
  
      outAVCodecContext->height = 1080;
  
      //outAVCodecContext->height = 500;
  
      outAVCodecContext->gop_size = 3;
  
      outAVCodecContext->max_b_frames = 2;
  
      outAVCodecContext->time_base.num = 1;
  
      outAVCodecContext->time_base.den = 30; // 15fps
  
  
  
      if (outAVCodecContext->codec_id == AV_CODEC_ID_H264)
  
      {
  
       av_opt_set(outAVCodecContext->priv_data, "preset", "slow", 0);
  
      }
  
  
  
      outAVCodec = avcodec_find_encoder(AV_CODEC_ID_MPEG4);
  
      if( !outAVCodec )
  
      {
  
       printf("\n\nError : avcodec_find_encoder()");
  
       return -1;
  
      }
  
  
  
      /* Some container formats (like MP4) require global headers to be 
  
   present
  
         Mark the encoder so that it behaves accordingly. */
  
  
  
      if ( outAVFormatContext->oformat->flags &amp; AVFMT_GLOBALHEADER)
  
      {
  
              outAVCodecContext->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
  
      }
  
  
  
      value = avcodec_open2(outAVCodecContext, outAVCodec, NULL);
  
      if( value &lt; 0)
  
      {
  
              printf("\n\nError : avcodec_open2()");
  
              return -1;
  
      }
  
  
  
  /* create empty video file */
  
      if ( !(outAVFormatContext->flags &amp; AVFMT_NOFILE) )
  
      {
  
       if( avio_open2(&amp;outAVFormatContext->pb , video_dst_filename, 
  
  AVIO_FLAG_WRITE ,NULL, NULL) &lt; 0 )
  
       {
  
        printf("\n\nError : avio_open2()");
  
       }
  
      }
  
  
  
      if(!outAVFormatContext->nb_streams)
  
      {
  
              printf("\n\nError : Output file dose not contain any stream");
  
        return -1;
  
      }
  
  
  
      /* imp: mp4 container or some advanced container file required header 
  
   information*/
  
      value = avformat_write_header(outAVFormatContext , NULL);
  
      if(value &lt; 0)
  
      {
  
              printf("\n\nError : avformat_write_header()");
  
              return -1;
  
      }
  
  
  
      printf("\n\nOutput file information :\n\n");
  
      av_dump_format(outAVFormatContext , 0 ,video_dst_filename ,1);
  
  
  
  
  
      int flag;
  
      int frameFinished;
  
  
  
  
  
      value = 0;
  
  
  
      pAVPacket = (AVPacket *)av_malloc(sizeof(AVPacket));
  
      av_init_packet(pAVPacket);
  
  
  
      pAVFrame = av_frame_alloc();
  
      if( !pAVFrame )
  
      {
  
       printf("\n\nError : av_frame_alloc()");
  
       return -1;
  
      }
  
  
  
      outFrame = av_frame_alloc();//Allocate an AVFrame and set its fields to 
  
   default values.
  
      if( !outFrame )
  
      {
  
       printf("\n\nError : av_frame_alloc()");
  
       return -1;
  
      }
  
  
  
      nbytes = av_image_get_buffer_size(outAVCodecContext-
  
   >pix_fmt,outAVCodecContext->width,outAVCodecContext->height,32);
  
      video_outbuf = (uint8_t*)av_malloc(nbytes);
  
      if( video_outbuf == NULL )
  
      {
  
      printf("\n\nError : av_malloc()");
  
      }
  
  
  
  
  
      value = av_image_fill_arrays( outFrame->data, outFrame->linesize, 
  
   video_outbuf , AV_PIX_FMT_YUV420P, outAVCodecContext-
  
   >width,outAVCodecContext->height,1 ); // returns : the size in bytes 
  
   required for src
  
      if(value &lt; 0)
  
      {
  
      printf("\n\nError : av_image_fill_arrays()");
  
      }
  
  
  
      SwsContext* swsCtx_ ;
  
  
  
      // Allocate and return swsContext.
  
      // a pointer to an allocated context, or NULL in case of error
  
      // Deprecated : Use sws_getCachedContext() instead.
  
      swsCtx_ = sws_getContext(video_dec_ctx->width,
  
                              video_dec_ctx->height,
  
                              video_dec_ctx->pix_fmt,
  
                              video_dec_ctx->width,
  
                              video_dec_ctx->height,
  
                              video_dec_ctx->pix_fmt,
  
                              SWS_BICUBIC, NULL, NULL, NULL);
  
  
  
  
  
      AVPacket outPacket;
  
  
  
      int got_picture;
  
  
  
      while( av_read_frame( fmt_ctx , pAVPacket ) >= 0 )
  
      {
  
              if(pAVPacket->stream_index == video_stream_idx)
  
              {
  
                      value = avcodec_decode_video2(video_dec_ctx , pAVFrame , 
  
    &amp;frameFinished , pAVPacket );
  
                      if( value &lt; 0)
  
                      {
  
                              printf("Error : avcodec_decode_video2()");
  
                      }
  
  
  
                      if(frameFinished)// Frame successfully decoded :)
  
                      {
  
                              sws_scale(swsCtx_, pAVFrame->data, pAVFrame-
  
   >linesize,0, video_dec_ctx->height, outFrame->data,outFrame->linesize);
  
  //                              sws_scale(swsCtx_, pAVFrame->data, pAVFrame-
  
  >linesize,0, video_dec_ctx->height, outFrame->data,outFrame->linesize);
  
                              av_init_packet(&amp;outPacket);
  
                              outPacket.data = NULL;    // packet data will be 
  
   allocated by the encoder
  
                              outPacket.size = 0;
  
  
  
                              avcodec_encode_video2(outAVCodecContext , 
  
   &amp;outPacket ,outFrame , &amp;got_picture);
  
  
  
                              if(got_picture)
  
                              {
  
                                      if(outPacket.pts != AV_NOPTS_VALUE)
  
                                              outPacket.pts = 
  
   av_rescale_q(outPacket.pts, video_st->codec->time_base, video_st-
  
   >time_base);
  
                                      if(outPacket.dts != AV_NOPTS_VALUE)
  
                                              outPacket.dts = 
  
   av_rescale_q(outPacket.dts, video_st->codec->time_base, video_st-
  
   >time_base);
  
  
  
                                      printf("Write frame %3d (size= %2d)\n", 
  
   j++, outPacket.size/1000);
  
                                      if(av_write_frame(outAVFormatContext , 
  
   &amp;outPacket) != 0)
  
                                      {
  
                                              printf("\n\nError : 
  
   av_write_frame()");
  
                                      }
  
  
  
                              av_packet_unref(&amp;outPacket);
  
                              } // got_picture
  
  
  
                      av_packet_unref(&amp;outPacket);
  
                      } // frameFinished
  
  
  
              }
  
      }// End of while-loop
  
  
  
      value = av_write_trailer(outAVFormatContext);
  
      if( value &lt; 0)
  
      {
  
              printf("\n\nError : av_write_trailer()");
  
      }
  
  
  
  
  
      //THIS WAS ADDED LATER
  
      av_free(video_outbuf);
  
  
  
  end:
  
      avcodec_free_context(&amp;video_dec_ctx);
  
      avcodec_free_context(&amp;audio_dec_ctx);
  
      avformat_close_input(&amp;fmt_ctx);
  
      if (video_dst_file)
  
          fclose(video_dst_file);
  
     if (audio_dst_file)
  
         fclose(audio_dst_file);
  
     //av_frame_free(&amp;frame);
  
     av_free(video_dst_data[0]);
  
     return ret &lt; 0;
  
  }

  Problem with above code is that it rotates a video to left by 90 degree.

  Snapshot of video given as input to above program

  Snapshot of output video. It is rotated by 90 degree to left.

  I compiled program using below command :

  g++ -D__STDC_CONSTANT_MACROS -Wall -g ScreenRecorder.cpp -I/home/harry/Documents/compressor/ffmpeg-3.3/ -I/root/android-ndk-r14b/platforms/android-21/arch-x86_64/usr/include/ -c -o ScreenRecorder.o -w

  And linked it using below command :

  g++ -Wall -g ScreenRecorder.o -I/home/harry/Documents/compressor/ffmpeg-3.3/ -I/root/android-ndk-r14b/platforms/android-21/arch-x86_64/usr/include/ -L/usr/lib64 -L/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7/ -L/home/harry/Documents/compressor/ffmpeg-3.3/ffmpeg-build -L/root/android-ndk-r14b/platforms/android-21/arch-x86_64/usr/lib64 -o ScreenRecorder.exe -lavformat -lavcodec -lavutil -lavdevice -lavfilter -lswscale -lx264 -lswresample -lm -lpthread -ldl -lstdc++ -lc -lrt

  Program is being run using below command :

  ./ScreenRecorder.exe vertical.MOV videoH.mp4 audioH.mp3

  Note :
  - Source video is taken from iphone and is of .mov format.
  - Output video is being stored in .mp4 file.

  Can anyone please tell me why it is rotating video by 90 degree ?

  One thing i noticed in dump is shown below :

    Duration: 00:00:06.04, start: 0.000000, bitrate: 17087 kb/s
  
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080, 17014 kb/s, 29.98 fps, 29.97 tbr, 600 tbn, 1200 tbc (default)
  
  Metadata:
  
    rotate          : 90
  
    creation_time   : 2017-07-09T10:56:42.000000Z
  
    handler_name    : Core Media Data Handler
  
    encoder         : H.264
  
  Side data:
  
    displaymatrix: rotation of -90.00 degrees

  it says displaymatrix: rotation of -90.00 degrees. Is it responsible for rotating video by 90 degree ?