Recherche avancée

Sur d’autres sites (9003)

• FFmpeg filtergraph memory leak

  5 juillet 2017, par Leif Andersen

  I have an FFmpeg program that :

  1. Demuxes and decodes a video file.
  2. Passes it through a filtergraph
  3. encodes and muxes the new video.

  The filtergraph itself is rather complex, and can be run directly from the command line as such :

  ffmpeg -i demo.mp4 -filter_complex \
  
   "[audio3]atrim=end=30:start=10[audio2];\
  
    [video5]trim=end=30:start=10[video4];[audio2]anull[audio6];\
  
    [video4]scale=width=1920:height=1080[video7];[audio6]anull[audio8];\
  
    [video7]fps=fps=30[video9];[audio8]anull[audio10];\
  
    [video9]format=pix_fmts=yuv420p[video11];\
  
    [audio10]asetpts=expr=PTS-STARTPTS[audio12];\
  
    [video11]setpts=expr=PTS-STARTPTS[video13];\
  
    [audio15]concat=v=0:a=1:n=1[audio14];\
  
    [video17]concat=v=1:a=0:n=1[video16];\
  
    [audio12]afifo[audio15];[video13]fifo[video17];\
  
    [audio14]afifo[audio18];[video16]fifo[video19];\
  
    [audio18]anull[audio20];\
  
    [video19]pad=width=1920:height=1080[video21];\
  
    [audio20]anull[audio22];[video21]fps=fps=25[video23];\
  
    [audio22]aformat=sample_fmts=fltp:sample_rates=44100:channel_layouts=stereo[fa];\
  
    [video23]format=pix_fmts=yuv420p[fv];[0:a]afifo[audio3];\
  
    [0:v]fifo[video5]" \
  
  -map "[fv]" -map "[fa]" out.mp4

  I realize this is a massive filtergraph with a lot of no-op filters, it was autogenerated rather than being hand written. Here is a more cleaner version of the graph. (Its a graphviz file, you can run it in the command line or here.)

  Anyway, when I run the program that uses this filtergraph my memory usage spikes. I end up using about 7 GB of RAM for a 30 second clip. However, when I run the program using the ffmpeg command above, it peaks out at about 600 MB of RAM. This causes me to believe that the problem is not the ungodly size of the filtergraph, but a problem with how my program is using it.

  The program sets up the filtergraph (using av_filter_parse_ptr, giving the filtergraph string shown above), encoder, muxer, decoder, and demuxer, then spawns two threads, one that sends frames into the filtergraph, and one that receives them. The frame that sends them looks something like :

  void decode () {
  
      while(... more_frames ...) {
  
          AVFrame *frame = av_frame_alloc();
  
          ... fill next frame of stream ...
  
          av_buffersrc_write_frame(ctx, frame);
  
          av_frame_free(&frame);
  
      }
  
  }

  (I have elided the av_send_packet/av_receive_frame functions as they don’t seem to be leaking memory. I have also elided the process of flushing the buffersrc as that won’t happen until the end, and the memory spikes long before that.)

  And the encoder thread looks similar :

  void encode() {
  
      while(... nodes_in_graph ...) {
  
          AVFrame *frame = av_frame_alloc();
  
          av_buffersink_get_frame(ctx, frame);
  
          ... ensure frame actually was filled ...
  
          ... send frame to encoder ...
  
          av_frame_free(&frame);
  
      }
  
  }

  As with the decoder, I have elided the send_frame/receive_packet combo as they don’t seem to be leaking memory. Additionally I have elided the details of ensuring that the frame actually was filled. The code loops until the frame eventually does get filled.

  Every frame I allocate I fairly quickly deallocate. I additionally handled all of the error cases that the ffmpeg can give (Elided in the example).

  I have also tried having only one frame for the encoder and one for the decoder (and calling av_frame_unref in each iteration of the loop).

  Am I forgetting to free something, or am I just using the calls to libavfilter incorrectly such that it has to buffer all of the data ? I don’t think the leak is caused by the memory graph because running it from the command line doesn’t seem to cause the same memory explosion.

  FWIW, the actual code is here, although its written in Racket. I do have a minimal example that also seems to duplicate this behavior (modified from the doc/example/filtering_video.c file from the ffmpeg code :

  #include 
  
  
  
  #include <libavcodec></libavcodec>avcodec.h>
  
  #include <libavformat></libavformat>avformat.h>
  
  #include <libavfilter></libavfilter>avfiltergraph.h>
  
  #include <libavfilter></libavfilter>buffersink.h>
  
  #include <libavfilter></libavfilter>buffersrc.h>
  
  #include <libavutil></libavutil>opt.h>
  
  
  
  const char *filter_descr = "trim=start=10:end=30,scale=78:24,transpose=cclock";
  
  
  
  static AVFormatContext *fmt_ctx;
  
  static AVCodecContext *dec_ctx;
  
  AVFilterContext *buffersink_ctx;
  
  AVFilterContext *buffersrc_ctx;
  
  AVFilterGraph *filter_graph;
  
  static int video_stream_index = -1;
  
  static int64_t last_pts = AV_NOPTS_VALUE;
  
  
  
  static int open_input_file(const char *filename)
  
  {
  
      int ret;
  
      AVCodec *dec;
  
  
  
      if ((ret = avformat_open_input(&amp;fmt_ctx, filename, NULL, NULL)) &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot open input file\n");
  
          return ret;
  
      }
  
  
  
      if ((ret = avformat_find_stream_info(fmt_ctx, NULL)) &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot find stream information\n");
  
          return ret;
  
      }
  
  
  
      /* select the video stream */
  
      ret = av_find_best_stream(fmt_ctx, AVMEDIA_TYPE_VIDEO, -1, -1, &amp;dec, 0);
  
      if (ret &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot find a video stream in the input file\n");
  
          return ret;
  
      }
  
      video_stream_index = ret;
  
  
  
      /* create decoding context */
  
      dec_ctx = avcodec_alloc_context3(dec);
  
      if (!dec_ctx)
  
          return AVERROR(ENOMEM);
  
      avcodec_parameters_to_context(dec_ctx, fmt_ctx->streams[video_stream_index]->codecpar);
  
      av_opt_set_int(dec_ctx, "refcounted_frames", 1, 0);
  
  
  
      /* init the video decoder */
  
      if ((ret = avcodec_open2(dec_ctx, dec, NULL)) &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot open video decoder\n");
  
          return ret;
  
      }
  
  
  
      return 0;
  
  }
  
  
  
  static int init_filters(const char *filters_descr)
  
  {
  
      char args[512];
  
      int ret = 0;
  
      AVFilter *buffersrc  = avfilter_get_by_name("buffer");
  
      AVFilter *buffersink = avfilter_get_by_name("buffersink");
  
      AVFilterInOut *outputs = avfilter_inout_alloc();
  
      AVFilterInOut *inputs  = avfilter_inout_alloc();
  
      AVRational time_base = fmt_ctx->streams[video_stream_index]->time_base;
  
      enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_GRAY8, AV_PIX_FMT_NONE };
  
  
  
      filter_graph = avfilter_graph_alloc();
  
      if (!outputs || !inputs || !filter_graph) {
  
          ret = AVERROR(ENOMEM);
  
          goto end;
  
      }
  
  
  
      /* buffer video source: the decoded frames from the decoder will be inserted here. */
  
      snprintf(args, sizeof(args),
  
              "video_size=%dx%d:pix_fmt=%d:time_base=%d/%d:pixel_aspect=%d/%d",
  
              dec_ctx->width, dec_ctx->height, dec_ctx->pix_fmt,
  
              time_base.num, time_base.den,
  
              dec_ctx->sample_aspect_ratio.num, dec_ctx->sample_aspect_ratio.den);
  
  
  
      ret = avfilter_graph_create_filter(&amp;buffersrc_ctx, buffersrc, "in",
  
                                     args, NULL, filter_graph);
  
      if (ret &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot create buffer source\n");
  
          goto end;
  
      }
  
  
  
      /* buffer video sink: to terminate the filter chain. */
  
      ret = avfilter_graph_create_filter(&amp;buffersink_ctx, buffersink, "out",
  
                                         NULL, NULL, filter_graph);
  
      if (ret &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot create buffer sink\n");
  
          goto end;
  
      }
  
  
  
      ret = av_opt_set_int_list(buffersink_ctx, "pix_fmts", pix_fmts,
  
                                AV_PIX_FMT_NONE, AV_OPT_SEARCH_CHILDREN);
  
      if (ret &lt; 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot set output pixel format\n");
  
          goto end;
  
      }
  
  
  
      outputs->name       = av_strdup("in");
  
      outputs->filter_ctx = buffersrc_ctx;
  
      outputs->pad_idx    = 0;
  
      outputs->next       = NULL;
  
      inputs->name       = av_strdup("out");
  
      inputs->filter_ctx = buffersink_ctx;
  
      inputs->pad_idx    = 0;
  
      inputs->next       = NULL;
  
  
  
      if ((ret = avfilter_graph_parse_ptr(filter_graph, filters_descr,
  
                                      &amp;inputs, &amp;outputs, NULL)) &lt; 0)
  
          goto end;
  
  
  
      if ((ret = avfilter_graph_config(filter_graph, NULL)) &lt; 0)
  
          goto end;
  
  
  
  end:
  
      avfilter_inout_free(&amp;inputs);
  
      avfilter_inout_free(&amp;outputs);
  
  
  
      return ret;
  
  }
  
  
  
  int main(int argc, char **argv)
  
  {
  
      int ret;
  
      AVPacket packet;
  
      AVFrame *frame = av_frame_alloc();
  
      AVFrame *filt_frame = av_frame_alloc();
  
  
  
      if (!frame || !filt_frame) {
  
          perror("Could not allocate frame");
  
          exit(1);
  
      }
  
      if (argc != 2) {
  
          fprintf(stderr, "Usage: %s file\n", argv[0]);
  
          exit(1);
  
      }
  
  
  
      av_register_all();
  
      avfilter_register_all();
  
  
  
      if ((ret = open_input_file(argv[1])) &lt; 0)
  
          goto end;
  
      if ((ret = init_filters(filter_descr)) &lt; 0)
  
          goto end;
  
  
  
      /* read all packets */
  
      while (1) {
  
          if ((ret = av_read_frame(fmt_ctx, &amp;packet)) &lt; 0)
  
              break;
  
  
  
          if (packet.stream_index == video_stream_index) {
  
              ret = avcodec_send_packet(dec_ctx, &amp;packet);
  
              if (ret &lt; 0) {
  
                  av_log(NULL, AV_LOG_ERROR, "Error while sending a packet to the decoder\n");
  
                  break;
  
              }
  
  
  
              while (ret >= 0) {
  
                  ret = avcodec_receive_frame(dec_ctx, frame);
  
                  if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF) {
  
                      break;
  
                  } else if (ret &lt; 0) {
  
                      av_log(NULL, AV_LOG_ERROR, "Error while receiving a frame from the decoder\n");
  
                      goto end;
  
                  }
  
  
  
                  if (ret >= 0) {
  
                       frame->pts = av_frame_get_best_effort_timestamp(frame);
  
  
  
                      /* push the decoded frame into the filtergraph */
  
                      if (av_buffersrc_add_frame_flags(buffersrc_ctx, frame, AV_BUFFERSRC_FLAG_KEEP_REF) &lt; 0) {
  
                          av_log(NULL, AV_LOG_ERROR, "Error while feeding the filtergraph\n");
  
                          break;
  
                      }
  
  
  
                      /* pull filtered frames from the filtergraph */
  
                      while (1) {
  
                          ret = av_buffersink_get_frame(buffersink_ctx, filt_frame);
  
                          if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
  
                              break;
  
                          if (ret &lt; 0)
  
                              goto end;
  
                          av_frame_unref(filt_frame);
  
                      }
  
                      av_frame_unref(frame);
  
                  }
  
              }
  
          }
  
          av_packet_unref(&amp;packet);
  
      }
  
  end:
  
      avfilter_graph_free(&amp;filter_graph);
  
      avcodec_free_context(&amp;dec_ctx);
  
      avformat_close_input(&amp;fmt_ctx);
  
      av_frame_free(&amp;frame);
  
      av_frame_free(&amp;filt_frame);
  
  
  
      return ret;
  
  }

• cpp code to use ffmpeg static library, compress and convert from one format to another [on hold]

  3 juillet 2017, par Herdesh Verma

  I am new to ffmpeg library and don’t know much about ffmpeg library. I need to write a cpp code that could use static ffmpeg library and do compression along with conversion of format. Till now i managed to get below program working which can convert .mov file to .mpeg format and compress 125MB file to 18MB.

  extern "C"
  
  {
  
  #include <libavutil></libavutil>imgutils.h>
  
  #include <libavutil></libavutil>opt.h>
  
  #include <libavcodec></libavcodec>avcodec.h>
  
  #include <libavutil></libavutil>mathematics.h>
  
  #include <libavutil></libavutil>samplefmt.h>
  
  #include <libavutil></libavutil>timestamp.h>
  
  #include <libavformat></libavformat>avformat.h>
  
  #include <libavfilter></libavfilter>avfiltergraph.h>
  
  }
  
  #include 
  
  static AVFormatContext *fmt_ctx = NULL;
  
  static AVCodecContext *video_dec_ctx = NULL, *audio_dec_ctx;
  
  static int width, height;
  
  static enum AVPixelFormat pix_fmt;
  
  static AVStream *video_stream = NULL, *audio_stream = NULL;
  
  static const char *src_filename = NULL;
  
  static const char *video_dst_filename = NULL;
  
  static const char *audio_dst_filename = NULL;
  
  static FILE *video_dst_file = NULL;
  
  static FILE *audio_dst_file = NULL;
  
  static uint8_t *video_dst_data[4] = {NULL};
  
  static int      video_dst_linesize[4];
  
  static int video_dst_bufsize;
  
  static int video_stream_idx = -1, audio_stream_idx = -1;
  
  static AVPacket *pkt=NULL;
  
  static AVPacket *pkt1=NULL;
  
  static AVFrame *frame = NULL;
  
  //static AVPacket pkt;
  
  static int video_frame_count = 0;
  
  static int audio_frame_count = 0;
  
  static int refcount = 0;
  
  AVCodec *codec;
  
   AVCodecContext *c= NULL;
  
   int i, out_size, size, x, y, outbuf_size;
  
   AVFrame *picture;
  
   uint8_t *outbuf, *picture_buf;
  
  
  
  static void encode_packet(int *got_frame)
  
  {
  
  int ret, got_output;
  
  /* send the frame to the encoder */
  
  if (frame)
  
  frame->pts=video_dec_ctx->frame_number;
  
      printf("Send frame %3"PRId64"\n", frame->pts);
  
  ret = avcodec_encode_video2(c, pkt1, frame, got_frame);
  
  printf("ret=%d\n", ret);
  
  if(ret &lt; 0)
  
  {
  
      printf("Error while encoding frame\n");
  
      exit(1);
  
  }
  
  if (*got_frame) {
  
           printf("Write frame %3d (size=%5d)\n", i, pkt1->size);
  
           fwrite(pkt1->data, 1, pkt1->size, video_dst_file);
  
          av_free_packet(pkt1);
  
      }
  
  }
  
  
  
  static int decode_packet(int *got_frame, int cached)
  
  {
  
  AVPacket pkt1;
  
  int ret = 0;
  
  int decoded = pkt->size;
  
  *got_frame = 0;
  
  if (pkt->stream_index == video_stream_idx) {
  
      /* decode video frame */
  
      ret = avcodec_decode_video2(video_dec_ctx, frame, got_frame, pkt);
  
  printf("ret=%d and got_frame=%d\n", ret, *got_frame);
  
  
  
      if (ret &lt; 0) {
  
          printf("Error decoding video frame (%s)\n", av_err2str(ret));
  
          return ret;
  
      }
  
      if (*got_frame) {
  
          if (frame->width != width || frame->height != height ||
  
              frame->format != pix_fmt) {
  
              /* To handle this change, one could call av_image_alloc again 
  
  and
  
               * decode the following frames into another rawvideo file. */
  
              return -1;
  
          }
  
          printf("video_frame%s n:%d coded_n:%d\n",
  
                 cached ? "(cached)" : "",
  
                 video_frame_count++, frame->coded_picture_number);
  
  }
  
  encode_packet(got_frame);
  
  } 
  
  else if (pkt->stream_index == audio_stream_idx) {
  
      ret = avcodec_decode_audio4(audio_dec_ctx, frame, got_frame, pkt);
  
      if (ret &lt; 0) {
  
          printf("Error decoding audio frame (%s)\n", av_err2str(ret));
  
          return ret;
  
      }
  
      /* Some audio decoders decode only part of the packet, and have to be
  
       * called again with the remainder of the packet data.
  
       * Sample: fate-suite/lossless-audio/luckynight-partial.shn
  
       * Also, some decoders might over-read the packet. */
  
      decoded = FFMIN(ret, pkt->size);
  
      if (*got_frame) {
  
          size_t unpadded_linesize = frame->nb_samples * 
  
  av_get_bytes_per_sample((AVSampleFormat)frame->format);
  
          printf("audio_frame%s n:%d nb_samples:%d pts:%s\n",
  
                 cached ? "(cached)" : "",
  
                 audio_frame_count++, frame->nb_samples,
  
                 av_ts2timestr(frame->pts, &amp;audio_dec_ctx->time_base));
  
          /* Write the raw audio data samples of the first plane. This works
  
           * fine for packed formats (e.g. AV_SAMPLE_FMT_S16). However,
  
           * most audio decoders output planar audio, which uses a separate
  
           * plane of audio samples for each channel (e.g. 
  
  AV_SAMPLE_FMT_S16P).
  
           * In other words, this code will write only the first audio channel
  
           * in these cases.
  
           * You should use libswresample or libavfilter to convert the frame
  
           * to packed data. */
  
          fwrite(frame->extended_data[0], 1, unpadded_linesize, 
  
  audio_dst_file);
  
      }
  
  }
  
  /* If we use frame reference counting, we own the data and need
  
   * to de-reference it when we don't use it anymore */
  
  if (*got_frame &amp;&amp; refcount)
  
      av_frame_unref(frame);
  
  return decoded;
  
  }
  
  
  
  static int open_codec_context(int *stream_idx,
  
                            AVCodecContext **dec_ctx, AVFormatContext 
  
  *fmt_ctx, enum AVMediaType type)
  
  {
  
  int ret, stream_index;
  
  AVStream *st;
  
  AVCodec *dec = NULL;
  
  AVDictionary *opts = NULL;
  
  ret = av_find_best_stream(fmt_ctx, type, -1, -1, NULL, 0);
  
  if (ret &lt; 0) {
  
      printf("Could not find %s stream in input file '%s'\n",
  
              av_get_media_type_string(type), src_filename);
  
      return ret;
  
  } else {
  
      stream_index = ret;
  
      st = fmt_ctx->streams[stream_index];
  
      /* find decoder for the stream */
  
      dec = avcodec_find_decoder(st->codecpar->codec_id);
  
      if (!dec) {
  
          printf("Failed to find %s codec\n",
  
                  av_get_media_type_string(type));
  
          return AVERROR(EINVAL);
  
      }
  
      /* Allocate a codec context for the decoder */
  
      *dec_ctx = avcodec_alloc_context3(dec);
  
      if (!*dec_ctx) {
  
          printf("Failed to allocate the %s codec context\n",
  
                  av_get_media_type_string(type));
  
          return AVERROR(ENOMEM);
  
      }
  
      /* Copy codec parameters from input stream to output codec context */
  
      if ((ret = avcodec_parameters_to_context(*dec_ctx, st->codecpar)) &lt; 0) {
  
          printf("Failed to copy %s codec parameters to decoder context\n",
  
                  av_get_media_type_string(type));
  
          return ret;
  
      }
  
      /* Init the decoders, with or without reference counting */
  
      av_dict_set(&amp;opts, "refcounted_frames", refcount ? "1" : "0", 0);
  
      if ((ret = avcodec_open2(*dec_ctx, dec, &amp;opts)) &lt; 0) {
  
          printf("Failed to open %s codec\n",
  
                  av_get_media_type_string(type));
  
          return ret;
  
      }
  
      *stream_idx = stream_index;
  
  }
  
  return 0;
  
  }
  
  
  
  
  
  
  
  int main (int argc, char **argv)
  
  {
  
  int ret = 0, got_frame;
  
  src_filename = argv[1];
  
  video_dst_filename = argv[2];
  
  audio_dst_filename = argv[3];
  
  av_register_all();
  
  avcodec_register_all();
  
  printf("Registered all\n");
  
  
  
  /* open input file, and allocate format context */
  
  if (avformat_open_input(&amp;fmt_ctx, src_filename, NULL, NULL) &lt; 0) {
  
      printf("Could not open source file %s\n", src_filename);
  
      exit(1);
  
  }
  
  
  
  /* retrieve stream information */
  
  if (avformat_find_stream_info(fmt_ctx, NULL) &lt; 0) {
  
      printf("Could not find stream information\n");
  
      exit(1);
  
  }
  
  
  
  if (open_codec_context(&amp;video_stream_idx, &amp;video_dec_ctx, fmt_ctx, 
  
  AVMEDIA_TYPE_VIDEO) >= 0) {
  
      video_stream = fmt_ctx->streams[video_stream_idx];
  
      video_dst_file = fopen(video_dst_filename, "wb");
  
      if (!video_dst_file) {
  
          printf("Could not open destination file %s\n", video_dst_filename);
  
          ret = 1;
  
  }
  
  
  
  /* allocate image where the decoded image will be put */
  
  width = video_dec_ctx->width;
  
  height = video_dec_ctx->height;
  
  pix_fmt = video_dec_ctx->pix_fmt;
  
  ret = av_image_alloc(video_dst_data, video_dst_linesize,
  
                      width, height, pix_fmt, 1);
  
  if (ret &lt; 0) {
  
      printf("Could not allocate raw video buffer\n");
  
      goto end;
  
  }
  
  video_dst_bufsize = ret;
  
  }
  
  if (open_codec_context(&amp;audio_stream_idx, &amp;audio_dec_ctx, fmt_ctx, 
  
  AVMEDIA_TYPE_AUDIO) >= 0) {
  
      audio_stream = fmt_ctx->streams[audio_stream_idx];
  
      audio_dst_file = fopen(audio_dst_filename, "wb");
  
      if (!audio_dst_file) {
  
          printf("Could not open destination file %s\n", audio_dst_filename);
  
          ret = 1;
  
          goto end;
  
      }
  
  }
  
  /* dump input information to stderr */
  
  av_dump_format(fmt_ctx, 0, src_filename, 0);
  
  
  
  if (!audio_stream &amp;&amp; !video_stream) {
  
      printf("Could not find audio or video stream in the input, aborting\n");
  
      ret = 1;
  
      goto end;
  
  }
  
  
  
  frame = av_frame_alloc();
  
  if (!frame) {
  
      printf("Could not allocate frame\n");
  
      ret = AVERROR(ENOMEM);
  
      goto end;
  
  }
  
  
  
  pkt = av_packet_alloc();
  
  if (!pkt)
  
      exit(1);
  
  /* initialize packet, set data to NULL, let the demuxer fill it */
  
  av_init_packet(pkt);
  
  printf("after dump\n");
  
  pkt->data = NULL;
  
  pkt->size = 0;
  
  if (video_stream)
  
      printf("Demuxing video from file '%s' into '%s'\n", src_filename, 
  
  video_dst_filename);
  
  if (audio_stream)
  
      printf("Demuxing audio from file '%s' into '%s'\n", src_filename, 
  
  audio_dst_filename);
  
  
  
  
  
      codec = avcodec_find_encoder(AV_CODEC_ID_MPEG1VIDEO);
  
  if (!codec) {
  
      printf("Codec AV_CODEC_ID_MPEG1VIDEO not found\n");
  
      exit(1);
  
  }
  
  
  
  c = avcodec_alloc_context3(codec);
  
  if (!c) {
  
      printf("Could not allocate video codec context\n");
  
      exit(1);
  
  }
  
  
  
  /* put sample parameters */
  
  c->bit_rate = 400000;
  
  /* resolution must be a multiple of two */
  
  c->width = width;
  
  c->height = height;
  
  /* frames per second */
  
  c->time_base = (AVRational){1, 25};
  
  c->framerate = (AVRational){25, 1};
  
  /* emit one intra frame every ten frames
  
   * check frame pict_type before passing frame
  
   * to encoder, if frame->pict_type is AV_PICTURE_TYPE_I
  
   * then gop_size is ignored and the output of encoder
  
   * will always be I frame irrespective to gop_size
  
   */
  
  c->gop_size = 10;
  
  c->max_b_frames = 1;
  
  c->pix_fmt = AV_PIX_FMT_YUV420P;
  
  
  
  pkt1 = av_packet_alloc();
  
  if (!pkt1)
  
      exit(1);
  
  av_init_packet(pkt1);
  
  pkt1->data=NULL;
  
  pkt1->size=0;
  
  ret = avcodec_open2(c, codec, NULL);
  
  if (ret &lt; 0) {
  
      printf("Could not open codec: %s\n", av_err2str(ret));
  
      exit(1);
  
  }
  
      /* read frames from the file */
  
  while (av_read_frame(fmt_ctx, pkt) >= 0) {
  
      AVPacket orig_pkt = *pkt;
  
      do {
  
          ret = decode_packet(&amp;got_frame, 0);
  
          if (ret &lt; 0)
  
              break;
  
          pkt->data += ret;
  
          pkt->size -= ret;
  
  
  
      } while (pkt->size > 0);
  
      av_packet_unref(&amp;orig_pkt);
  
  }
  
  end:
  
      avcodec_free_context(&amp;video_dec_ctx);
  
      avcodec_free_context(&amp;audio_dec_ctx);
  
      avformat_close_input(&amp;fmt_ctx);
  
      if (video_dst_file)
  
          fclose(video_dst_file);
  
      if (audio_dst_file)
  
          fclose(audio_dst_file);
  
      //    av_frame_free(&amp;frame);
  
      av_free(video_dst_data[0]);
  
      return ret &lt; 0;
  
  }

  I want above program to support compress and conversion of different format also.
  Please help me getting the program ready.

• ffmpeg encoding, Opus sound in the webm container does not work

  10 mars 2019, par Mockarutan

  I’m trying to encode audio and video to a webm file with VP8 and Opus encoding. It almost works. (I use FFmpeg 3.3.2)

  I can make a only video webm file and play it in VLC, FFPlay and upload it to YouTube (and all works). If I add Opus sound to the file, it still works in VLC but not in FFPlay or on youtube, on youtube the sound becomes just "ticks".

  I have the same problem if I encode only Opus audio to the webm file ; it only works in VLC. But if I encode only Opus audio to a ogg container it works everywhere, and I can even use FFmpeg to combine the ogg file with a video only webm file and produce a fully working webm file with audio and video.

  So it seems to me that only when I use my code to encode Opus into a webm container, it just wont work in most players and YouTube. I need it to work in youtube.

  Here is the code for the opus to webm only encoding (you can toggle ogg/webm with the define) : https://pastebin.com/jyQ4s3tB

  #include <algorithm>
  
  #include <iterator>
  
  
  
  extern "C"
  
  {
  
  
  
  //#define OGG
  
  
  
  #include "libavcodec/avcodec.h"
  
  #include "libavdevice/avdevice.h"
  
  #include "libavfilter/avfilter.h"
  
  #include "libavformat/avformat.h"
  
  #include "libavutil/avutil.h"
  
  #include "libavutil/imgutils.h"
  
  #include "libswscale/swscale.h"
  
  #include "libswresample/swresample.h"
  
  
  
      enum InfoCodes
  
      {
  
          ENCODED_VIDEO,
  
          ENCODED_AUDIO,
  
          ENCODED_AUDIO_AND_VIDEO,
  
          NOT_ENOUGH_AUDIO_DATA,
  
      };
  
  
  
      enum ErrorCodes
  
      {
  
          RES_NOT_MUL_OF_TWO = -1,
  
          ERROR_FINDING_VID_CODEC = -2,
  
          ERROR_CONTEXT_CREATION = -3,
  
          ERROR_CONTEXT_ALLOCATING = -4,
  
          ERROR_OPENING_VID_CODEC = -5,
  
          ERROR_OPENING_FILE = -6,
  
          ERROR_ALLOCATING_FRAME = -7,
  
          ERROR_ALLOCATING_PIC_BUF = -8,
  
          ERROR_ENCODING_FRAME_SEND = -9,
  
          ERROR_ENCODING_FRAME_RECEIVE = -10,
  
          ERROR_FINDING_AUD_CODEC = -11,
  
          ERROR_OPENING_AUD_CODEC = -12,
  
          ERROR_INIT_RESMPL_CONTEXT = -13,
  
          ERROR_ENCODING_SAMPLES_SEND = -14,
  
          ERROR_ENCODING_SAMPLES_RECEIVE = -15,
  
          ERROR_WRITING_HEADER = -16,
  
          ERROR_INIT_AUDIO_RESPAMLER = -17,
  
      };
  
  
  
      AVCodecID aud_codec_comp_id = AV_CODEC_ID_OPUS;
  
      AVSampleFormat sample_fmt_comp = AV_SAMPLE_FMT_FLT;
  
  
  
      AVCodecID aud_codec_id;
  
      AVSampleFormat sample_fmt;
  
  
  
  #ifndef OGG
  
      char* compressed_cont = "webm";
  
  #endif
  
  #ifdef OGG
  
      char* compressed_cont = "ogg";
  
  #endif
  
  
  
      AVCodec *aud_codec = NULL;
  
      AVCodecContext *aud_codec_context = NULL;
  
      AVFormatContext *outctx;
  
      AVStream *audio_st;
  
      AVFrame *aud_frame;
  
      SwrContext *audio_swr_ctx;
  
  
  
      int vid_frame_counter, aud_frame_counter;
  
      int vid_width, vid_height;
  
  
  
      char* concat(const char *s1, const char *s2)
  
      {
  
          char *result = (char*)malloc(strlen(s1) + strlen(s2) + 1);
  
  
  
          strcpy(result, s1);
  
          strcat(result, s2);
  
  
  
          return result;
  
      }
  
  
  
      int setup_audio_codec()
  
      {
  
          aud_codec_id = aud_codec_comp_id;
  
          sample_fmt = sample_fmt_comp;
  
  
  
          // Fixup audio codec
  
          if (aud_codec == NULL)
  
          {
  
              aud_codec = avcodec_find_encoder(aud_codec_id);
  
              avcodec_register(aud_codec);
  
          }
  
  
  
          if (!aud_codec)
  
              return ERROR_FINDING_AUD_CODEC;
  
  
  
          return 0;
  
      }
  
  
  
      int initialize_audio_stream(AVFormatContext *local_outctx, int sample_rate, int per_frame_audio_samples, int audio_bitrate)
  
      {
  
          aud_codec_context = avcodec_alloc_context3(aud_codec);
  
          if (!aud_codec_context)
  
              return ERROR_CONTEXT_CREATION;
  
  
  
          aud_codec_context->bit_rate = audio_bitrate;
  
          aud_codec_context->sample_rate = sample_rate;
  
          aud_codec_context->sample_fmt = sample_fmt;
  
          aud_codec_context->channel_layout = AV_CH_LAYOUT_STEREO;
  
          aud_codec_context->channels = av_get_channel_layout_nb_channels(aud_codec_context->channel_layout);
  
          //aud_codec_context->profile = FF_PROFILE_AAC_MAIN;
  
  
  
          aud_codec_context->codec = aud_codec;
  
          aud_codec_context->codec_id = aud_codec_id;
  
  
  
          AVRational time_base;
  
          time_base.num = per_frame_audio_samples;
  
          time_base.den = aud_codec_context->sample_rate;
  
          aud_codec_context->time_base = time_base;
  
  
  
          int ret = avcodec_open2(aud_codec_context, aud_codec, NULL);
  
  
  
          if (ret &lt; 0)
  
              return ERROR_OPENING_AUD_CODEC;
  
  
  
          local_outctx->audio_codec = aud_codec;
  
          local_outctx->audio_codec_id = aud_codec_id;
  
  
  
          audio_st = avformat_new_stream(local_outctx, aud_codec);
  
  
  
          audio_st->codecpar->bit_rate = aud_codec_context->bit_rate;
  
          audio_st->codecpar->sample_rate = aud_codec_context->sample_rate;
  
          audio_st->codecpar->channels = aud_codec_context->channels;
  
          audio_st->codecpar->channel_layout = aud_codec_context->channel_layout;
  
          audio_st->codecpar->codec_id = aud_codec_context->codec_id;
  
          audio_st->codecpar->codec_type = AVMEDIA_TYPE_AUDIO;
  
          audio_st->codecpar->format = aud_codec_context->sample_fmt;
  
          audio_st->codecpar->frame_size = aud_codec_context->frame_size;
  
          audio_st->codecpar->block_align = aud_codec_context->block_align;
  
          audio_st->codecpar->initial_padding = aud_codec_context->initial_padding;
  
          audio_st->codecpar->extradata = aud_codec_context->extradata;
  
          audio_st->codecpar->extradata_size = aud_codec_context->extradata_size;
  
  
  
          aud_frame = av_frame_alloc();
  
          aud_frame->nb_samples = aud_codec_context->frame_size;
  
          aud_frame->format = aud_codec_context->sample_fmt;
  
          aud_frame->channel_layout = aud_codec_context->channel_layout;
  
          aud_frame->sample_rate = aud_codec_context->sample_rate;
  
  
  
          int buffer_size;
  
          if (aud_codec_context->frame_size == 0)
  
          {
  
              buffer_size = per_frame_audio_samples * 2 * 4;
  
              aud_frame->nb_samples = per_frame_audio_samples;
  
          }
  
          else
  
          {
  
              buffer_size = av_samples_get_buffer_size(NULL, aud_codec_context->channels, aud_codec_context->frame_size,
  
                  aud_codec_context->sample_fmt, 0);
  
          }
  
  
  
          if (av_sample_fmt_is_planar(sample_fmt))
  
              ret = av_frame_get_buffer(aud_frame, buffer_size / 2);
  
          else
  
              ret = av_frame_get_buffer(aud_frame, buffer_size);
  
  
  
          if (!aud_frame || ret &lt; 0)
  
              return ERROR_ALLOCATING_FRAME;
  
  
  
          aud_frame_counter = 0;
  
  
  
          return 0;
  
      }
  
  
  
      int initialize_audio_only_encoding(int sample_rate, int per_frame_audio_samples, int audio_bitrate, const char *filename)
  
      {
  
          int ret;
  
  
  
          avcodec_register_all();
  
          av_register_all();
  
  
  
          outctx = avformat_alloc_context();
  
  
  
          char* with_dot = concat(filename, ".");
  
          char* full_filename = concat(with_dot, compressed_cont);
  
  
  
          ret = avformat_alloc_output_context2(&amp;outctx, NULL, compressed_cont, full_filename);
  
  
  
          free(with_dot);
  
  
  
          if (ret &lt; 0)
  
          {
  
              free(full_filename);
  
              return ERROR_CONTEXT_CREATION;
  
          }
  
  
  
          ret = setup_audio_codec();
  
          if (ret &lt; 0)
  
              return ret;
  
  
  
          // Setup Audio
  
          ret = initialize_audio_stream(outctx, sample_rate, per_frame_audio_samples, audio_bitrate);
  
          if (ret &lt; 0)
  
              return ret;
  
  
  
          av_dump_format(outctx, 0, full_filename, 1);
  
  
  
          if (!(outctx->oformat->flags &amp; AVFMT_NOFILE))
  
          {
  
              if (avio_open(&amp;outctx->pb, full_filename, AVIO_FLAG_WRITE) &lt; 0)
  
              {
  
                  free(full_filename);
  
                  return ERROR_OPENING_FILE;
  
              }
  
          }
  
  
  
          free(full_filename);
  
  
  
          ret = avformat_write_header(outctx, NULL);
  
          if (ret &lt; 0)
  
              return ERROR_WRITING_HEADER;
  
  
  
          return 0;
  
      }
  
  
  
      int write_interleaved_audio_frame(float_t *aud_sample)
  
      {
  
          int ret;
  
  
  
          aud_frame->data[0] = (uint8_t*)aud_sample;
  
          aud_frame->extended_data[0] = (uint8_t*)aud_sample;
  
  
  
          aud_frame->pts = aud_frame_counter++;
  
  
  
          ret = avcodec_send_frame(aud_codec_context, aud_frame);
  
  
  
          AVPacket pkt;
  
          av_init_packet(&amp;pkt);
  
          pkt.data = NULL;
  
          pkt.size = 0;
  
  
  
          while (true)
  
          {
  
              ret = avcodec_receive_packet(aud_codec_context, &amp;pkt);
  
              if (!ret)
  
              {
  
                  av_packet_rescale_ts(&amp;pkt, aud_codec_context->time_base, audio_st->time_base);
  
  
  
                  pkt.stream_index = audio_st->index;
  
  
  
                  av_interleaved_write_frame(outctx, &amp;pkt);
  
  
  
                  av_packet_unref(&amp;pkt);
  
              }
  
              if (ret == AVERROR(EAGAIN))
  
                  break;
  
              else if (ret &lt; 0)
  
                  return ERROR_ENCODING_SAMPLES_RECEIVE;
  
              else
  
                  break;
  
          }
  
  
  
          return ENCODED_AUDIO;
  
      }
  
  
  
      int write_audio_frame(float_t *aud_sample)
  
      {
  
          int ret;
  
          aud_frame->data[0] = (uint8_t*)aud_sample;
  
          aud_frame->extended_data[0] = (uint8_t*)aud_sample;
  
  
  
          aud_frame->pts = aud_frame_counter++;
  
  
  
          ret = avcodec_send_frame(aud_codec_context, aud_frame);
  
          if (ret &lt; 0)
  
              return ERROR_ENCODING_FRAME_SEND;
  
  
  
          AVPacket pkt;
  
          av_init_packet(&amp;pkt);
  
          pkt.data = NULL;
  
          pkt.size = 0;
  
  
  
          fflush(stdout);
  
  
  
          while (true)
  
          {
  
              ret = avcodec_receive_packet(aud_codec_context, &amp;pkt);
  
              if (!ret)
  
                  if (pkt.pts != AV_NOPTS_VALUE)
  
                      pkt.pts = av_rescale_q(pkt.pts, aud_codec_context->time_base, audio_st->time_base);
  
              if (pkt.dts != AV_NOPTS_VALUE)
  
                  pkt.dts = av_rescale_q(pkt.dts, aud_codec_context->time_base, audio_st->time_base);
  
              {
  
  
  
                  av_write_frame(outctx, &amp;pkt);
  
                  av_packet_unref(&amp;pkt);
  
              }
  
              if (ret == AVERROR(EAGAIN))
  
                  break;
  
              else if (ret &lt; 0)
  
                  return ERROR_ENCODING_FRAME_RECEIVE;
  
              else
  
                  break;
  
          }
  
  
  
          return ENCODED_AUDIO;
  
      }
  
  
  
      int finish_audio_encoding()
  
      {
  
          AVPacket pkt;
  
          av_init_packet(&amp;pkt);
  
          pkt.data = NULL;
  
          pkt.size = 0;
  
  
  
          fflush(stdout);
  
  
  
          int ret = avcodec_send_frame(aud_codec_context, NULL);
  
          if (ret &lt; 0)
  
              return ERROR_ENCODING_FRAME_SEND;
  
  
  
          while (true)
  
          {
  
              ret = avcodec_receive_packet(aud_codec_context, &amp;pkt);
  
              if (!ret)
  
              {
  
                  if (pkt.pts != AV_NOPTS_VALUE)
  
                      pkt.pts = av_rescale_q(pkt.pts, aud_codec_context->time_base, audio_st->time_base);
  
                  if (pkt.dts != AV_NOPTS_VALUE)
  
                      pkt.dts = av_rescale_q(pkt.dts, aud_codec_context->time_base, audio_st->time_base);
  
  
  
                  av_write_frame(outctx, &amp;pkt);
  
                  av_packet_unref(&amp;pkt);
  
              }
  
              if (ret == -AVERROR(AVERROR_EOF))
  
                  break;
  
              else if (ret &lt; 0)
  
                  return ERROR_ENCODING_FRAME_RECEIVE;
  
          }
  
  
  
          av_write_trailer(outctx);
  
  
  
          return 0;
  
      }
  
  
  
      void cleanup()
  
      {
  
          if (aud_frame)
  
          {
  
              av_frame_free(&amp;aud_frame);
  
          }
  
          if (outctx)
  
          {
  
              for (int i = 0; i &lt; outctx->nb_streams; i++)
  
                  av_freep(&amp;outctx->streams[i]);
  
  
  
              avio_close(outctx->pb);
  
              av_free(outctx);
  
          }
  
  
  
          if (aud_codec_context)
  
          {
  
              avcodec_close(aud_codec_context);
  
              av_free(aud_codec_context);
  
          }
  
      }
  
  
  
      void fill_samples(float_t *dst, int nb_samples, int nb_channels, int sample_rate, float_t *t)
  
      {
  
          int i, j;
  
          float_t tincr = 1.0 / sample_rate;
  
          const float_t c = 2 * M_PI * 440.0;
  
  
  
          for (i = 0; i &lt; nb_samples; i++) {
  
              *dst = sin(c * *t);
  
              for (j = 1; j &lt; nb_channels; j++)
  
                  dst[j] = dst[0];
  
              dst += nb_channels;
  
              *t += tincr;
  
          }
  
      }
  
  
  
      int main()
  
      {
  
          int sec = 5;
  
          int frame_rate = 30;
  
          float t = 0, tincr = 0, tincr2 = 0;
  
  
  
          int src_samples_linesize;
  
          int src_nb_samples = 960;
  
          int src_channels = 2;
  
          int sample_rate = 48000;
  
  
  
          uint8_t **src_data = NULL;
  
  
  
          int ret;
  
  
  
          initialize_audio_only_encoding(48000, src_nb_samples, 192000, "sound_FLT_960");
  
  
  
          ret = av_samples_alloc_array_and_samples(&amp;src_data, &amp;src_samples_linesize, src_channels,
  
              src_nb_samples, AV_SAMPLE_FMT_FLT, 0);
  
  
  
          for (size_t i = 0; i &lt; frame_rate * sec; i++)
  
          {
  
                  fill_samples((float *)src_data[0], src_nb_samples, src_channels, sample_rate, &amp;t);
  
                  write_interleaved_audio_frame((float *)src_data[0]);
  
          }
  
  
  
          finish_audio_encoding();
  
  
  
          cleanup();
  
  
  
          return 0;
  
      }
  
  }
  
  </iterator></algorithm>

  And some of the files :

  The webm audio file that does not work (only in VLC) :
  https://drive.google.com/file/d/0B16rIXjPXJCqcU5HVllIYW1iODg/view?usp=sharing

  The ogg audio file that works :
  https://drive.google.com/file/d/0B16rIXjPXJCqMUZhbW0tTDFjT1E/view?usp=sharing

  Video and Audio file that only works in VLC : https://drive.google.com/file/d/0B16rIXjPXJCqX3pEN3B0QVlrekU/view?usp=sharing

  If a play the ogg file in FFPlay it says "aq= 30kb", but if I play the webm audio only file i get "aq= 0kb". So that does not seem right either.

  Any idea ? Thanks in advance !

  Edit : So I made it work by just encoding both VP8 and Opus into the ogg container and then simply renaming it to .webm and uploading it YouTube. I did not actually know ogg could have video inside of it. I do not really know what how it affects the encoding and stuff... I can upload the original ogg file with video and it also works on YouTube. But the whole reason I went for webm was the licensing it has (https://www.webmproject.org/license/)... So I’m a bit confused now.

  I need to read up on what exactly a "container" means in the context and what just changing the extension means.

  Any comments to shed some light on this appreciated !