Recherche avancée

Sur d’autres sites (6580)

• Confused about x264 and encoding video frames

  26 février 2015, par spartygw

  I built a test driver for encoding a series of images I have captured. I am using libx264 and based my driver off of this guy’s answer :

  StackOverflow link

  In my case I am starting out by reading in a JPG image and converting to YUV and passing that same frame over and over in a loop to the x264 encoder.

  My expectation was that since the frame is the same that the output from the encoder would be very small and constant.

  Instead I find that the NAL payload is varied from a few bytes to a few KB and also varies highly depending on the frame rate I specify in the encoder parameters.

  Obviously I don’t understand video encoding. Why does the output size vary so much ?

  int main() 
  
  {
  
    Image image(WIDTH, HEIGHT);
  
    image.FromJpeg("frame-1.jpg");
  
  
  
    unsigned char *data = image.GetRGB();
  
  
  
    x264_param_t param;
  
  
  
    x264_param_default_preset(&param, "fast", "zerolatency");
  
    param.i_threads = 1;
  
    param.i_width = WIDTH;
  
    param.i_height = HEIGHT;
  
    param.i_fps_num = FPS;
  
    param.i_fps_den = 1;
  
  
  
    // Intra refres:
  
    param.i_keyint_max = FPS;
  
    param.b_intra_refresh = 1;
  
  
  
    //Rate control:
  
    param.rc.i_rc_method = X264_RC_CRF;
  
    param.rc.f_rf_constant = FPS-5;
  
    param.rc.f_rf_constant_max = FPS+5;
  
  
  
    //For streaming:
  
    param.b_repeat_headers = 1;
  
    param.b_annexb = 1;
  
  
  
    x264_param_apply_profile(&param, "baseline");
  
  
  
    // initialize the encoder
  
    x264_t* encoder = x264_encoder_open(&param);
  
    x264_picture_t pic_in, pic_out;
  
    x264_picture_alloc(&pic_in, X264_CSP_I420, WIDTH, HEIGHT);
  
    // X264 expects YUV420P data use libswscale 
  
    // (from ffmpeg) to convert images to the right format
  
    struct SwsContext* convertCtx =
  
          sws_getContext(WIDTH, HEIGHT, PIX_FMT_RGB24, WIDTH, HEIGHT,
  
                         PIX_FMT_YUV420P, SWS_FAST_BILINEAR,
  
                         NULL, NULL, NULL);
  
  
  
    // encoding is as simple as this then, for each frame do:
  
    // data is a pointer to your RGB structure
  
    int srcstride = WIDTH*3; //RGB stride is just 3*width
  
    sws_scale(convertCtx, &data, &srcstride, 0, HEIGHT,
  
              pic_in.img.plane, pic_in.img.i_stride);
  
    x264_nal_t* nals;
  
    int i_nals;
  
    int frame_size =
  
          x264_encoder_encode(encoder, &nals, &i_nals, &pic_in, &pic_out);
  
  
  
    int max_loop=15;
  
    int this_loop=1;
  
  
  
    while (frame_size >= 0 && --max_loop)
  
    {
  
        cout << "------------" << this_loop++ << "-----------------\n";
  
        cout << "Frame size = " << frame_size << endl;
  
        cout << "output has " << pic_out.img.i_csp << " colorspace\n";
  
        cout << "output has " << pic_out.img.i_plane << " # img planes\n";
  
  
  
        cout << "i_nals = " << i_nals << endl;
  
        for (int n=0; n

• Trouble syncing libavformat/ffmpeg with x264 and RTP

  26 décembre 2012, par Jacob Peddicord

  I've been working on some streaming software that takes live feeds
  from various kinds of cameras and streams over the network using
  H.264. To accomplish this, I'm using the x264 encoder directly (with
  the "zerolatency" preset) and feeding NALs as they are available to
  libavformat to pack into RTP (ultimately RTSP). Ideally, this
  application should be as real-time as possible. For the most part,
  this has been working well.

  Unfortunately, however, there is some sort of synchronization issue :
  any video playback on clients seems to show a few smooth frames,
  followed by a short pause, then more frames ; repeat. Additionally,
  there appears to be approximately a 4-second delay. This happens with
  every video player I've tried : Totem, VLC, and basic gstreamer pipes.

  I've boiled it all down to a somewhat small test case :

  #include 
  
  #include 
  
  #include 
  
  #include 
  
  #include <libavformat></libavformat>avformat.h>
  
  #include <libswscale></libswscale>swscale.h>
  
  
  
  #define WIDTH       640
  
  #define HEIGHT      480
  
  #define FPS         30
  
  #define BITRATE     400000
  
  #define RTP_ADDRESS "127.0.0.1"
  
  #define RTP_PORT    49990
  
  
  
  struct AVFormatContext* avctx;
  
  struct x264_t* encoder;
  
  struct SwsContext* imgctx;
  
  
  
  uint8_t test = 0x80;
  
  
  
  
  
  void create_sample_picture(x264_picture_t* picture)
  
  {
  
      // create a frame to store in
  
      x264_picture_alloc(picture, X264_CSP_I420, WIDTH, HEIGHT);
  
  
  
      // fake image generation
  
      // disregard how wrong this is; just writing a quick test
  
      int strides = WIDTH / 8;
  
      uint8_t* data = malloc(WIDTH * HEIGHT * 3);
  
      memset(data, test, WIDTH * HEIGHT * 3);
  
      test = (test &lt;&lt; 1) | (test >> (8 - 1));
  
  
  
      // scale the image
  
      sws_scale(imgctx, (const uint8_t* const*) &amp;data, &amp;strides, 0, HEIGHT,
  
                picture->img.plane, picture->img.i_stride);
  
  }
  
  
  
  int encode_frame(x264_picture_t* picture, x264_nal_t** nals)
  
  {
  
      // encode a frame
  
      x264_picture_t pic_out;
  
      int num_nals;
  
      int frame_size = x264_encoder_encode(encoder, nals, &amp;num_nals, picture, &amp;pic_out);
  
  
  
      // ignore bad frames
  
      if (frame_size &lt; 0)
  
      {
  
          return frame_size;
  
      }
  
  
  
      return num_nals;
  
  }
  
  
  
  void stream_frame(uint8_t* payload, int size)
  
  {
  
      // initalize a packet
  
      AVPacket p;
  
      av_init_packet(&amp;p);
  
      p.data = payload;
  
      p.size = size;
  
      p.stream_index = 0;
  
      p.flags = AV_PKT_FLAG_KEY;
  
      p.pts = AV_NOPTS_VALUE;
  
      p.dts = AV_NOPTS_VALUE;
  
  
  
      // send it out
  
      av_interleaved_write_frame(avctx, &amp;p);
  
  }
  
  
  
  int main(int argc, char* argv[])
  
  {
  
      // initalize ffmpeg
  
      av_register_all();
  
  
  
      // set up image scaler
  
      // (in-width, in-height, in-format, out-width, out-height, out-format, scaling-method, 0, 0, 0)
  
      imgctx = sws_getContext(WIDTH, HEIGHT, PIX_FMT_MONOWHITE,
  
                              WIDTH, HEIGHT, PIX_FMT_YUV420P,
  
                              SWS_FAST_BILINEAR, NULL, NULL, NULL);
  
  
  
      // set up encoder presets
  
      x264_param_t param;
  
      x264_param_default_preset(&amp;param, "ultrafast", "zerolatency");
  
  
  
      param.i_threads = 3;
  
      param.i_width = WIDTH;
  
      param.i_height = HEIGHT;
  
      param.i_fps_num = FPS;
  
      param.i_fps_den = 1;
  
      param.i_keyint_max = FPS;
  
      param.b_intra_refresh = 0;
  
      param.rc.i_bitrate = BITRATE;
  
      param.b_repeat_headers = 1; // whether to repeat headers or write just once
  
      param.b_annexb = 1;         // place start codes (1) or sizes (0)
  
  
  
      // initalize
  
      x264_param_apply_profile(&amp;param, "high");
  
      encoder = x264_encoder_open(&amp;param);
  
  
  
      // at this point, x264_encoder_headers can be used, but it has had no effect
  
  
  
      // set up streaming context. a lot of error handling has been ommitted
  
      // for brevity, but this should be pretty standard.
  
      avctx = avformat_alloc_context();
  
      struct AVOutputFormat* fmt = av_guess_format("rtp", NULL, NULL);
  
      avctx->oformat = fmt;
  
  
  
      snprintf(avctx->filename, sizeof(avctx->filename), "rtp://%s:%d", RTP_ADDRESS, RTP_PORT);
  
      if (url_fopen(&amp;avctx->pb, avctx->filename, URL_WRONLY) &lt; 0)
  
      {
  
          perror("url_fopen failed");
  
          return 1;
  
      }
  
      struct AVStream* stream = av_new_stream(avctx, 1);
  
  
  
      // initalize codec
  
      AVCodecContext* c = stream->codec;
  
      c->codec_id = CODEC_ID_H264;
  
      c->codec_type = AVMEDIA_TYPE_VIDEO;
  
      c->flags = CODEC_FLAG_GLOBAL_HEADER;
  
      c->width = WIDTH;
  
      c->height = HEIGHT;
  
      c->time_base.den = FPS;
  
      c->time_base.num = 1;
  
      c->gop_size = FPS;
  
      c->bit_rate = BITRATE;
  
      avctx->flags = AVFMT_FLAG_RTP_HINT;
  
  
  
      // write the header
  
      av_write_header(avctx);
  
  
  
      // make some frames
  
      for (int frame = 0; frame &lt; 10000; frame++)
  
      {
  
          // create a sample moving frame
  
          x264_picture_t* pic = (x264_picture_t*) malloc(sizeof(x264_picture_t));
  
          create_sample_picture(pic);
  
  
  
          // encode the frame
  
          x264_nal_t* nals;
  
          int num_nals = encode_frame(pic, &amp;nals);
  
  
  
          if (num_nals &lt; 0)
  
              printf("invalid frame size: %d\n", num_nals);
  
  
  
          // send out NALs
  
          for (int i = 0; i &lt; num_nals; i++)
  
          {
  
              stream_frame(nals[i].p_payload, nals[i].i_payload);
  
          }
  
  
  
          // free up resources
  
          x264_picture_clean(pic);
  
          free(pic);
  
  
  
          // stream at approx 30 fps
  
          printf("frame %d\n", frame);
  
          usleep(33333);
  
      }
  
  
  
      return 0;
  
  }

  This test shows black lines on a white background that
  should move smoothly to the left. It has been written for ffmpeg 0.6.5
  but the problem can be reproduced on 0.8 and 0.10 (from what I've tested so far). I've taken some shortcuts in error handling to make this example as short as
  possible while still showing the problem, so please excuse some of the
  nasty code. I should also note that while an SDP is not used here, I
  have tried using that already with similar results. The test can be
  compiled with :

  gcc -g -std=gnu99 streamtest.c -lswscale -lavformat -lx264 -lm -lpthread -o streamtest

  It can be played with gtreamer directly :

  gst-launch udpsrc port=49990 ! application/x-rtp,payload=96,clock-rate=90000 ! rtph264depay ! decodebin ! xvimagesink

  You should immediately notice the stuttering. One common "fix" I've
  seen all over the Internet is to add sync=false to the pipeline :

  gst-launch udpsrc port=49990 ! application/x-rtp,payload=96,clock-rate=90000 ! rtph264depay ! decodebin ! xvimagesink sync=false

  This causes playback to be smooth (and near-realtime), but is a
  non-solution and only works with gstreamer. I'd like to fix the
  problem at the source. I've been able to stream with near-identical
  parameters using raw ffmpeg and haven't had any issues :

  ffmpeg -re -i sample.mp4 -vcodec libx264 -vpre ultrafast -vpre baseline -b 400000 -an -f rtp rtp://127.0.0.1:49990 -an

  So clearly I'm doing something wrong. But what is it ?

• Trouble syncing libavformat/ffmpeg with x264 and RTP

  26 décembre 2012, par Jacob Peddicord

  I've been working on some streaming software that takes live feeds
  from various kinds of cameras and streams over the network using
  H.264. To accomplish this, I'm using the x264 encoder directly (with
  the "zerolatency" preset) and feeding NALs as they are available to
  libavformat to pack into RTP (ultimately RTSP). Ideally, this
  application should be as real-time as possible. For the most part,
  this has been working well.

  Unfortunately, however, there is some sort of synchronization issue :
  any video playback on clients seems to show a few smooth frames,
  followed by a short pause, then more frames ; repeat. Additionally,
  there appears to be approximately a 4-second delay. This happens with
  every video player I've tried : Totem, VLC, and basic gstreamer pipes.

  I've boiled it all down to a somewhat small test case :

  #include 
  
  #include 
  
  #include 
  
  #include 
  
  #include <libavformat></libavformat>avformat.h>
  
  #include <libswscale></libswscale>swscale.h>
  
  
  
  #define WIDTH       640
  
  #define HEIGHT      480
  
  #define FPS         30
  
  #define BITRATE     400000
  
  #define RTP_ADDRESS "127.0.0.1"
  
  #define RTP_PORT    49990
  
  
  
  struct AVFormatContext* avctx;
  
  struct x264_t* encoder;
  
  struct SwsContext* imgctx;
  
  
  
  uint8_t test = 0x80;
  
  
  
  
  
  void create_sample_picture(x264_picture_t* picture)
  
  {
  
      // create a frame to store in
  
      x264_picture_alloc(picture, X264_CSP_I420, WIDTH, HEIGHT);
  
  
  
      // fake image generation
  
      // disregard how wrong this is; just writing a quick test
  
      int strides = WIDTH / 8;
  
      uint8_t* data = malloc(WIDTH * HEIGHT * 3);
  
      memset(data, test, WIDTH * HEIGHT * 3);
  
      test = (test &lt;&lt; 1) | (test >> (8 - 1));
  
  
  
      // scale the image
  
      sws_scale(imgctx, (const uint8_t* const*) &amp;data, &amp;strides, 0, HEIGHT,
  
                picture->img.plane, picture->img.i_stride);
  
  }
  
  
  
  int encode_frame(x264_picture_t* picture, x264_nal_t** nals)
  
  {
  
      // encode a frame
  
      x264_picture_t pic_out;
  
      int num_nals;
  
      int frame_size = x264_encoder_encode(encoder, nals, &amp;num_nals, picture, &amp;pic_out);
  
  
  
      // ignore bad frames
  
      if (frame_size &lt; 0)
  
      {
  
          return frame_size;
  
      }
  
  
  
      return num_nals;
  
  }
  
  
  
  void stream_frame(uint8_t* payload, int size)
  
  {
  
      // initalize a packet
  
      AVPacket p;
  
      av_init_packet(&amp;p);
  
      p.data = payload;
  
      p.size = size;
  
      p.stream_index = 0;
  
      p.flags = AV_PKT_FLAG_KEY;
  
      p.pts = AV_NOPTS_VALUE;
  
      p.dts = AV_NOPTS_VALUE;
  
  
  
      // send it out
  
      av_interleaved_write_frame(avctx, &amp;p);
  
  }
  
  
  
  int main(int argc, char* argv[])
  
  {
  
      // initalize ffmpeg
  
      av_register_all();
  
  
  
      // set up image scaler
  
      // (in-width, in-height, in-format, out-width, out-height, out-format, scaling-method, 0, 0, 0)
  
      imgctx = sws_getContext(WIDTH, HEIGHT, PIX_FMT_MONOWHITE,
  
                              WIDTH, HEIGHT, PIX_FMT_YUV420P,
  
                              SWS_FAST_BILINEAR, NULL, NULL, NULL);
  
  
  
      // set up encoder presets
  
      x264_param_t param;
  
      x264_param_default_preset(&amp;param, "ultrafast", "zerolatency");
  
  
  
      param.i_threads = 3;
  
      param.i_width = WIDTH;
  
      param.i_height = HEIGHT;
  
      param.i_fps_num = FPS;
  
      param.i_fps_den = 1;
  
      param.i_keyint_max = FPS;
  
      param.b_intra_refresh = 0;
  
      param.rc.i_bitrate = BITRATE;
  
      param.b_repeat_headers = 1; // whether to repeat headers or write just once
  
      param.b_annexb = 1;         // place start codes (1) or sizes (0)
  
  
  
      // initalize
  
      x264_param_apply_profile(&amp;param, "high");
  
      encoder = x264_encoder_open(&amp;param);
  
  
  
      // at this point, x264_encoder_headers can be used, but it has had no effect
  
  
  
      // set up streaming context. a lot of error handling has been ommitted
  
      // for brevity, but this should be pretty standard.
  
      avctx = avformat_alloc_context();
  
      struct AVOutputFormat* fmt = av_guess_format("rtp", NULL, NULL);
  
      avctx->oformat = fmt;
  
  
  
      snprintf(avctx->filename, sizeof(avctx->filename), "rtp://%s:%d", RTP_ADDRESS, RTP_PORT);
  
      if (url_fopen(&amp;avctx->pb, avctx->filename, URL_WRONLY) &lt; 0)
  
      {
  
          perror("url_fopen failed");
  
          return 1;
  
      }
  
      struct AVStream* stream = av_new_stream(avctx, 1);
  
  
  
      // initalize codec
  
      AVCodecContext* c = stream->codec;
  
      c->codec_id = CODEC_ID_H264;
  
      c->codec_type = AVMEDIA_TYPE_VIDEO;
  
      c->flags = CODEC_FLAG_GLOBAL_HEADER;
  
      c->width = WIDTH;
  
      c->height = HEIGHT;
  
      c->time_base.den = FPS;
  
      c->time_base.num = 1;
  
      c->gop_size = FPS;
  
      c->bit_rate = BITRATE;
  
      avctx->flags = AVFMT_FLAG_RTP_HINT;
  
  
  
      // write the header
  
      av_write_header(avctx);
  
  
  
      // make some frames
  
      for (int frame = 0; frame &lt; 10000; frame++)
  
      {
  
          // create a sample moving frame
  
          x264_picture_t* pic = (x264_picture_t*) malloc(sizeof(x264_picture_t));
  
          create_sample_picture(pic);
  
  
  
          // encode the frame
  
          x264_nal_t* nals;
  
          int num_nals = encode_frame(pic, &amp;nals);
  
  
  
          if (num_nals &lt; 0)
  
              printf("invalid frame size: %d\n", num_nals);
  
  
  
          // send out NALs
  
          for (int i = 0; i &lt; num_nals; i++)
  
          {
  
              stream_frame(nals[i].p_payload, nals[i].i_payload);
  
          }
  
  
  
          // free up resources
  
          x264_picture_clean(pic);
  
          free(pic);
  
  
  
          // stream at approx 30 fps
  
          printf("frame %d\n", frame);
  
          usleep(33333);
  
      }
  
  
  
      return 0;
  
  }

  This test shows black lines on a white background that
  should move smoothly to the left. It has been written for ffmpeg 0.6.5
  but the problem can be reproduced on 0.8 and 0.10 (from what I've tested so far). I've taken some shortcuts in error handling to make this example as short as
  possible while still showing the problem, so please excuse some of the
  nasty code. I should also note that while an SDP is not used here, I
  have tried using that already with similar results. The test can be
  compiled with :

  gcc -g -std=gnu99 streamtest.c -lswscale -lavformat -lx264 -lm -lpthread -o streamtest

  It can be played with gtreamer directly :

  gst-launch udpsrc port=49990 ! application/x-rtp,payload=96,clock-rate=90000 ! rtph264depay ! decodebin ! xvimagesink

  You should immediately notice the stuttering. One common "fix" I've
  seen all over the Internet is to add sync=false to the pipeline :

  gst-launch udpsrc port=49990 ! application/x-rtp,payload=96,clock-rate=90000 ! rtph264depay ! decodebin ! xvimagesink sync=false

  This causes playback to be smooth (and near-realtime), but is a
  non-solution and only works with gstreamer. I'd like to fix the
  problem at the source. I've been able to stream with near-identical
  parameters using raw ffmpeg and haven't had any issues :

  ffmpeg -re -i sample.mp4 -vcodec libx264 -vpre ultrafast -vpre baseline -b 400000 -an -f rtp rtp://127.0.0.1:49990 -an

  So clearly I'm doing something wrong. But what is it ?