Recherche avancée

Sur d’autres sites (11490)

• Image overlay in FFMpeg using the C library

  11 juin 2018, par Ludwig Pumberger

  I would like to dynamically manipulate a video to show a number that is pulled from an external source.

  This works well if I just the drawText filter. Unfortunately the customer would like the overlay to look a lot fancier. I have an image with digits in the required format and I can use that to dynamically generate an image containing the number. However, I am unsure how to use the FFMPEG overlay filter to merge this image with the video.

  I have found samples for doing this on the command line with a command like this one :

  ffmpeg -itsoffset 5 -i in.mp4 -r 25 -loop 1 -i intro.png -filter_complex "[1:v] fade=out:125:25:alpha=1 [intro]; [0:v][intro] overlay [v]" -map "[v]" -map 0:a -acodec copy out.mp4

  The main problem is that I cannot use the command line tool. I need to manipulate the video on the fly (the video data is sent to a third party streaming library). The working solution I have, which uses drawtext only needs one source (the original video). For the overlay I would have to somehow specify a second source (the image) and I have no idea how I can do this.

  Currently I use this method (taken from the FFMPEG filter example) to create the filter :

  static int init_filters(AVFormatContext *fmt_ctx,
  
  AVCodecContext *dec_ctx,
  
  AVFilterContext **buffersink_ctx,
  
  AVFilterContext **buffersrc_ctx,
  
  AVFilterGraph **filter_graph,
  
  int video_stream_index,
  
  const char *filters_descr)
  
  {
  
      char args[512];
  
      int ret = 0;
  
      AVFilter *buffersrc = avfilter_get_by_name("buffer");
  
      AVFilter *buffersink = avfilter_get_by_name("buffersink");
  
      AVFilterInOut *outputs = avfilter_inout_alloc();
  
      AVFilterInOut *inputs = avfilter_inout_alloc();
  
      AVRational time_base = fmt_ctx->streams[video_stream_index]->time_base;
  
      enum AVPixelFormat pix_fmts[] = { dec_ctx->pix_fmt, AV_PIX_FMT_NONE };
  
      *filter_graph = avfilter_graph_alloc();
  
      if (!outputs || !inputs || !filter_graph) {
  
          ret = AVERROR(ENOMEM);
  
          goto end;
  
      }
  
  
  
      /* buffer video source: the decoded frames from the decoder will be inserted here. */
  
      snprintf(args, sizeof(args),
  
          "video_size=%dx%d:pix_fmt=%d:time_base=%d/%d:pixel_aspect=%d/%d",
  
          dec_ctx->width, dec_ctx->height, dec_ctx->pix_fmt,
  
          time_base.num, time_base.den,
  
          dec_ctx->sample_aspect_ratio.num, dec_ctx->sample_aspect_ratio.den);
  
  
  
      ret = avfilter_graph_create_filter(buffersrc_ctx, buffersrc, "in",
  
          args, NULL, *filter_graph);
  
      if (ret < 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot create buffer source\n");
  
          goto end;
  
      }
  
  
  
      /* buffer video sink: to terminate the filter chain. */
  
      ret = avfilter_graph_create_filter(buffersink_ctx, buffersink, "out",
  
          NULL, NULL, *filter_graph);
  
      if (ret < 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot create buffer sink\n");
  
          goto end;
  
      }
  
  
  
      ret = av_opt_set_int_list(*buffersink_ctx, "pix_fmts", pix_fmts,
  
          AV_PIX_FMT_NONE, AV_OPT_SEARCH_CHILDREN);
  
      if (ret < 0) {
  
          av_log(NULL, AV_LOG_ERROR, "Cannot set output pixel format\n");
  
          goto end;
  
      }
  
  
  
      /*
  
      * Set the endpoints for the filter graph. The filter_graph will
  
      * be linked to the graph described by filters_descr.
  
      */
  
  
  
      /*
  
      * The buffer source output must be connected to the input pad of
  
      * the first filter described by filters_descr; since the first
  
      * filter input label is not specified, it is set to "in" by
  
      * default.
  
      */
  
      outputs->name = av_strdup("in");
  
      outputs->filter_ctx = *buffersrc_ctx;
  
      outputs->pad_idx = 0;
  
      outputs->next = NULL;
  
  
  
      /*
  
      * The buffer sink input must be connected to the output pad of
  
      * the last filter described by filters_descr; since the last
  
      * filter output label is not specified, it is set to "out" by
  
      * default.
  
      */
  
      inputs->name = av_strdup("out");
  
      inputs->filter_ctx = *buffersink_ctx;
  
      inputs->pad_idx = 0;
  
      inputs->next = NULL;
  
  
  
      if ((ret = avfilter_graph_parse_ptr(*filter_graph, filters_descr,
  
          &inputs, &outputs, NULL)) < 0)
  
          goto end;
  
  
  
      if ((ret = avfilter_graph_config(*filter_graph, NULL)) < 0)
  
          goto end;
  
  
  
  end:
  
      avfilter_inout_free(&inputs);
  
      avfilter_inout_free(&outputs);
  
  
  
      return ret;
  
  }

  I can use the same kind of filter strings that would be used on the console. This code later applies the filter to the current frame (this->pVideoFrame) :

  if (av_buffersrc_add_frame_flags(buffersrc_ctx, this->pVideoFrame, AV_BUFFERSRC_FLAG_KEEP_REF) < 0) {
  
      av_log(NULL, AV_LOG_ERROR, "Error while feeding the filtergraph\n");
  
      break;
  
  }
  
  
  
  /* pull filtered frames from the filtergraph */
  
  while (1) {
  
      int ret = av_buffersink_get_frame(buffersink_ctx, this->pVideoFilterFrame);
  
      if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
  
      {
  
          break;
  
      }
  
  }

  In order for this filter to work with the overlay filter specification I would need to get the image into the system somehow and I have no idea how to do this.

  Ideally I would like to feed the system with an in memory buffer. But, if necessary, I can write the generated image to a tempfile first. The image will be valid for at least a second before the number changes. The C++ library that uses FFMPEG to manipulate and transmit the image is part of a larger .NET project.

  Any help would be appreciated.

  Regards,
  Ludwig

• X264 Encoder API

  16 avril 2014, par user1884325

  I’m studying the X264 API for encoding images.

  So far I’ve built the X264 library and the following code snippet shows how far I am :

    int frame_size;
  
    x264_t* encoder;
  
    x264_picture_t pic_in, pic_out;
  
    x264_param_t x264Param;
  
    int fps = 20;
  
    int width = 1280;
  
    int height = 720;
  
    x264_nal_t* nals;
  
    int i_nals;
  
  
  
    x264_param_default_preset(&x264Param, "veryfast", "zerolatency");
  
    x264Param.i_threads = 1;
  
    x264Param.i_width = 1280;
  
    x264Param.i_height = 720;
  
    x264Param.i_fps_num = fps;
  
    x264Param.i_fps_den = 1;
  
    x264Param.i_keyint_max = fps;
  
    x264Param.b_intra_refresh = 1;
  
    x264Param.rc.i_rc_method = X264_RC_CRF;
  
    x264Param.rc.f_rf_constant = 25;
  
    x264Param.rc.f_rf_constant_max = 35;
  
    x264Param.b_repeat_headers = 1;
  
    x264Param.b_annexb = 1;
  
    x264_param_apply_profile(&x264Param, "baseline");
  
  
  
    encoder = x264_encoder_open(&x264Param);
  
  
  
    x264_picture_alloc(&pic_in, X264_CSP_BGR, width, height);
  
  
  
    /* How to fill in bitmap data? */
  
  
  
    frame_size = x264_encoder_encode(encoder, &nals, &i_nals, &pic_in, &pic_out);
  
    if (frame_size >= 0)
  
    {
  
        printf("OK\n");
  
    }

  So I’m trying to encode a 24bit BGR bitmap image. However, the x264 header file doesn’t show any API function for writing the bitmap image to the encoder. How is this done ?

  EDIT

  This code snippet seems to work. I would appreciate a review and some comments. Thanks.

    int frame_size;
  
    int accum_frame_size;
  
    x264_t* encoder;
  
    x264_picture_t pic_in, pic_out;
  
    x264_param_t x264Param;
  
    int fps = 20;
  
    int width = 1280;
  
    int height = 720;
  
    x264_nal_t* nals;
  
    int i_nals;
  
    int64_t frameCount = 0;
  
    int k;
  
  
  
    for (k = 0; k < (1280*3*720); k++)
  
    {
  
       bgr[k] = rand();
  
    }
  
  
  
    x264_param_default_preset(&x264Param, "veryfast", "zerolatency");
  
    x264Param.i_threads = 1;
  
    x264Param.i_width = 1280;
  
    x264Param.i_height = 720;
  
    x264Param.i_fps_num = fps;
  
    x264Param.i_fps_den = 1;
  
    x264Param.i_keyint_max = fps;
  
    x264Param.b_intra_refresh = 1;
  
    x264Param.rc.i_rc_method = X264_RC_CRF;
  
    x264Param.i_csp = X264_CSP_BGR;
  
    x264Param.rc.f_rf_constant = 25;
  
    x264Param.rc.f_rf_constant_max = 35;
  
    x264Param.b_repeat_headers = 1;
  
    x264Param.b_annexb = 1;
  
    x264_param_apply_profile(&x264Param, "baseline");
  
  
  
    encoder = x264_encoder_open(&x264Param);
  
  
  
    x264_picture_alloc(&pic_in, X264_CSP_BGR, width, height);
  
  
  
    /* Load 24-bit BGR bitmap */
  
    pic_in.img.i_csp = X264_CSP_BGR;
  
    pic_in.img.i_plane = 1;
  
    pic_in.img.i_stride[0] = 3 * 1280;
  
    pic_in.img.plane[0] = bgr;
  
    pic_in.i_pts = frameCount;
  
    pic_in.i_type = X264_TYPE_AUTO;
  
    pic_out.i_pts = frameCount;
  
  
  
    /* Returns a frame size of 912 for first frame in this case */
  
    frame_size = x264_encoder_encode(encoder, &nals, &i_nals, &pic_in, &pic_out);
  
  
  
    printf("Decoder returned frame size = %d \n", frame_size);
  
    printf("Decoder returned %d NAL units \n", i_nals);
  
    if (frame_size >= 0)
  
    {
  
       int i;
  
       int j;
  
  
  
  
  
       accum_frame_size = 0;
  
       for (i = 0; i < i_nals; i++)
  
       {
  
          printf("******************* NAL %d (%d bytes) *******************\n", i, nals[i].i_payload);
  
          for (j = 0; j < nals[i].i_payload; j++)
  
          {
  
             if (j == 0) printf("First 10 bytes: ");
  
             if (j < 10) printf("%02X |", nals[i].p_payload[j]);
  
             accum_frame_size++;
  
          }
  
          printf("\n");
  
  
  
       }
  
    }
  
  
  
    printf("Verified frame size = %d \n", accum_frame_size);

  EDIT #2
  The encoder outputs this :

       x264 [error]: baseline profile doesn't support 4:4:4
  
       x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
  
       x264 [info]: profile High 4:4:4 Predictive, level 3.1, 4:4:4 8-bit
  
       Decoder returned frame size = 1467194
  
       Decoder returned 4 NAL units
  
       ******************* NAL 0 (31 bytes) *******************
  
       First 10 bytes: 00 |00 |00 |01 |67 |F4 |00 |1F |91 |89 |
  
       ******************* NAL 1 (8 bytes) *******************
  
       First 10 bytes: 00 |00 |00 |01 |68 |EF |1F |2C |
  
       ******************* NAL 2 (595 bytes) *******************
  
       First 10 bytes: 00 |00 |01 |06 |05 |FF |FF |4C |DC |45 |
  
       ******************* NAL 3 (1466560 bytes) *******************
  
       First 10 bytes: 00 |00 |01 |65 |88 |82 |0A |FF |F5 |B0 |
  
       Verified frame size = 1467194

  Isn’t each NAL unit supposed to start with 0x00 0x00 0x00 0x01 ?

  szatmary : I appreciate your valuable feedback. So you’re saying that each NAL unit does not necessarily start with 0,0,0,1. However, I’m a bit unclear on your answer. Are you implying that with a certain configuration the NAL units will start with 0,0,0,1 ? If so, which configuration is that ? I need to make sure that each NAL unit I transmit out on the network to a remote receiver starts with 0,0,0,1. Prior to exploring the x264 library I was using the x264 exe and piped BMP data in and encoded data out from the x264 process. I then parsed the encoder output and looked for NAL units by looking for 0,0,0,1. How do I accomplish the same with the x264 library ?

  Regarding libswscale :

  I downloaded the ffmpeg source and ran configure and make in MINGW. After the process had completed I couldn’t find anything but a number of .exe files. How do I build actual static libraries (.lib) which I can use in a Visual Studio project ?

• Why is ffmpeg biased when converting to 16-bit integer or 32-bit float RGB output ?

  8 mars 2023, par F.X.

  Consider the following Python invocations, that exercise ffmpeg's ability to convert a 8-bit input into 16-bit integer values or 32-bit float values :

  


  import cv2
import subprocess
import numpy as np

ffmpeg = "ffmpeg -hide_banner -loglevel error -y"
flags = "-sws_flags accurate_rnd+bitexact+full_chroma_int+neighbor -sws_dither none"

for input_color_range in ("tv", "pc"):

    # Generate YUV444 video and encode it losslessly
    subprocess.check_call(rf"{ffmpeg} -y {flags} -color_range {input_color_range} -f lavfi -i yuvtestsrc {flags} -pix_fmt yuv444p -color_range {input_color_range} -x264-params qp=0 -frames 1 -c:v libx264 video_temp.mp4")

    # Extract YUV444 frame, as well as 8-bit int, 16-bit int and 32-bit float frame
    subprocess.check_call(rf"{ffmpeg} -y {flags} -i video_temp.mp4 {flags} -f rawvideo video_temp_444_frame1.yuv")
    subprocess.check_call(rf"{ffmpeg} -y {flags} -i video_temp.mp4 {flags} -color_range pc -pix_fmt rgb24 video_temp_444_frame1_8u.png")
    subprocess.check_call(rf"{ffmpeg} -y {flags} -i video_temp.mp4 {flags} -color_range pc -pix_fmt rgb48be video_temp_444_frame1_16u.png")
    subprocess.check_call(rf"{ffmpeg} -y {flags} -i video_temp.mp4 {flags} -color_range pc -pix_fmt gbrpf32be -f rawvideo video_temp_444_frame1_32f.gbrpf32be")

    # Reead these frames into (height, width, 3) Numpy arrays containing YUV or RGB data
    data_8u = cv2.imread("video_temp_444_frame1_8u.png", cv2.IMREAD_UNCHANGED)[..., ::-1]
    data_16u = cv2.imread("video_temp_444_frame1_16u.png", cv2.IMREAD_UNCHANGED)[..., ::-1]
    data_yuv = np.rollaxis(np.frombuffer(open("video_temp_444_frame1.yuv", "rb").read()).view(np.uint8).reshape((3, 240, 320)), 0, 3).copy()
    data_32f = np.rollaxis(np.frombuffer(open("video_temp_444_frame1_32f.gbrpf32be", "rb").read()).view(np.dtype(">f4")).reshape((3, 240, 320)), 0, 3).copy()
    data_32f[..., (0, 1, 2)] = data_32f[..., (2, 0, 1)]

    # This pixel in yuvtestsrc corresponds to limited-range YUV=(235, 128,
    # 128)=(100%, 0.0, 0.0), which should correspond to 100% white, i.e.
    # RGB=(100%, 100%, 100%).
    if input_color_range == "tv":
        i, j = 10, 294
    # This pixel in yuvtestsrc corresponds to full-range YUV=(255, 128,
    # 128)=(100%, 0.0, 0.0), which should correspond to 100% white, i.e.
    # RGB=(100%, 100%, 100%).
    elif input_color_range == "pc":
        i, j = 10, 319
    else:
        raise Exception(input_color_range)

    # Print pixel values
    print("")
    print(f"Values for {input_color_range}-range input video at ({i}, {j}):")
    print("- 8-bit YUV input   = %s" % data_yuv[i, j, :])
    print("- 8-bit RGB output  = %s (= %s)" % (data_8u[i, j, :], data_8u[i, j, :] / 255))
    print("- 16-bit RGB output = %s (= %s)" % (data_16u[i, j, :], data_16u[i, j, :] / 65535))
    print("- Float RGB output  = %s" % data_32f[i, j, :])


  


  The script first generates a video frame from ffmpeg's YUV test source, encoded with no chroma subsampling (4:4:4) as losslessly as possible.

  


  That video is then used as a reference source to extract the following frames :

  


  

  • The input reference YUV data

  


  • A 8-bit RGB conversion

  


  • A 16-bit RGB conversion

  


  • A 32-bit floating-point RGB conversion

  


  


  One pixel is extracted from each frame, which should contain a 100% white value. The output of the last series of print statements is the following :

  


  Values for tv-range input video at (10, 294):
- 8-bit YUV input   = [235 128 128]
- 8-bit RGB output  = [255 255 255] (= [1. 1. 1.])
- 16-bit RGB output = [65283 65283 65283] (= [0.99615473 0.99615473 0.99615473])
- Float RGB output  = [0.9961547 0.9961547 0.9961547]

Values for pc-range input video at (10, 319):
- 8-bit YUV input   = [255 128 128]
- 8-bit RGB output  = [255 255 255] (= [1. 1. 1.])
- 16-bit RGB output = [65280 65280 65280] (= [0.99610895 0.99610895 0.99610895])
- Float RGB output  = [0.99610895 0.99610895 0.99610895]


  


  While the 8-bit RGB output values are correct, none of the others correctly output a 100% white signal. Why is that ?