Recherche avancée

Sur d’autres sites (16290)

• Revision 3273 : On utilise notation et on définit un critère _dist pour notation plutot

  18 avril 2010, par kent1 — Log

  On utilise notation et on définit un critère _dist pour notation plutot

• Glitchy audio or broken video in fragmented MP4

  6 septembre 2022, par PookyFan

  I'm working on small C++ library for muxing audio and video. This is basically a facade for FFMPEG functions and structures. The code is here with minimal reproduction testing code here and as of now it seems like it's working fine... almost.

  


  For the record - my MP4 file is so-called "fragmented MP4", with headers moved at the beginning of the file in a way that would allow to stream that file (ie. play it in a browser while it's being buffered). That's what these movflags I'm setting in Mp4Muxer::writeHeader() are for.

  


  While testing this library with raw H264 video stream and MP3 file (video is <1 min long, MP3 - a few minutes long), I observed that :

  &#xA;

  &#xA;
  • if I don't limit muxing audio stream when it's way "ahead" of the video (and it will be since MP3 is longer, so eventually video frames stop coming but audio frames still come in), all muxes just fine with no errors, but playing output MP4 with ffplay after just a few seconds results in the following log (and also frozen video, while audio keeps playing) :

  &#xA;

  &#xA;

  [h264 @ 0x7f90a40ae2c0] Invalid NAL unit size (2162119 > 76779).0
[h264 @ 0x7f90a40ae2c0] Error splitting the input into NAL units.
[mp3float @ 0x7f90a4009540] Header missing  515KB sq=    0B f=0/0
[h264 @ 0x7f90a40cb0c0] Invalid NAL unit size (-860010620 > 17931).
[h264 @ 0x7f90a40cb0c0] Error splitting the input into NAL units.
[h264 @ 0x7f90a42bf440] Invalid NAL unit size (-168012642 > 8000).
[h264 @ 0x7f90a42bf440] Error splitting the input into NAL units.
[h264 @ 0x7f90a42fa780] Invalid NAL unit size (-1843711407 > 5683).
[ and it repeats...]


  &#xA;

  &#xA;
  • even if I limit how much a stream can be "ahead" of the other, limiting it too much results in no muxed data in the output

  &#xA;

  • any other intermediate level of limiting how much one stream can be buffered in muxer relative to the other stream results in glitchy audio, with the following errors popping out every now and then in ffplay (the more strict limit is, the more often they are printed) :

  &#xA;

  &#xA;

  [mp3float @ 0x7f744c01b640] overread, skip -6 enddists: -1 -1=0/0 


  &#xA;

  Not limitting muxed audio (at all or enough) relative to muxed video also results in following messages in my muxing application :

  &#xA;

  [mp4 @ 0x55d0c6c21940] Delay between the first packet and last packet in the muxing queue is 10004898 > 10000000: forcing output


  &#xA;

  For now, the fix is quite ugly and I don't even understand why it works, but before writting MP4 header I manually set a limit for frames buffered by muxer, like so :

  &#xA;

  formatCtxt->max_interleave_delta = 10000000LL * 10LL;


  &#xA;

  This way the muxer can store more packets of one stream that's way "ahead" of the other (maximum difference between DTS of the packets at the beginning and at the end of queue is set to 10x larger than default ; it also gets rid of information log mentioned above). Obviously, I'd like to resolve it more properly, without hacking things like that.

  &#xA;

  I was trying various things, including manual skipping of ID3 tags in MP3 file (but seems like FFMPEG handles them just fine and it didn't change anything). I was also experimenting with FLAC in MP4 instead of MP3. and while I know it's rather experimental thing, I encountered very similar problems with glitching audio (no problem with video being frozen when lots of audio data gets muxed, though). It also seems that problem with glitching audio or frozen video varies in scale depending on how large are input data chunks that I feed muxer with. For now, honestly, I'm out of ideas.

  &#xA;

• The hardware decoding was successful, but the hw_frames_ctx in the received frame is empty

  15 juillet 2024, par mercuric taylor

  I tried to use QSV hardware decoding under ffmpeg, using the integrated graphics 730 on my computer. Here's the code I used to initialize the decoder

  &#xA;

  const AVCodec* codec = NULL;
int ret;
int err = 0;
// Create the QSV hardware device.
    ret = av_hwdevice_ctx_create(&hw_device_ctx, AV_HWDEVICE_TYPE_QSV, "auto", NULL, 0);
    if (ret < 0)
    {
        char error_string[AV_ERROR_MAX_STRING_SIZE];
        av_make_error_string(error_string, AV_ERROR_MAX_STRING_SIZE, ret);
        LError("Error creating QSV device: {}", error_string);
        return NULL;
    }
 // Search for QSV decoders, either for H.264 or H.265.
    codec = avcodec_find_decoder_by_name(codec_name);
    if (!codec)
    {
        LError("Failed to find QSV decoder.");
        return NULL;
    }

    // Creating a decoder context and associating it with the hardware device.
    decoder_ctx = avcodec_alloc_context3(codec);
    if (!decoder_ctx)
    {
        ret = AVERROR(ENOMEM);
        LError("Failed to allocate decoder context.\n");
        return NULL;
    }
    decoder_ctx->codec_id = AV_CODEC_ID_H264;  
    decoder_ctx->opaque = &hw_device_ctx;
    decoder_ctx->get_format = get_format;
// Open the decoder.
    if ((ret = avcodec_open2(decoder_ctx, NULL, NULL)) < 0)
    {
        LError("Failed to open decoder: %d\n", ret);
        return NULL;
    }

    parser_ctx = av_parser_init(avcodec_find_encoder_by_name(codec_name)->id);


  &#xA;

  The following is the process of decoding using the decoder :

  &#xA;

  AVFrame* frame = av_frame_alloc();
    AVFrame* dstFrame = av_frame_alloc();
    res = avcodec_send_packet(decoder_ctx, pkt);
    if (res < 0)
    {
        return;
    }
    int num = 0;
    while (res >= 0)
    {
        res = avcodec_receive_frame(decoder_ctx, frame);

        if (res == AVERROR(EAGAIN) || res == AVERROR_EOF)
        {
            //if (res == AVERROR(EAGAIN)) 
            //{
            //   LInfo("AVERROR(EAGAIN):");
            //}
            //if (res == AVERROR_EOF) 
            //{
            //  //  LInfo("AVERROR_EOF");
            //}
           // av_frame_unref(frame);
            break;
        }
        else if (res < 0)
        {
          //  av_frame_unref(frame);
            return;
        }


        frameNumbers_++;
        if (frame->hw_frames_ctx == NULL)
        {
            LError("hw_frames_ctx is null");
            LError("avcodec_receive_frame return is {}", res);
        }


  &#xA;

  My issue is that I've successfully decoded the video. The return value of avcodec_receive_frame is 0, and the width and height of the AVFrame are the same as the input video stream.

  &#xA;

  However,** the hw_frames_ctx field of the AVFrame is empty**. Why would this happen in a successful hardware decoding scenario ?

  &#xA;

  Could it be due to some incorrect configurations ? I've set up a get_format function like this

  &#xA;

  static enum AVPixelFormat get_format(AVCodecContext *avctx, const enum AVPixelFormat *pix_fmts)
{
    while (*pix_fmts != AV_PIX_FMT_NONE) {
        if (*pix_fmts == AV_PIX_FMT_QSV) {
            DecodeContext *decode = (DecodeContext*)avctx->opaque;
            AVHWFramesContext  *frames_ctx;
            AVQSVFramesContext *frames_hwctx;
            int ret;
            /* create a pool of surfaces to be used by the decoder */
            avctx->hw_frames_ctx = av_hwframe_ctx_alloc(decode->hw_device_ref);
            if (!avctx->hw_frames_ctx)
                return AV_PIX_FMT_NONE;
            frames_ctx = (AVHWFramesContext*)avctx->hw_frames_ctx->data;
            frames_hwctx = (AVQSVFramesContext*)frames_ctx->hwctx;
            frames_ctx->format = AV_PIX_FMT_QSV;
            frames_ctx->sw_format = avctx->sw_pix_fmt;
            frames_ctx->width = FFALIGN(avctx->coded_width, 32);
            frames_ctx->height = FFALIGN(avctx->coded_height, 32);
            frames_ctx->initial_pool_size = 32;
            frames_hwctx->frame_type = MFX_MEMTYPE_VIDEO_MEMORY_DECODER_TARGET;
            ret = av_hwframe_ctx_init(avctx->hw_frames_ctx);
            if (ret < 0)
                return AV_PIX_FMT_NONE;
            return AV_PIX_FMT_QSV;
        }
        pix_fmts++;
    }
    fprintf(stderr, "The QSV pixel format not offered in get_format()\n");
    return AV_PIX_FMT_NONE;
}


  &#xA;

  But I also noticed that even though I set decoder_ctx->get_format = get_format ; this function is not being executed later on.

  &#xA;

  I observed that my GPU is also being utilized during program execution, indicating a successful hardware decoding. My subsequent goal is to render a frame from this decoded AVFrame. It seems like the hw_frames_ctx of the AVFrame is a texture handle on the GPU. I wish to directly use this field for D3D11 rendering and display it on the screen.&#xA;My questions are :

  &#xA;

  &#xA;
  1. Is the hw_frames_ctx field empty in the case of successful hardware decoding ?

  &#xA;

  2. Does it represent a texture handle on the GPU ?

  &#xA;

  3. If my rendering approach is wrong, how can I correctly render this AVFrame using D3D11 ?

  &#xA;

  &#xA;