Recherche avancée

Sur d’autres sites (15639)

• Missing piece between libjpeg-turbo & h264 ffmpeg C/C++

  15 octobre 2022, par Nelstaar

  On the left side I have a buffer with decoded pixels that I can get in two formats :

  


  RGB interleaved/packed where bytes in buffer are R0G0B0R1G1B1....

  


  or

  


  YUV444 interleaved/packed where bytes in buffer are Y0U0V0Y1U1V1...

  


  (JCS_RGB or JCS_YCbCr in jpeglib.h)

  


  (Please note that I use libjpeg-turbo because I need to decompress a cropped region of the image. (jpeg_crop_scanline()))

  


  On the right side I have x264 codec via ffmpeg that support only planar pixel formats :

  


  yuv420p, yuvj420p, yuv422p, yuvj422p, yuv444p, yuvj444p, nv12, nv16, nv21, yuv420p10le, yuv422p10le, yuv444p10le, nv20le

  


  yuv444p where bytes in buffer are Y0Y1Y2...U0U1...V0V1...

  


  according to ffmpeg -h encoder=libx264

  


  I have some ideas already :

  


  

  • Decompress Jpeg to RBG888 in buffer 1 then libswscale to yuv420p in buffer 2 and encoding. (copy)

  


  • Decompress Jpeg to YUV444 interleaved in buffer 1 then SSSE3 magic in buffer 1 to yuv444p and encoding. (no copy)

  


  • or else.

  


  


  What would be the most effective fastest way ?

  


  I which to avoid buffer copy.

  


  Movie have the same width & height than Jpegs.

  


• lavc/vaapi_decode : add missing flag when picking best pixel format

  5 août 2022, par Philip Langdale

  lavc/vaapi_decode : add missing flag when picking best pixel format
  vaapi_decode_find_best_format currently does not set the
  
  VA_SURFACE_ATTRIB_SETTABLE flag on the pixel format attribute that it
  
  returns.
  Without this flag, the attribute will be ignored by vaCreateSurfaces,
  
  meaning that the driver's default logic for picking a pixel format will
  
  kick in.
  So far, this hasn't produced visible problems, but when trying to
  
  decode 4:4:4 content, at least on Intel, the driver will pick the
  
  444P planar format, even though the decoder can only return the AYUV
  
  packed format.
  The hwcontext_vaapi code that sets surface attributes when picking
  
  formats does not have this bug.
  Applications may use their own logic for finding the best format, and
  
  so may not hit this bug. eg : mpv is unaffected.
  

  • [DH] libavcodec/vaapi_decode.c

• Upsample and encode audio stream

  8 juin 2022, par Anton Issaikin

  Basically after transcoding pcm_alaw 8khz to mp3, I can hear only some brief or even swift sound in first 1-2 seconds, unrecognizable sound. So something is wrong with pts/dts, packed to planar convertion, or upsampling.

  


  My application does transcoding rtsp camera stream to file. Video and audio. Video works fine and audio remuxing as well. Now I have pcm_alaw 8khz audio stream and want to transcode it to mp4 file along with video.

  


  Code is quite cumbersome to construct reproducible part, so firstly I want to know if my logic is right. Here is my draft process (assume all error are checked and handled) :

  


  create encoder :

  


      codec_ = avcodec_find_encoder(AV_CODEC_ID_MP3);

    enc_ctx_ = avcodec_alloc_context3(codec_);

    enc_ctx_->bit_rate = 64000;
    enc_ctx_->codec_type = AVMEDIA_TYPE_AUDIO;

    enc_ctx_->sample_fmt   = codec_->sample_fmts ? codec_->sample_fmts[0] : AV_SAMPLE_FMT_S32P;

    // functions from here https://www.ffmpeg.org/doxygen/4.1/encode_audio_8c-example.html
    enc_ctx_->sample_rate    = select_sample_rate(codec_);
    enc_ctx_->channel_layout = select_channel_layout(codec_);
    enc_ctx_->channels       = av_get_channel_layout_nb_channels(enc_ctx_->channel_layout);
    enc_ctx_->time_base = (AVRational){1, enc_ctx_->sample_rate};
    enc_ctx_->strict_std_compliance = FF_COMPLIANCE_EXPERIMENTAL;

    if (is_global_header) {
        enc_ctx_->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
    }

    avcodec_open2(enc_ctx_, codec_, nullptr);


  


  create resampler (in_frame) :

  


      audio_fifo_ = av_audio_fifo_alloc(enc_ctx_->sample_fmt, enc_ctx_->channels, 1));
       
    in_ch_layout_ = in_frame->channel_layout;
    in_sample_fmt = in_frame->format;
    in_sample_rate_ = in_frame->sample_rate;

    swr_ctx_ = swr_alloc_set_opts(NULL,                       // we're allocating a new context
                             enc_ctx_->channel_layout,        // out_ch_layout
                             enc_ctx_->sample_fmt,            // out_sample_fmt
                             enc_ctx_->sample_rate,           // out_sample_rate
                             in_frame->channel_layout,        // in_ch_layout
                             (AVSampleFormat)in_frame->format, // in_sample_fmt
                             in_frame->sample_rate,            // in_sample_rate
                             0,                                // log_offset
                             NULL);                            // log_ctx
                             
    swr_init(swr_ctx_);


  


  resample (in_frame, start_pts, start_dts) :

  


      auto resampled_frame = av_frame_alloc();&#xA;&#xA;    auto dst_nb_samples = av_rescale_rnd(swr_get_delay(swr_ctx_, in_frame->sample_rate) &#x2B;&#xA;                                    in_frame->nb_samples, enc_ctx_->sample_rate, in_frame->sample_rate, AV_ROUND_UP);&#xA;&#xA;    // resampled_frame->nb_samples     = dst_nb_samples;&#xA;    resampled_frame->format         = enc_ctx_->sample_fmt;&#xA;    resampled_frame->channel_layout = enc_ctx_->channel_layout;&#xA;    // resampled_frame->channels       = enc_ctx_->channels;&#xA;    resampled_frame->sample_rate    = enc_ctx_->sample_rate;&#xA;&#xA;    error = swr_convert_frame(swr_ctx_, resampled_frame, in_frame);&#xA;&#xA;    /* Make the FIFO as large as it needs to be to hold both,&#xA;     * the old and the new samples. */&#xA;    if (av_audio_fifo_size(audio_fifo_) &lt; dst_nb_samples) {&#xA;        av_audio_fifo_realloc(audio_fifo_, dst_nb_samples);&#xA;    }&#xA;&#xA;    /* Store the new samples in the FIFO buffer. */&#xA;    auto nb_samples = av_audio_fifo_write(audio_fifo_,&#xA;                                          reinterpret_cast<void>(resampled_frame->extended_data),&#xA;                                          resampled_frame->nb_samples);&#xA;&#xA;&#xA;    int delay = 0;&#xA;    // trying to split resampled frame to desired chunks&#xA;    while (av_audio_fifo_size(audio_fifo_) > 0) {&#xA;        const int frame_size = FFMIN(av_audio_fifo_size(audio_fifo_), enc_ctx_->frame_size);&#xA;&#xA;        auto out_frame = av_frame_alloc();&#xA;&#xA;&#xA;        out_frame->nb_samples       = frame_size;&#xA;        out_frame->format           = enc_ctx_->sample_fmt;&#xA;        out_frame->channel_layout   = enc_ctx_->channel_layout;&#xA;        out_frame->channels         = enc_ctx_->channels;&#xA;        out_frame->sample_rate      = enc_ctx_->sample_rate;&#xA;&#xA;        av_frame_get_buffer(out_frame, 0);&#xA;        &#xA;        av_audio_fifo_read(audio_fifo_, (void **)out_frame->data, frame_size) &lt; frame_size);&#xA;&#xA;    // ***** tried both cases&#xA;        out_frame->pts = in_frame->pts &#x2B; delay;&#xA;        out_frame->pkt_dts = in_frame->pkt_dts &#x2B; delay;&#xA;        // swr_next_pts(swr_ctx_, in_frame->pts) &#x2B; delay;&#xA;        // swr_next_pts(swr_ctx_, in_frame->pkt_dts) &#x2B; delay;&#xA;&#xA;        result.push_back(out_frame);&#xA;&#xA;        delay &#x2B;= frame_size;&#xA;    }&#xA;&#xA;    return result;&#xA;</void>

  


  encoding and muxing (in_frame) :

  


      bool DoesNeedResample(const AVFrame * in_frame) {
        assert(("DoesNeedResample: in_frame is empty", in_frame));
        assert(("DoesNeedResample: encoder is not started", is_init_));

        if (in_frame->sample_rate != enc_ctx_->sample_rate ||
        in_frame->channel_layout != enc_ctx_->channel_layout ||
        in_frame->channels != enc_ctx_->channels ||
        in_frame->format != enc_ctx_->sample_fmt) {
        return true;
        }

        return false;
    }

    av_frame_make_writable(in_frame);


    streamserver::AVFrames encoding_frames;
    if (DoesNeedResample(in_frame)) {
        encoding_frames = Resample(in_frame, 
        av_rescale_q(in_frame->pts, in_audio_stream_timebase_, out_audio_stream_->time_base),
        av_rescale_q(in_frame->pkt_dts, in_audio_stream_timebase_, out_audio_stream_->time_base));
    } else {
        encoding_frames.push_back(av_frame_clone(in_frame));
    }


    for (auto frame : encoding_frames) {
        if ((err = avcodec_send_frame(encoder_ctx, frame)) < 0) {
            AVFrameFree(&frame);
        }

        while (err >= 0) {
            pkt_->data = NULL;
            pkt_->size = 0;
            av_init_packet(pkt_);

            err = avcodec_receive_packet(encoder_ctx, pkt_);
            if (err == AVERROR(EAGAIN) || err == AVERROR_EOF) {
                break;
            } else if (err < 0) {
                break;
            }

            pkt_->stream_index = out_audio_stream_->index;

            av_interleaved_write_frame(ofmt_ctx_, pkt_);
        }

        av_packet_unref(pkt_);
    }


  


  Sound in resulted video is corrupted, see first paragraph for description.

  


  In https://www.ffmpeg.org/doxygen/4.1/transcode_aac_8c-example.html
there are lines :

  


          /*
        * Perform a sanity check so that the number of converted samples is
        * not greater than the number of samples to be converted.
        * If the sample rates differ, this case has to be handled differently
        */
        av_assert0(output_codec_context->sample_rate == input_codec_context->sample_rate);


  


  How to handle such cases ? I tried to split resampled frames via fifo in example above !