Advanced search

On other websites (5526)

• ffmpeg live stream transcoding. A/V sync issues on fast camera movement

  21 August 2020, by Kelsnare
  

  1. I create a webrtc peer connection with my server(only stun)

  


  2. Using pion webrtc for the server

  


  3. I write the received RTP packets as VP8 and opus streams, as described here, to two pipes (the writers; created with os.Pipe() in golang)

  


  4. The read ends of these two pipes are received by ffmpeg as inputs (via exec.Command.ExtraFiles) for transcoding using libx264 and aac into a single stream. The command:

  


  


  ffmpeg -re -i pipe:3 -re -r pipe:4 -c:a aac -af aresample=48000 -c:v libx264 -x264-params keyint=48:min-keyint=24 -profile:v main -preset ultrafast -tune zerolatency -crf 20 -fflags genpts -avoid_negative_ts make_zero -vsync vfr -map 0:0,0:0 -map 1:0,0:0 -f matroska -strict -2 pipe:5


  


  

  5. The above command outputs to a pipe(:5) the read end of which is being taken as input by the following:

  


  


  ffmpeg -hide_banner -y -re -i pipe:3 -sn -vf scale=-1:'min(ih,360)' -c:v libx264 -pix_fmt yuv420p -ca aac -b:a 128k -b:v 1400k -maxrate 1498k -bufsize 2100k -hls_time 1 -hls_playlist_type event -hls_base_url /workdir/streamID/360p -hls_segment_filename /workdir/streamID/360p/360_%%03d.ts -f hls /workdir/streamID/360p.m3u8


  


  

  5. This works fine as long as there are no movements of my webcam. The moment that happens the video speed suddenly increases for a split second and audio delay gets introduced. This delay keeps increasing each time I shake my webcam.

  


  


  The first command in point 4 above - if written to a file separately will be absolutely fine, in terms of a/v sync, even with vigorous camera shaking. The weird audio delay is only when transcoding for hls output irrespective of whether I'm actually viewing it live or playing it back later.

  


  This is my first time working with ffmpeg/hls/webrtc - would be really helpful if I could be pointed in the correct direction at least to be able to debug this or even know why this happens. Any and all help is greatly appreciated

  


• Issues with FFMPEG swresample and channel mapping

  19 March 2024, by Joseph Katrinka

  Ive linked FFMPEG libs to a C++ project Im working on in VS2017 and have had mostly no problems with it so far.
Im currently trying to use it to extract audio from a movie file and save it as a wav file.
I managed to do this just fine, and my next step is to make a "split into mono". To make 2 mono wav files, one from the left channel and one from the right channel of a stereo file.
My code works for the orignal usecase, stereo to stereo (one wav file) and also works for when Im just doing the left, but when doing the right it fails.

  


  Exception thrown at 0x00007FF9603F2D20 (swresample-4.dll) in Matchbox.exe: 0xC0000005: Access violation reading location 0x0000000000000000.

  


  This is the relevent code involved. It is failing inside the processing loop on line:
int ret = swr_convert_frame(swrCtx, resampledFrame, frame);

  


  /// FFMPEG init
    AVFormatContext* formatCtx = nullptr;
    AVCodecContext* codecCtx = nullptr;
    const AVCodec* codec = nullptr;
    SwrContext* swrCtx = nullptr;
    AVPacket* packet = nullptr;
    AVFrame* frame = nullptr;
    AVFrame* resampledFrame = nullptr;
    int audioStreamIndex = -1;

    int sampleRate;

    uint64_t in_channel_layout = AV_CH_LAYOUT_STEREO; /// we only care about stereo in for now
    uint64_t out_channel_layout = doMono ? AV_CH_LAYOUT_MONO : AV_CH_LAYOUT_STEREO;

    /// some redundancy in this stream setup/checking
    if (avformat_open_input(&formatCtx, movieFilePath.toRawUTF8(), nullptr, nullptr) < 0)
    {
        DBG("failed to open input file");
        return false;
    }
    if (avformat_find_stream_info(formatCtx, nullptr) < 0)
    {
        DBG("failed to find stream info");
        return false;
    }

    for (unsigned i = 0; i < formatCtx->nb_streams; i++) {
        if (formatCtx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_AUDIO)
        {
            audioStreamIndex = i;
            break;
        }
    }
    if (audioStreamIndex == -1)
    {
        DBG("audio stream not found");
        return false;
    }

    codec = avcodec_find_decoder(formatCtx->streams[audioStreamIndex]->codecpar->codec_id);
    if (!codec)
    {
        DBG("decoder not found for audio stream");
        return false;
    }

    codecCtx = avcodec_alloc_context3(codec);
    if (avcodec_parameters_to_context(codecCtx, formatCtx->streams[audioStreamIndex]->codecpar) < 0)
    {
        DBG("failed to copy codec params to codec context");
        return false;
    }
    if (avcodec_open2(codecCtx, codec, nullptr) < 0)
    {
        DBG("failed to open codec");
        return false;
    }

    if (codecCtx->channels != 2)
    {
        DBG("unsupported channel layout");
        return false;
    }

    if (!codecCtx->channel_layout)
    {
        codecCtx->channel_layout = av_get_default_channel_layout(codecCtx->channels);
    }

    sampleRate = codecCtx->sample_rate; /// using this later for timecode calculation

    packet = av_packet_alloc();
    frame = av_frame_alloc();
    resampledFrame = av_frame_alloc();

    /// the input frame requires this info
    frame->format = codecCtx->sample_fmt;
    frame->channel_layout = codecCtx->channel_layout;
    frame->sample_rate = codecCtx->sample_rate;

    // does the resampled frame really need this?
    resampledFrame->format = AV_SAMPLE_FMT_S16;
    resampledFrame->channel_layout = out_channel_layout;
    resampledFrame->sample_rate = codecCtx->sample_rate;

    if (!packet || !frame || !resampledFrame)
    {
        DBG("failed to allocate packed or frame");
        return false;
    }

    /// Set up swrCtx for channel mapping if we need it
    swrCtx = swr_alloc();
    av_opt_set_int(swrCtx, "in_channel_layout", codecCtx->channel_layout, 0);
    av_opt_set_int(swrCtx, "out_channel_layout", out_channel_layout, 0);
    av_opt_set_int(swrCtx, "in_sample_rate", codecCtx->sample_rate, 0);
    av_opt_set_int(swrCtx, "out_sample_rate", codecCtx->sample_rate, 0);
    av_opt_set_sample_fmt(swrCtx, "in_sample_fmt", codecCtx->sample_fmt, 0);
    av_opt_set_sample_fmt(swrCtx, "out_sample_fmt", AV_SAMPLE_FMT_S16, 0);

    /// channel mapping if we are doing mono
    if (doMono)
    {
        DBG("WE ARE DOING MONO");
        int in_channel_map[2];
        if (useLeft)
        {
            DBG("WE ARE DOING LEFT");
            in_channel_map[0] = AV_CH_FRONT_LEFT;
        }
        else
        {
            DBG("WE ARE DOING RIGHT");
            in_channel_map[0] = AV_CH_FRONT_RIGHT;
        }
        in_channel_map[1] = -1;
        swr_set_channel_mapping(swrCtx, in_channel_map);
    }
    

    if (swr_init(swrCtx) < 0)
    {
        DBG("failed to init resampler");
        return false;
    }

    AVFormatContext* outFormatCtx = nullptr;
    avformat_alloc_output_context2(&outFormatCtx, nullptr, nullptr, wavFilePath.toRawUTF8());
    if (!outFormatCtx)
    {
        DBG("failed to allocate output format context");
        return false;
    }

    AVStream* outStream = avformat_new_stream(outFormatCtx, nullptr);
    if (!outStream)
    {
        DBG("failed to create new stream for output");
        return false;
    }

    AVCodecContext* outCodecCtx = avcodec_alloc_context3(avcodec_find_encoder(AV_CODEC_ID_PCM_S16LE));
    if (!outCodecCtx)
    {
        DBG("failed to allocate output codec context");
        return false;
    }

    outCodecCtx->sample_rate = codecCtx->sample_rate;
    outCodecCtx->channel_layout = out_channel_layout;
    outCodecCtx->channels = doMono ? 1 : 2;
    outCodecCtx->sample_fmt = AV_SAMPLE_FMT_S16;
    outCodecCtx->bit_rate = 192000;
    outCodecCtx->time_base = (AVRational{ 1, outCodecCtx->sample_rate });

    if (avcodec_open2(outCodecCtx, avcodec_find_encoder(AV_CODEC_ID_PCM_S16LE), nullptr) < 0)
    {
        DBG("failed to open output codec");
        return false;
    }

    outStream->time_base = outCodecCtx->time_base;
    if (avcodec_parameters_from_context(outStream->codecpar, outCodecCtx) < 0)
    {
        DBG("failed to copy codec params from context to stream");
        return false;
    }

    if (!(outFormatCtx->oformat->flags & AVFMT_NOFILE))
    {
        if (avio_open(&outFormatCtx->pb, wavFilePath.toRawUTF8(), AVIO_FLAG_WRITE) < 0)
        {
            DBG("failed to open output file");
            return false;
        }
    }

    if (avformat_write_header(outFormatCtx, nullptr) < 0)
    {
        DBG("failed to write header to output file");
        return false;
    }

    int64_t total_samples = 0;

    while (av_read_frame(formatCtx, packet) >= 0)
    {
        if (packet->stream_index == audioStreamIndex)
        {
            if (avcodec_send_packet(codecCtx, packet) < 0)
            {
                DBG("failed secnding packet to decoder");
                av_packet_unref(packet);
                continue;
            }
            while (avcodec_receive_frame(codecCtx, frame) == 0)
            {
                int ret = swr_convert_frame(swrCtx, resampledFrame, frame);
                if (ret < 0)
                {
                    char errBuf[AV_ERROR_MAX_STRING_SIZE];
                    av_strerror(ret, errBuf, sizeof(errBuf));
                    DBG("Error resampling frame: " + std::string(errBuf));
                    continue;
                }

                resampledFrame->pts = av_rescale_q(total_samples, (AVRational{ 1, outCodecCtx->sample_rate }), outCodecCtx->time_base);
                total_samples += resampledFrame->nb_samples;

                AVPacket outPacket;
                av_init_packet(&outPacket);
                if (avcodec_send_frame(outCodecCtx, resampledFrame) < 0)
                {
                    DBG("error sending frame to decoder");
                    continue;
                }
                while (avcodec_receive_packet(outCodecCtx, &outPacket) == 0)
                {
                    outPacket.stream_index = outStream->index;
                    if (av_interleaved_write_frame(outFormatCtx, &outPacket) < 0)
                    {
                        av_packet_unref(&outPacket);
                        DBG("error writing packet to output file");
                        return false;
                    }
                    av_packet_unref(&outPacket);
                }
            }
        }
        av_packet_unref(packet);
    }

    /// trailer and cleanup
    av_write_trailer(outFormatCtx);

    avcodec_close(outCodecCtx);
    avcodec_close(codecCtx);
    avformat_close_input(&formatCtx);
    av_packet_free(&packet);
    av_frame_free(&frame);
    av_frame_free(&resampledFrame);
    swr_free(&swrCtx);
    if (!(outFormatCtx->oformat->flags & AVFMT_NOFILE)) avio_closep(&outFormatCtx->pb);
    avformat_free_context(outFormatCtx);


  


  Ive tried some debugging to see if the the values of either of the frames were unexpected or to see if the SwrContext might be unsual but did not manage anything. The thing I dont understand is the distinction that would cause this to fail for the right but not for when Im just doing the left.

  


• ffmpeg: UDP to RTMP stream - issues with PES/out of range/corrupted macroblock

  15 August 2019, by MZL

  I´m trying to convert a UDP multicast transportstream video to a RTMP video stream with ffmpeg.

  The stream is generated with a Teracue ENC-300 hardware encoder. The encoder sends the stream as TS/UDP packets to the multicast IP 239.252.20.100:4444.
  ffmpeg can convert the stream, but only at a really low bitrate and/or with a lot of errors.

  My ffmpeg command looks as follows:

  ffmpeg -re -i udp://@239.252.20.100:4444?buffer_size=32000k 
  
  -b:v 900k -maxrate 1000k -bufsize 32000k -f flv 
  
  "rtmp://127.0.0.1/live/live1" -loglevel debug

  I´ve already tried some easyer code:

  ffmpeg -re -i udp://@239.252.20.100:4444 -f flv 
  
  "rtmp://127.0.0.1/live/live1" -loglevel debug

  The Teracue generates the stream with a bitrate of 1500kbps. I also tried some higher or lower bitrates. But the output of ffmpeg has only a maximum bitrate of about 400kbps. If I increase the bitrate with

  -b:v 900k

  or sometimes, when no output bitrate is set, I get a lot of error messages, especially

  PES packet size mismatch
  
  error while decoding MB X X
  
  corrupted macroblock X X

  Here is the error code:

  >[h264 @ 000002883adfe800] nal_unit_type: 6(SEI), nal_ref_idc: 0
  
  
  
  >[h264 @ 000002883adfe800] nal_unit_type: 1(Coded slice of a non-IDR picture), >nal_ref_idc: 3
  
  
  
  >[h264 @ 000002883adfe800] ct_type:1 pic_struct:0
  
  
  
  >[mpegts @ 000002883a329340] Continuity check failed for pid 192 expected 4 >got 7
  
  
  
  >[mpegts @ 000002883a329340] pid=c0 pes_code=0x1c0
  
  
  
  >    Last message repeated 1 times
  
  
  
  >[mpegts @ 000002883a329340] pid=2c pes_code=0x1e0
  
  
  
  >[NULL @ 000002883a33f7c0] ct_type:1 pic_struct:0
  
  
  
  >udp://@239.252.20.100:4444?buffer_size=32000k: corrupt decoded frame in 
  
  
  
  >stream 1
  
  
  
  >[h264 @ 000002883a35a240] nal_unit_type: 9(AUD), nal_ref_idc: 0
  
  
  
  >[h264 @ 000002883a35a240] nal_unit_type: 6(SEI), nal_ref_idc: 0
  
  
  
  >[h264 @ 000002883a35a240] nal_unit_type: 1(Coded slice of a non-IDR picture), 
  
  
  
  >nal_ref_idc: 3
  
  
  
  >[h264 @ 000002883a35a240] ct_type:1 pic_struct:0
  
  
  
  >[h264 @ 000002883a35a240] out of range intra chroma pred mode
  
  
  
  >[h264 @ 000002883a35a240] error while decoding MB 14 8
  
  
  
  >[mpegts @ 000002883a329340] Continuity check failed for pid 44 expected 2 got 
  
  >9
  
  
  
  >[mpegts @ 000002883a329340] PES packet size mismatch
  
  
  
  >[mpegts @ 000002883a329340] pid=2c pes_code=0x1e0
  
  
  
  >[NULL @ 000002883a33f7c0] ct_type:1 pic_struct:0
  
  
  
  >[h264 @ 000002883a35a240] concealing 845 DC, 845 AC, 845 MV errors in P frame
  
  
  
  >[h264 @ 000002883a38f6c0] nal_unit_type: 9(AUD), nal_ref_idc: 0
  
  
  
  >[h264 @ 000002883a38f6c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
  
  
  
  >[h264 @ 000002883a38f6c0] nal_unit_type: 1(Coded slice of a non-IDR picture), >nal_ref_idc: 3
  
  
  
  >[h264 @ 000002883a38f6c0] ct_type:1 pic_struct:0
  
  
  
  >[h264 @ 000002883a38f6c0] Frame num gap 28 26
  
  
  
  >[mpegts @ 000002883a329340] Continuity check failed for pid 192 expected 9 >got 12
  
  
  
  >[mpegts @ 000002883a329340] pid=c0 pes_code=0x1c0
  
  
  
  >[mpegts @ 000002883a329340] Continuity check failed for pid 44 expected 1 got >15
  
  
  
  >[mpegts @ 000002883a329340] PES packet size mismatch
  
  
  
  >[mpegts @ 000002883a329340] pid=2c pes_code=0x1e0
  
  
  
  >[NULL @ 000002883a33f7c0] ct_type:1 pic_struct:0
  
  
  
  >udp://@239.252.20.100:4444?buffer_size=32000k: corrupt decoded frame in 
  
  >stream 1
  
  
  
  >[h264 @ 000002883adf1980] nal_unit_type: 9(AUD), nal_ref_idc: 0
  
  
  
  >[h264 @ 000002883adf1980] nal_unit_type: 6(SEI), nal_ref_idc: 0
  
  
  
  >[h264 @ 000002883adf1980] nal_unit_type: 1(Coded slice of a non-IDR picture), 
  
  
  
  >nal_ref_idc: 3
  
  
  
  >[h264 @ 000002883adf1980] ct_type:1 pic_struct:0
  
  
  
  >[h264 @ 000002883adf1980] mb_type 41 in P slice too large at 18 9
  
  
  
  >[h264 @ 000002883adf1980] error while decoding MB 18 9
  
  
  
  >[h264 @ 000002883adf1980] concealing 796 DC, 796 AC, 796 MV errors in P frame
  
  
  
  >[mpegts @ 000002883a329340] pid=c0 pes_code=0x1c0
  
  
  
  >[NULL @ 000002883a33f7c0] ct_type:1 pic_struct:0
  
  
  
  >[h264 @ 000002883adf20c0] nal_unit_type: 9(AUD), nal_ref_idc: 0
  
  
  
  >[h264 @ 000002883adf20c0] nal_unit_type: 7(SPS), nal_ref_idc: 3
  
  
  
  >[h264 @ 000002883adf20c0] nal_unit_type: 8(PPS), nal_ref_idc: 3
  
  
  
  >[h264 @ 000002883adf20c0] nal_unit_type: 6(SEI), nal_ref_idc: 0
  
  
  
  >[h264 @ 000002883adf20c0] nal_unit_type: 5(IDR), nal_ref_idc: 3
  
  
  
  >[h264 @ 000002883adf20c0] ct_type:1 pic_struct:0
  
  
  
  >[h264 @ 000002883adf20c0] corrupted macroblock 3 0 (total_coeff=-1) 
  
  
  
  >114.0kbits/s speed=   1x
  
  
  
  >[h264 @ 000002883adf20c0] error while decoding MB 3 0
  
  
  
  >[mpegts @ 000002883a329340] pid=c0 pes_code=0x1c0

  Any solutions?