Advanced search

On other websites (9858)

• Algorithm when recording SegmentTimeline of MPEG-DASH MPD in shaka packager

  25 March 2021, by jgkim0518

  I packaged a media stream by mpeg-dash transcoded video at twice the speed and audio at normal speed.
Here is the command:

  


  ffmpeg -re -stream_loop -1 -i timing_logic.mp4 -c:v hevc_nvenc -filter:v "setpts=2*PTS" -c:a libfdk_aac -map 0:v -map 0:a -f mpegts udp://xxx.xxx.xxx.xxx:xxxx?pkt_size=1316


  


  The source information used here is as follows:

  


  Input #0, mpegts, from '/home/test/timing_logic/timing_logic.mp4':
Duration: 00:00:30.22, start: 1.400000, bitrate: 16171 kb/s
Program 1
Metadata:
service_name : Service01
service_provider: FFmpeg
Stream #0:0[0x100]: Video: hevc (Main 10) (HEVC / 0x43564548), yuv420p10le(tv, bt709), 3840x2160 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 50 tbc
Stream #0:1[0x101](eng): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 128 kb/s Stream mapping:
Stream #0:0 -> #0:0 (hevc (native) -> hevc (hevc_nvenc))
Stream #0:1 -> #0:1 (mp2 (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
frame= 0 fps=0.0 q=0.0 size= 0kB time=-577014:32:22.77 bitrate= -0.0kbits/s speed=N/A
frame= 0 fps=0.0 q=0.0 size= 0kB time=-577014:32:22.77 bitrate= -0.0kbits/s speed=N/A
Output #0, mpegts, to 'udp://xxx.xxx.xxx.xxx:xxxx?pkt_size=1316':
Metadata:
encoder : Lavf58.45.100
Stream #0:0: Video: hevc (hevc_nvenc) (Main 10), p010le, 3840x2160 [SAR 1:1 DAR 16:9], q=-1--1, 2000 kb/s, 50 fps, 90k tbn, 50 tbc
Metadata:
encoder : Lavc58.91.100 hevc_nvenc
Side data:
cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: N/A
Stream #0:1(eng): Audio: aac (libfdk_aac), 48000 Hz, stereo, s16, 139 kb/s
Metadata:
encoder : Lavc58.91.100 libfdk_aac


  


  My shaka packager command is:

  


  packager \ 'in=udp://xxx.xxx.xxx.xxx:xxxx,stream=video,init_segment=/home/test/timing_logic/package/video/0/video.mp4,segment_template=/home/test/timing_logic/package/video/0/$Time$.m4s' \
'in=udp://xxx.xxx.xxx.xxx:xxxx,stream=audio,init_segment=/home/test/timing_logic/package/audio/0/audio.mp4,segment_template=/home/test/timing_logic/package/audio/0/$Time$.m4a' \
--segment_duration 5 --fragment_duration 5 --minimum_update_period 5 --min_buffer_time 5 \
--preserved_segments_outside_live_window 24 --time_shift_buffer_depth 40 \
--allow_codec_switching --allow_approximate_segment_timeline --log_file_generation_deletion \
--mpd_output $OUTPUT/$output_mpd.mpd &


  


  Looking at the mpd generated as a result of packaging, the duration of the audio is twice that of the video, and the number of segments is half.
The following is the content of mpd:

  


  &lt;?xml version="1.0" encoding="UTF-8"?>&#xA;&#xA;<mpd xmlns="urn:mpeg:dash:schema:mpd:2011" profiles="urn:mpeg:dash:profile:isoff-live:2011" minbuffertime="PT5S" type="dynamic" publishtime="2021-03-23T10:01:57Z" availabilitystarttime="2021-03-23T09:59:55Z" minimumupdateperiod="PT5S" timeshiftbufferdepth="PT40S">&#xA;<period start="PT0S">&#xA;<adaptationset contenttype="audio" segmentalignment="true">&#xA;<representation bandwidth="140020" codecs="mp4a.40.2" mimetype="audio/mp4" audiosamplingrate="48000">&#xA;<audiochannelconfiguration schemeiduri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"></audiochannelconfiguration>&#xA;<segmenttemplate timescale="90000" initialization="/home/test/timing_logic/package/audio/0/audio.mp4" media="/home/test/timing_logic/package/audio/0/$Time$.m4a" startnumber="16">&#xA;<segmenttimeline>&#xA;<s t="6751436" d="449280"></s>&#xA;<s t="7200716" d="451200"></s>&#xA;<s t="7651916" d="449280"></s>&#xA;<s t="8101196" d="374400"></s>&#xA;<s t="8551196" d="449280"></s>&#xA;<s t="9000476" d="451200"></s>&#xA;<s t="9451674" d="449280"></s>&#xA;<s t="9900956" d="449280"></s>&#xA;<s t="10350236" d="451200"></s>&#xA;<s t="10801434" d="374400"></s>&#xA;</segmenttimeline>&#xA;</segmenttemplate>&#xA;</representation>&#xA;</adaptationset>&#xA;<adaptationset contenttype="video" width="3840" height="2160" framerate="90000/3600" segmentalignment="true" par="16:9">&#xA;<representation bandwidth="1520392" codecs="hvc1.2.4.L153" mimetype="video/mp4" sar="1:1">&#xA;<segmenttemplate timescale="90000" initialization="/home/test/timing_logic/package/video/0/video.mp4" media="/home/test/timing_logic/package/video/0/$Time$.m4s" startnumber="19">&#xA;<segmenttimeline>&#xA;<s t="16954440" d="900000" r="4"></s>&#xA;</segmenttimeline>&#xA;</segmenttemplate>&#xA;</representation>&#xA;</adaptationset>&#xA;</period>&#xA;</mpd>&#xA;

  


  When looking at the MPD, it seems that the shaka packager checks the PTS or DTS of the TS when creating a segment and recording the contents of the SegmentTimeline.
But I couldn't understand even by looking at the MPEG standard documentation and the DASH-IF documentation.

  


  My question is whether the packager refers to PTS or DTS when creating a segment.
How are SegmentTimeline's S@t and S@d recorded?
What algorithm is the SegmentTimeline recorded with? Please help me. Thank you.

  


• Algorithm when recording SegmentTimeline

  24 March 2021, by jgkim0518

  I packaged a media stream by mpeg-dash transcoded video at twice the speed and audio at normal speed.
Here is the command:

  


  ffmpeg -re -stream_loop -1 -i timing_logic.mp4 -c:v hevc_nvenc -filter:v "setpts=2*PTS" -c:a libfdk_aac -map 0:v -map 0:a -f mpegts udp://xxx.xxx.xxx.xxx:xxxx?pkt_size=1316


  


  The source information used here is as follows:

  


  Input #0, mpegts, from '/home/test/timing_logic/timing_logic.mp4':
Duration: 00:00:30.22, start: 1.400000, bitrate: 16171 kb/s
Program 1
Metadata:
service_name : Service01
service_provider: FFmpeg
Stream #0:0[0x100]: Video: hevc (Main 10) (HEVC / 0x43564548), yuv420p10le(tv, bt709), 3840x2160 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 50 tbc
Stream #0:1[0x101](eng): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 128 kb/s Stream mapping:
Stream #0:0 -> #0:0 (hevc (native) -> hevc (hevc_nvenc))
Stream #0:1 -> #0:1 (mp2 (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
frame= 0 fps=0.0 q=0.0 size= 0kB time=-577014:32:22.77 bitrate= -0.0kbits/s speed=N/A
frame= 0 fps=0.0 q=0.0 size= 0kB time=-577014:32:22.77 bitrate= -0.0kbits/s speed=N/A
Output #0, mpegts, to 'udp://xxx.xxx.xxx.xxx:xxxx?pkt_size=1316':
Metadata:
encoder : Lavf58.45.100
Stream #0:0: Video: hevc (hevc_nvenc) (Main 10), p010le, 3840x2160 [SAR 1:1 DAR 16:9], q=-1--1, 2000 kb/s, 50 fps, 90k tbn, 50 tbc
Metadata:
encoder : Lavc58.91.100 hevc_nvenc
Side data:
cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: N/A
Stream #0:1(eng): Audio: aac (libfdk_aac), 48000 Hz, stereo, s16, 139 kb/s
Metadata:
encoder : Lavc58.91.100 libfdk_aac


  


  My shaka packager command is:

  


  packager \ 'in=udp://xxx.xxx.xxx.xxx:xxxx,stream=video,init_segment=/home/test/timing_logic/package/video/0/video.mp4,segment_template=/home/test/timing_logic/package/video/0/$Time$.m4s' \
'in=udp://xxx.xxx.xxx.xxx:xxxx,stream=audio,init_segment=/home/test/timing_logic/package/audio/0/audio.mp4,segment_template=/home/test/timing_logic/package/audio/0/$Time$.m4a' \
--segment_duration 5 --fragment_duration 5 --minimum_update_period 5 --min_buffer_time 5 \
--preserved_segments_outside_live_window 24 --time_shift_buffer_depth 40 \
--allow_codec_switching --allow_approximate_segment_timeline --log_file_generation_deletion \
--mpd_output $OUTPUT/$output_mpd.mpd &


  


  Looking at the mpd generated as a result of packaging, the duration of the audio is twice that of the video, and the number of segments is half.
The following is the content of mpd:

  


  &lt;?xml version="1.0" encoding="UTF-8"?>&#xA;&#xA;<mpd xmlns="urn:mpeg:dash:schema:mpd:2011" profiles="urn:mpeg:dash:profile:isoff-live:2011" minbuffertime="PT5S" type="dynamic" publishtime="2021-03-23T10:01:57Z" availabilitystarttime="2021-03-23T09:59:55Z" minimumupdateperiod="PT5S" timeshiftbufferdepth="PT40S">&#xA;<period start="PT0S">&#xA;<adaptationset contenttype="audio" segmentalignment="true">&#xA;<representation bandwidth="140020" codecs="mp4a.40.2" mimetype="audio/mp4" audiosamplingrate="48000">&#xA;<audiochannelconfiguration schemeiduri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"></audiochannelconfiguration>&#xA;<segmenttemplate timescale="90000" initialization="/home/test/timing_logic/package/audio/0/audio.mp4" media="/home/test/timing_logic/package/audio/0/$Time$.m4a" startnumber="16">&#xA;<segmenttimeline>&#xA;<s t="6751436" d="449280"></s>&#xA;<s t="7200716" d="451200"></s>&#xA;<s t="7651916" d="449280"></s>&#xA;<s t="8101196" d="374400"></s>&#xA;<s t="8551196" d="449280"></s>&#xA;<s t="9000476" d="451200"></s>&#xA;<s t="9451674" d="449280"></s>&#xA;<s t="9900956" d="449280"></s>&#xA;<s t="10350236" d="451200"></s>&#xA;<s t="10801434" d="374400"></s>&#xA;</segmenttimeline>&#xA;</segmenttemplate>&#xA;</representation>&#xA;</adaptationset>&#xA;<adaptationset contenttype="video" width="3840" height="2160" framerate="90000/3600" segmentalignment="true" par="16:9">&#xA;<representation bandwidth="1520392" codecs="hvc1.2.4.L153" mimetype="video/mp4" sar="1:1">&#xA;<segmenttemplate timescale="90000" initialization="/home/test/timing_logic/package/video/0/video.mp4" media="/home/test/timing_logic/package/video/0/$Time$.m4s" startnumber="19">&#xA;<segmenttimeline>&#xA;<s t="16954440" d="900000" r="4"></s>&#xA;</segmenttimeline>&#xA;</segmenttemplate>&#xA;</representation>&#xA;</adaptationset>&#xA;</period>&#xA;</mpd>&#xA;

  


  When looking at the MPD, it seems that the shaka packager checks the PTS or DTS of the TS when creating a segment and recording the contents of the SegmentTimeline.
But I couldn't understand even by looking at the MPEG standard documentation and the DASH-IF documentation.

  


  My question is whether the packager refers to PTS or DTS when creating a segment.
How are SegmentTimeline's S@t and S@d recorded?
What algorithm is the SegmentTimeline recorded with? Please help me. Thank you.

  


• h264 encoding of bgr images into .mp4 file with libav

  10 February 2021, by xyfix

  I'm trying to encode(h264) a series of .png into a mp4 file. A cv::Mat holds the png data (BGR) and that is converted to YUV420P which is then encoded and written to a .mp4 file. I have added two block statements in the code to store images on the disk (before and after encoding). The first image before it gets encoded is correct but the second one after encoding is not. The avcodec_send_frame returns 0 so up to that point everything works.
Edit : I get an mp4 file of 1 MB but I can't open it with vlc
ecodec.h

  


  class ECodec
{
public:

    MovieCodec();

    ~MovieCodec();

    void MatToFrame( cv::Mat& image );

    void encode( AVFrame *frame, AVPacket *pkt );

private:
 
    FILE* m_file;

    AVCodec* m_encoder = NULL;

    AVCodecContext* m_codecContextOut = NULL;

    AVPacket* m_packet = NULL;

};


  


  ecodec.cpp

  


  ECodec::ECodec() :
//    m_encoder( avcodec_find_encoder_by_name( videoCodec.c_str()))
    m_encoder( avcodec_find_encoder( AV_CODEC_ID_H264 ))
{
    m_file = fopen( "c:\\tmp\\outputVideo.mp4", "wb");
}



void ECodec::MatToFrame( cv::Mat& image )
{
    int ret( 0 );
    int frameRate( 24 );
    AVFrame *frame = NULL;

    m_encoder( avcodec_find_encoder( AV_CODEC_ID_H264 ))
    m_codecContextOut = avcodec_alloc_context3( m_encoder );

    m_codecContextOut->width = 800;
    m_codecContextOut->height = 640;
    m_codecContextOut->bit_rate = 400000;//m_codecContextOut->width * m_codecContextOut->height * 3;
    m_codecContextOut->time_base = (AVRational){1, 24};
    m_codecContextOut->framerate = (AVRational){24, 1};
    m_codecContextOut->codec_tag = AV_CODEC_ID_H264;
    m_codecContextOut->pix_fmt = AV_PIX_FMT_YUV420P;
    m_codecContextOut->codec_type = AVMEDIA_TYPE_VIDEO;
    m_codecContextOut->gop_size = 1;
    m_codecContextOut->max_b_frames = 1;

    av_log_set_level(AV_LOG_VERBOSE);

    ret = av_opt_set(m_codecContextOut->priv_data, "preset", "slow", 0);

    ret = avcodec_open2(m_codecContextOut, m_encoder, NULL);

    frame = av_frame_alloc();

    frame->format = AV_PIX_FMT_YUV420P;
    frame->width = image.cols();
    frame->height = image.rows();


    ret = av_image_alloc(frame->data, frame->linesize, frame->width,  frame->height, AV_PIX_FMT_YUV420P, 1);

    if (ret < 0)
    {
        return;
    }

    struct SwsContext *sws_ctx;
    sws_ctx = sws_getContext((int)image.cols(), (int)image.rows(), AV_PIX_FMT_RGB24,
                             (int)image.cols(), (int)image.rows(), AV_PIX_FMT_YUV420P,
                             0, NULL, NULL, NULL);

    const uint8_t* rgbData[1] = { (uint8_t* )image.getData() };
    int rgbLineSize[1] = { 3 * image.cols() };

    sws_scale(sws_ctx, rgbData, rgbLineSize, 0, image.rows(), frame->data, frame->linesize);

    frame->pict_type = AV_PICTURE_TYPE_I;

cv::Mat yuv420p(frame->height + frame->height/2, frame->width, CV_8UC1,frame->data[0]);
cv::Mat cvmIm;
cv::cvtColor(yuv420p,cvmIm,CV_YUV420p2BGR);
cv::imwrite("c:\\tmp\\rawimage.png", cvmIm);
//OK

    m_packet = av_packet_alloc();
    ret = av_new_packet( m_packet, m_codecContextOut->width * m_codecContextOut->height * 3 );

    /* encode the image */
    encode( frame, m_packet );


    avcodec_free_context(&m_codecContextOut);
    av_frame_free(&frame);
    av_packet_free( &m_packet );
}




void ECodec::encode( AVFrame *frame, AVPacket *pkt )
{
    int ret;
    
    /* send the frame to the encoder */
    ret = avcodec_send_frame( m_codecContextOut, frame);

    if (ret < 0)
    {
        fprintf(stderr, "Error sending a frame for encoding\n");
        exit(1);
    }

    do
    {
        ret = avcodec_receive_packet(m_codecContextOut, pkt);
        if (ret == 0)
        {

cv::Mat yuv420p(frame->height + frame->height/2, frame->width, CV_8UC1,pkt->data);
cv::Mat cvmIm;
cv::cvtColor(yuv420p,cvmIm,CV_YUV420p2BGR);
cv::imwrite("c:\\tmp\\rawencodedimage.png", cvmIm);
//NOT OK
            fwrite(pkt->data, 1, pkt->size, m_file );
            av_packet_unref(pkt);

            break;
        }
        else if ((ret < 0) && (ret != AVERROR(EAGAIN)))
        {
            return;
        }
        else if (ret == AVERROR(EAGAIN))
        {
             ret = avcodec_send_frame(m_codecContextOut, NULL);
             if (0 > ret)
             {
                 return;
             }
        }
    } while (ret == 0);
}