Recherche avancée

Médias (1)

Mot : - Tags -/Rennes

Autres articles (62)

  • Participer à sa traduction

    10 avril 2011

    Vous pouvez nous aider à améliorer les locutions utilisées dans le logiciel ou à traduire celui-ci dans n’importe qu’elle nouvelle langue permettant sa diffusion à de nouvelles communautés linguistiques.
    Pour ce faire, on utilise l’interface de traduction de SPIP où l’ensemble des modules de langue de MediaSPIP sont à disposition. ll vous suffit de vous inscrire sur la liste de discussion des traducteurs pour demander plus d’informations.
    Actuellement MediaSPIP n’est disponible qu’en français et (...)

  • (Dés)Activation de fonctionnalités (plugins)

    18 février 2011, par

    Pour gérer l’ajout et la suppression de fonctionnalités supplémentaires (ou plugins), MediaSPIP utilise à partir de la version 0.2 SVP.
    SVP permet l’activation facile de plugins depuis l’espace de configuration de MediaSPIP.
    Pour y accéder, il suffit de se rendre dans l’espace de configuration puis de se rendre sur la page "Gestion des plugins".
    MediaSPIP est fourni par défaut avec l’ensemble des plugins dits "compatibles", ils ont été testés et intégrés afin de fonctionner parfaitement avec chaque (...)

  • Les tâches Cron régulières de la ferme

    1er décembre 2010, par

    La gestion de la ferme passe par l’exécution à intervalle régulier de plusieurs tâches répétitives dites Cron.
    Le super Cron (gestion_mutu_super_cron)
    Cette tâche, planifiée chaque minute, a pour simple effet d’appeler le Cron de l’ensemble des instances de la mutualisation régulièrement. Couplée avec un Cron système sur le site central de la mutualisation, cela permet de simplement générer des visites régulières sur les différents sites et éviter que les tâches des sites peu visités soient trop (...)

Sur d’autres sites (9774)

  • extracting video and data streams from MPEG2 TS over RTP in real-time

    10 janvier 2024, par Tejal Barnwal

    I have H264 video stream and KLV meta data encapsulated inside MPEG2 TS container which are sent over an RTP over UDP from a camera.
I intend to do the following :

    


      

    1. Extract both video and data streams from RTP

      2. 


    2. Process video feed using opencv in a seperate thread

      3. 


    3. process klv metadata in a seperate thread

      4. 


    


    My problem what exact arguments should I provide to ffmpeg so as to read h264 video stream and show the images frame by frame using opencv ?

    


    With the help of some previous posts like Simultaneously map video and data streams to one subprocess pipeline in real-time, I was able to get some idea about how could I proceed to procees the stream over RTP.

    


    I started out by using the following script :

    


    #!/usr/bin/env python3
from asyncio import streams
from logging.handlers import QueueListener
import klvdata
import subprocess as sp
import shlex
import threading
import numpy as np
import cv2
import time
from io import BytesIO

# Video reader thread.
def video_reader(pipe):
    cols, rows = 1280, 720  # Assume we know frame size is 1280x720

    counter = 0
    while True:
        print("read image")
        raw_image = pipe.read(cols*rows*3)  # Read raw video frame

        # Break the loop when length is too small
        if len(raw_image) < cols*rows*3:
            break

        if (counter % 10) == 0:
            # Show video frame evey 60 frames
            image = np.frombuffer(raw_image, np.uint8).reshape([rows, cols, 3])
            cv2.imshow('Video', image) # Show video image for testing
            cv2.waitKey(1)
        counter += 1
        print("image showed on window")
        time.sleep(0.25)



# https://github.com/paretech/klvdata/tree/master/klvdata
def bytes_to_int(value, signed=False):
    """Return integer given bytes."""
    return int.from_bytes(bytes(value), byteorder='big', signed=signed)


# Data reader thread (read KLV data).
def data_reader(pipe):
    key_length = 16  # Assume key length is 16 bytes.

    f = open('data.bin', 'wb')  # For testing - store the KLV data to data.bin (binary file)

    while True:
        # https://en.wikipedia.org/wiki/KLV
        # The first few bytes are the Key, much like a key in a standard hash table data structure.
        # Keys can be 1, 2, 4, or 16 bytes in length.
        # Presumably in a separate specification document you would agree on a key length for a given application.
        key = pipe.read(key_length)  # Read the key
        
        if len(key) < key_length:
            break  # Break the loop when length is too small
        f.write(key)  # Write data to binary file for testing

        # https://github.com/paretech/klvdata/tree/master/klvdata
        # Length field
        len_byte = pipe.read(1)

        if len(len_byte) < 1:
            break  # Break the loop when length is too small
        f.write(len_byte)  # Write data to binary file for testing

        byte_length = bytes_to_int(len_byte)

        # https://github.com/paretech/klvdata/tree/master/klvdata                                                
        if byte_length < 128:
            # BER Short Form
            length = byte_length
            ber_len_bytes = b''
        else:
            # BER Long Form
            ber_len = byte_length - 128
            ber_len_bytes = pipe.read(ber_len)

            if len(ber_len_bytes) < ber_len:
                break  # Break the loop when length is too small
            f.write(ber_len_bytes)  # Write ber_len_bytes to binary file for testing

            length = bytes_to_int(ber_len_bytes)

        # Read the value (length bytes)
        value = pipe.read(length)
        if len(value) < length:
            break  # Break the loop when length is too small
        f.write(value)  # Write data to binary file for testing

        klv_data = key + len_byte + ber_len_bytes + value  # Concatenate key length and data
        klv_data_as_bytes_io = BytesIO(klv_data)  # Wrap klv_data with BytesIO (before parsing)

        # Parse the KLV data
        for packet in klvdata.StreamParser(klv_data_as_bytes_io): 
            metadata = packet.MetadataList()
            for key, value in metadata.items():
                print(key, value)
                
            print("\n") # New line

# Execute FFmpeg as sub-process
# Map the video to stderr and map the data to stdout
process = sp.Popen(shlex.split('ffmpeg -hide_banner -loglevel quiet '                        # Set loglevel to quiet for disabling the prints ot stderr
                               '-i "rtp://192.168.0.141:11024" '                                        # Input video "Day Flight.mpg"
                               '-map 0:v -c:v rawvideo -pix_fmt bgr24 -f:v rawvideo pipe:2 ' # rawvideo format is mapped to stderr pipe (raw video codec with bgr24 pixel format)
                               '-map 0:d -c copy -copy_unknown -f:d data pipe:1 '            # Copy the data without ddecoding.
                               '-report'),                                                   # Create a log file (because we can't the statuses that are usually printed to stderr).
                                stdout=sp.PIPE, stderr=sp.PIPE)


# Start video reader thread (pass stderr pipe as argument).
video_thread = threading.Thread(target=video_reader, args=(process.stderr,))
video_thread.start()

# Start data reader thread (pass stdout pipe as argument).
data_thread = threading.Thread(target=data_reader, args=(process.stdout,))
data_thread.start()


# Wait for threads (and process) to finish.
video_thread.join()
data_thread.join()
process.wait()



    


    With the above script, I was facing two issues :

    


      

    1. The second thread resulted in an attribute error

      2. 


    


    Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "video_data_extraction.py", line 97, in data_reader
    print(packet.MetadataList())
AttributeError: 'UnknownElement' object has no attribute 'MetadataList'



    


      

    1. With this though I continuously able to see following output on the terminal regarding reading the images

      2. 


    


    read image
image showed on window
read image
image showed on window
read image
image showed on window
read image
image showed on window
read image
image showed on window
read image
image showed on window


    


    The imshow windows wasnt updating properly ! It seemed stuck after a few frames.

    


    Further diving into the lane with the help of following command, I concluded that the video stream that I am reading has H264 encoding

    


    ffprobe -i rtp://192.168.0.141:11024 -show_streams -show_formats


    


    Output of the above command :

    


    ffprobe version 4.2.7-0ubuntu0.1 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[rtp @ 0xaaaac81ecce0] PES packet size mismatch
    Last message repeated 62 times
[NULL @ 0xaaaac81f09b0] non-existing PPS 0 referenced
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[rtp @ 0xaaaac81ecce0] PES packet size mismatch
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[rtp @ 0xaaaac81ecce0] PES packet size mismatch
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[rtp @ 0xaaaac81ecce0] PES packet size mismatch
    Last message repeated 187 times
Input #0, rtp, from 'rtp://192.168.0.141:11024':
  Duration: N/A, start: 1317.040656, bitrate: N/A
  Program 1 
    Stream #0:1: Video: h264 (Constrained Baseline) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1280x720, 25 fps, 25 tbr, 90k tbn
    Stream #0:0: Data: klv (KLVA / 0x41564C4B)
Unsupported codec with id 100356 for input stream 0
[STREAM]
index=0
codec_name=klv
codec_long_name=SMPTE 336M Key-Length-Value (KLV) metadata
profile=unknown
codec_type=data
codec_tag_string=KLVA
codec_tag=0x41564c4b
id=N/A
r_frame_rate=0/0
avg_frame_rate=0/0
time_base=1/90000
start_pts=118533659
start_time=1317.040656
duration_ts=N/A
duration=N/A
bit_rate=N/A
max_bit_rate=N/A
bits_per_raw_sample=N/A
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
[/STREAM]
[STREAM]
index=1
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=Constrained Baseline
codec_type=video
codec_time_base=1/50
codec_tag_string=[27][0][0][0]
codec_tag=0x001b
width=1280
height=720
coded_width=1280
coded_height=720
has_b_frames=0
sample_aspect_ratio=N/A
display_aspect_ratio=N/A
pix_fmt=yuv420p
level=31
color_range=unknown
color_space=unknown
color_transfer=unknown
color_primaries=unknown
chroma_location=left
field_order=progressive
timecode=N/A
refs=1
is_avc=false
nal_length_size=0
id=N/A
r_frame_rate=25/1
avg_frame_rate=25/1
time_base=1/90000
start_pts=118533659
start_time=1317.040656
duration_ts=N/A
duration=N/A
bit_rate=N/A
max_bit_rate=N/A
bits_per_raw_sample=8
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
[/STREAM]
[FORMAT]
filename=rtp://192.168.0.141:11024
nb_streams=2
nb_programs=1
format_name=rtp
format_long_name=RTP input
start_time=1317.040656
duration=N/A
size=N/A
bit_rate=N/A
probe_score=100
[/FORMAT]


    


    Further, in the log output, I see a lot of statements in regard to missed packets and PES packet mismatch

    


    [rtp @ 0xaaaaf31896c0] max delay reached. need to consume packet
[rtp @ 0xaaaaf31896c0] RTP: missed 98 packets
[rtp @ 0xaaaaf31896c0] Continuity check failed for pid 40 expected 14 got 10
[rtp @ 0xaaaaf31896c0] PES packet size mismatch
rtp://192.168.0.141:11024: corrupt input packet in stream 0
frame=  124 fps=2.6 q=-0.0 size=  334800kB time=00:00:05.32 bitrate=515406.0kbits/s dup=97 drop=0 speed=0.111x 


    


    What arguments do I provide to ffmpeg and in what order because my stream 0 is metadata and stream 1 is video so as to display image frame by frame with opencv ?
I would be grateful for any help that you could provide.

    


    Further, I also have a query regarding how does ffmpeg know to that it has to first convert the rtp packets into mpeg2 TS packets before segregating video stream and data stream ?

    


  • Muxing H264 packets into a MPEGTS container using libav*

    30 avril 2024, par Lucen

    I'm writing a C++ program where I need to encode packets in h264 format and mux them to a MPEG TS container. For the encoding part, I based my code on the encode_video example (https://ffmpeg.org/doxygen/trunk/encode_video_8c-example.html#a9) provided in FFMPEG documentation, and it seems to work fine. In particular, I generate a std::vector of packets which I sequentially write to an output .ts file for debug. Such .ts file plays fine with SMPlayer, and a ffproba command gives

    


    >> ffprobe  -print_format json -show_format -show_streams out.ts
Input #0, h264, from 'out.ts':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuv420p(progressive), 640x480 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 1200k tbn, 50 tbc
    "streams": [
        {
            "index": 0,
            "codec_name": "h264",
            "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
            "profile": "Main",
            "codec_type": "video",
            "codec_time_base": "1/50",
            "codec_tag_string": "[0][0][0][0]",
            "codec_tag": "0x0000",
            "width": 640,
            "height": 480,
            "coded_width": 640,
            "coded_height": 480,
            "has_b_frames": 1,
            "sample_aspect_ratio": "1:1",
            "display_aspect_ratio": "4:3",
            "pix_fmt": "yuv420p",
            "level": 30,
            "chroma_location": "left",
            "field_order": "progressive",
            "refs": 1,
            "is_avc": "false",
            "nal_length_size": "0",
            "r_frame_rate": "25/1",
            "avg_frame_rate": "25/1",
            "time_base": "1/1200000",
            "bits_per_raw_sample": "8",
            "disposition": {
                "default": 0,
                "dub": 0,
                "original": 0,
                "comment": 0,
                "lyrics": 0,
                "karaoke": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "visual_impaired": 0,
                "clean_effects": 0,
                "attached_pic": 0,
                "timed_thumbnails": 0
            }
        }
    ],
    "format": {
        "filename": "out.ts",
        "nb_streams": 1,
        "nb_programs": 0,
        "format_name": "h264",
        "format_long_name": "raw H.264 video",
        "size": "435443",
        "probe_score": 51
    }
}



    


    The dts and pts timestamps are also set.
However, if I try to mux them in MPEG TS format, using as a base the example mux.c (https://ffmpeg.org/doxygen/trunk/mux_8c-example.html), it doesn't work. A shortened version of my muxing code is as follows : (the variables ending with "_" are class fields)

    


    int MyProcessing::Mux(const std::string outputFilename) {
    AVFormatContext *muxingContest;
    avformat_alloc_output_context2(&muxingContest, NULL, NULL, m_output.c_str());

    auto outFormat = muxingContest->oformat;
    outFormat->video_codec = AV_CODEC_ID_H264;

    AVStream *outStream;
    const AVCodec *codec;

    Mux_AddStream(&outStream, muxingContest, &codec, outFormat->video_codec);

    AVDictionary *opt = nullptr;
    Mux_OpenVideo(muxingContest, codec, outStream, opt);
 
    if (!(muxingContest->flags & AVFMT_NOFILE)) {
      avio_open(&muxingContest->pb, m_output.c_str(), AVIO_FLAG_WRITE);
    }
    avformat_write_header(muxingContest, &opt);

    auto muxOk = true;
    size_t countMuxedFrames = 0;
    while ((muxOk) && (countMuxedFrames < packets_.size())) {
        muxOk = !MuxPacket(muxingContest, outStream, packets_[countMuxedFrames], &opt);
        countMuxedFrames++;
    }

    av_write_trailer(muxingContest);
    if (!(muxCodecContextPtr_->flags & AVFMT_NOFILE)) avio_closep(&muxingContest->pb);
 
    return 0;
}


int MyProcessing::Mux_AddStream(AVStream **stream, AVFormatContext *format, const AVCodec **codec, enum AVCodecID codecId) {
    *codec = avcodec_find_encoder(codecId);
    muxPacketTmpPtr_ = av_packet_alloc();
    *stream = avformat_new_stream(format, *codec);
    (*stream)->time_base = (AVRational){ 1, STREAM_FRAME_RATE };
    (*stream)->id = format->nb_streams-1;
    (*stream)->index = 0;
    muxCodecContextPtr_ = avcodec_alloc_context3(*codec);
    Mux_FillCodecContext(*muxCodecContextPtr_, codecId, **stream);
    if (format->oformat->flags & AVFMT_GLOBALHEADER)
        muxCodecContextPtr_->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
    return 0;
}


void MyProcessing::Mux_FillCodecContext(AVCodecContext &cc, enum AVCodecID codecId, AVStream &stream) {
    cc.codec_id = codecId;
    cc.bit_rate = 400000;
    cc.width    = outputWidth_;
    cc.height   = outputHeight_;
    cc.time_base  = stream.time_base;
    cc.gop_size = 10;
    cc.max_b_frames = 1;
    cc.gop_size      = 12;
    cc.pix_fmt       = AV_PIX_FMT_YUV420P;
    if (cc.codec_id == AV_CODEC_ID_MPEG2VIDEO) cc.max_b_frames = 2;
    if (cc.codec_id == AV_CODEC_ID_MPEG1VIDEO)  cc.mb_decision = 2;
    av_opt_set(&cc, "preset", "slow", 0);
    av_opt_set(&cc, "tune", "zerolatency", 0);
}


int MyProcessing::Mux_OpenVideo(AVFormatContext *format, const AVCodec *codec, AVStream *stream, AVDictionary *opt_arg) {
    AVDictionary *opt = nullptr;
    av_dict_copy(&opt, opt_arg, 0);
    muxCodecContextPtr_->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
    avcodec_open2(muxCodecContextPtr_, codec, &opt);
    av_dict_free(&opt);
    avcodec_parameters_from_context(stream->codecpar, muxCodecContextPtr_);
    return 0;
}

int MyProcessing::MuxPacket(AVFormatContext *format, AVStream *stream, AVPacket &pkt, AVDictionary **opt) {
    AVBitStreamFilterContext *bsf = av_bitstream_filter_init("h264_mp4toannexb");
    AVPacket filteredPkt = pkt;
    auto filterResult = av_bitstream_filter_filter(bsf, format->streams[stream->index]->codec, NULL,
                                         &filteredPkt.data, &filteredPkt.size,
                                         pkt.data, pkt.size,
                                         pkt.flags & AV_PKT_FLAG_KEY);

    if (filterResult < 0) return filterResult;
    else {
        filteredPkt.buf = av_buffer_create(filteredPkt.data, filteredPkt.size,
                                       av_buffer_default_free, NULL, 0);
    }
    av_bitstream_filter_close(bsf);
    filteredPkt.stream_index = stream->index;
    filteredPkt.dts = filteredPkt.pts;
    filteredPkt.duration = ((double)stream->time_base.num / (double)stream->time_base.den) / STREAM_FRAME_RATE;
    av_packet_rescale_ts(&filteredPkt, muxCodecContextPtr_->time_base, stream->time_base); // rescale output packet timestamp values from codec to stream timebase
    auto writePktResult = av_write_frame(format, &filteredPkt);
   // auto writePktResult = av_interleaved_write_frame(format, &filteredPkt);
    return 0;
}



    


    The console error is

    


    [mpegts @ 0x55555736edc0] H.264 bitstream malformed, no startcode found, use the video bitstream filter 'h264_mp4toannexb' to fix it ('-bsf:v h264_mp4toannexb' option with ffmpeg)


    


    It Is telling me to apply the h264_mp4toannexb filter. As you see from the code, I've put the filtering accordingly, but the error message persists (unless I'm applying the filter in a wrong way).

    


    In the last lines of method MuxPacket(), if I uncomment the line with av_interleaved_write_frame() and comment the previous one, I get the same error, as well as a seg fault. Inspecting with GDB, the call stack for the seg fault is as follows :

    


    #0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:440
#1  0x00007ffff67c7cb6 in av_packet_copy_props () at /lib/x86_64-linux-gnu/libavcodec.so.58
#2  0x00007ffff67c8447 in av_packet_ref () at /lib/x86_64-linux-gnu/libavcodec.so.58
#3  0x00007ffff7e2fa13 in  () at /lib/x86_64-linux-gnu/libavformat.so.58
#4  0x00007ffff7e2fb11 in  () at /lib/x86_64-linux-gnu/libavformat.so.58
#5  0x00007ffff7e30575 in av_interleaved_write_frame () at /lib/x86_64-linux-gnu/libavformat.so.58


    


    I tried to look at solutions online, but they are either old or they don't work. Some of the things I tried and didn't work :

    


      

    1. Putting the line

      2. 


    


    muxCodecContextPtr_->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;


    


    in Mux() after the call to avformat_alloc_output_context2.

    


      

    1. Setting

      2. 


    


    packet.flags |= AV_PKT_FLAG_KEY;


    


    before the call to av_write_frame / av_interleaved_write_frame.

    


      


    1. Trying to write by hand to the file the starting code as described here Need to convert h264 stream from annex-b format to AVCC format.

      


      2. 


    2. Playing with parameters in Mux_FillCodecContext().

      


      3. 


    


  • Timeline Display Issue When Merging Multi-Track MP4 Files with FFmpeg

    9 juillet 2024, par EasonWaii

    I need to merge two videos into one using ffmpeg. The resulting video should have two video tracks and one audio track, with the audio track taken from the longer video.

    


    When playing the merged video in a player like PotPlayer or VLC, it should default to playing the shorter video track. However, users should be able to switch to the other video track if they want.

    


    The problem I am facing :
Everything is working fine, except when the player defaults to the shorter video track, it shows the timeline of the longer video track.

    


     built with Apple clang version 15.0.0 (clang-1500.3.9.4)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/7.0.1 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox
  libavutil      59.  8.100 / 59.  8.100
  libavcodec     61.  3.100 / 61.  3.100
  libavformat    61.  1.100 / 61.  1.100
  libavdevice    61.  1.100 / 61.  1.100
  libavfilter    10.  1.100 / 10.  1.100
  libswscale      8.  1.100 /  8.  1.100
  libswresample   5.  1.100 /  5.  1.100
  libpostproc    58.  1.100 / 58.  1.100
{
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'output21.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf59.16.100
  Duration: 00:00:12.84, start: 0.000000, bitrate: 2207 kb/s
  Stream #0:0[0x1](und): Video: hevc (Main) (hev1 / 0x31766568), yuv420p(tv, progressive), 720x1280 [SAR 1:1 DAR 9:16], 1052 kb/s, 30 fps, 30 tbr, 15360 tbn
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 720x1280, 1293 kb/s, 29.83 fps, 29.83 tbr, 11456 tbn (default)
      Metadata:
        handler_name    : ?Mainconcept Video Media Handler
        vendor_id       : [0][0][0][0]
  Stream #0:2[0x3](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 130 kb/s (default)
      Metadata:
        handler_name    : SoundHandler
        vendor_id       : [0][0][0][0]
    "streams": [
        {
            "index": 0,
            "codec_name": "hevc",
            "codec_long_name": "H.265 / HEVC (High Efficiency Video Coding)",
            "profile": "Main",
            "codec_type": "video",
            "codec_tag_string": "hev1",
            "codec_tag": "0x31766568",
            "width": 720,
            "height": 1280,
            "coded_width": 720,
            "coded_height": 1280,
            "closed_captions": 0,
            "film_grain": 0,
            "has_b_frames": 2,
            "sample_aspect_ratio": "1:1",
            "display_aspect_ratio": "9:16",
            "pix_fmt": "yuv420p",
            "level": 93,
            "color_range": "tv",
            "chroma_location": "left",
            "field_order": "progressive",
            "refs": 1,
            "id": "0x1",
            "r_frame_rate": "30/1",
            "avg_frame_rate": "30/1",
            "time_base": "1/15360",
            "start_pts": 0,
            "start_time": "0.000000",
            "duration_ts": 196096,
            "duration": "12.766667",
            "bit_rate": "1052926",
            "nb_frames": "383",
            "extradata_size": 2480,
            "disposition": {
                "default": 0,
                "dub": 0,
                "original": 0,
                "comment": 0,
                "lyrics": 0,
                "karaoke": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "visual_impaired": 0,
                "clean_effects": 0,
                "attached_pic": 0,
                "timed_thumbnails": 0,
                "non_diegetic": 0,
                "captions": 0,
                "descriptions": 0,
                "metadata": 0,
                "dependent": 0,
                "still_image": 0
            },
            "tags": {
                "language": "und",
                "handler_name": "VideoHandler",
                "vendor_id": "[0][0][0][0]"
            }
        },
        {
            "index": 1,
            "codec_name": "h264",
            "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
            "profile": "High",
            "codec_type": "video",
            "codec_tag_string": "avc1",
            "codec_tag": "0x31637661",
            "width": 720,
            "height": 1280,
            "coded_width": 720,
            "coded_height": 1280,
            "closed_captions": 0,
            "film_grain": 0,
            "has_b_frames": 2,
            "pix_fmt": "yuv420p",
            "level": 31,
            "color_range": "tv",
            "color_space": "bt709",
            "color_transfer": "bt709",
            "color_primaries": "bt709",
            "chroma_location": "left",
            "field_order": "progressive",
            "refs": 1,
            "is_avc": "true",
            "nal_length_size": "4",
            "id": "0x2",
            "r_frame_rate": "179/6",
            "avg_frame_rate": "3448256/115597",
            "time_base": "1/11456",
            "start_pts": 0,
            "start_time": "0.000000",
            "duration_ts": 115597,
            "duration": "10.090520",
            "bit_rate": "1293284",
            "bits_per_raw_sample": "8",
            "nb_frames": "301",
            "extradata_size": 46,
            "disposition": {
                "default": 1,
                "dub": 0,
                "original": 0,
                "comment": 0,
                "lyrics": 0,
                "karaoke": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "visual_impaired": 0,
                "clean_effects": 0,
                "attached_pic": 0,
                "timed_thumbnails": 0,
                "non_diegetic": 0,
                "captions": 0,
                "descriptions": 0,
                "metadata": 0,
                "dependent": 0,
                "still_image": 0
            },
            "tags": {
                "language": "eng",
                "handler_name": "\u001fMainconcept Video Media Handler",
                "vendor_id": "[0][0][0][0]"
            }
        },
        {
            "index": 2,
            "codec_name": "aac",
            "codec_long_name": "AAC (Advanced Audio Coding)",
            "profile": "LC",
            "codec_type": "audio",
            "codec_tag_string": "mp4a",
            "codec_tag": "0x6134706d",
            "sample_fmt": "fltp",
            "sample_rate": "44100",
            "channels": 2,
            "channel_layout": "stereo",
            "bits_per_sample": 0,
            "initial_padding": 0,
            "id": "0x3",
            "r_frame_rate": "0/0",
            "avg_frame_rate": "0/0",
            "time_base": "1/44100",
            "start_pts": 0,
            "start_time": "0.000000",
            "duration_ts": 566244,
            "duration": "12.840000",
            "bit_rate": "130447",
            "nb_frames": "554",
            "extradata_size": 5,
            "disposition": {
                "default": 1,
                "dub": 0,
                "original": 0,
                "comment": 0,
                "lyrics": 0,
                "karaoke": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "visual_impaired": 0,
                "clean_effects": 0,
                "attached_pic": 0,
                "timed_thumbnails": 0,
                "non_diegetic": 0,
                "captions": 0,
                "descriptions": 0,
                "metadata": 0,
                "dependent": 0,
                "still_image": 0
            },
            "tags": {
                "language": "und",
                "handler_name": "SoundHandler",
                "vendor_id": "[0][0][0][0]"
            }
        }
    ],
    "format": {
        "filename": "output21.mp4",
        "nb_streams": 3,
        "nb_programs": 0,
        "nb_stream_groups": 0,
        "format_name": "mov,mp4,m4a,3gp,3g2,mj2",
        "format_long_name": "QuickTime / MOV",
        "start_time": "0.000000",
        "duration": "12.840000",
        "size": "3543155",
        "bit_rate": "2207573",
        "probe_score": 100,
        "tags": {
            "major_brand": "isom",
            "minor_version": "512",
            "compatible_brands": "isomiso2avc1mp41",
            "encoder": "Lavf59.16.100"
        }
    }
}



    


    I want the timeline to display correctly according to the shorter video track when it is being played, without truncating the timeline of the longer video track and affecting its playback.

    


Navigation