Recherche avancée

Sur d’autres sites (7613)

• extracting video and data streams from MPEG2 TS over RTP in real-time

  10 janvier 2024, par Tejal Barnwal

  I have H264 video stream and KLV meta data encapsulated inside MPEG2 TS container which are sent over an RTP over UDP from a camera.
I intend to do the following :

  


  

  1. Extract both video and data streams from RTP

  


  2. Process video feed using opencv in a seperate thread

  


  3. process klv metadata in a seperate thread

  


  


  My problem what exact arguments should I provide to ffmpeg so as to read h264 video stream and show the images frame by frame using opencv ?

  


  With the help of some previous posts like Simultaneously map video and data streams to one subprocess pipeline in real-time, I was able to get some idea about how could I proceed to procees the stream over RTP.

  


  I started out by using the following script :

  


  #!/usr/bin/env python3
from asyncio import streams
from logging.handlers import QueueListener
import klvdata
import subprocess as sp
import shlex
import threading
import numpy as np
import cv2
import time
from io import BytesIO

# Video reader thread.
def video_reader(pipe):
    cols, rows = 1280, 720  # Assume we know frame size is 1280x720

    counter = 0
    while True:
        print("read image")
        raw_image = pipe.read(cols*rows*3)  # Read raw video frame

        # Break the loop when length is too small
        if len(raw_image) < cols*rows*3:
            break

        if (counter % 10) == 0:
            # Show video frame evey 60 frames
            image = np.frombuffer(raw_image, np.uint8).reshape([rows, cols, 3])
            cv2.imshow('Video', image) # Show video image for testing
            cv2.waitKey(1)
        counter += 1
        print("image showed on window")
        time.sleep(0.25)



# https://github.com/paretech/klvdata/tree/master/klvdata
def bytes_to_int(value, signed=False):
    """Return integer given bytes."""
    return int.from_bytes(bytes(value), byteorder='big', signed=signed)


# Data reader thread (read KLV data).
def data_reader(pipe):
    key_length = 16  # Assume key length is 16 bytes.

    f = open('data.bin', 'wb')  # For testing - store the KLV data to data.bin (binary file)

    while True:
        # https://en.wikipedia.org/wiki/KLV
        # The first few bytes are the Key, much like a key in a standard hash table data structure.
        # Keys can be 1, 2, 4, or 16 bytes in length.
        # Presumably in a separate specification document you would agree on a key length for a given application.
        key = pipe.read(key_length)  # Read the key
        
        if len(key) < key_length:
            break  # Break the loop when length is too small
        f.write(key)  # Write data to binary file for testing

        # https://github.com/paretech/klvdata/tree/master/klvdata
        # Length field
        len_byte = pipe.read(1)

        if len(len_byte) < 1:
            break  # Break the loop when length is too small
        f.write(len_byte)  # Write data to binary file for testing

        byte_length = bytes_to_int(len_byte)

        # https://github.com/paretech/klvdata/tree/master/klvdata                                                
        if byte_length < 128:
            # BER Short Form
            length = byte_length
            ber_len_bytes = b''
        else:
            # BER Long Form
            ber_len = byte_length - 128
            ber_len_bytes = pipe.read(ber_len)

            if len(ber_len_bytes) < ber_len:
                break  # Break the loop when length is too small
            f.write(ber_len_bytes)  # Write ber_len_bytes to binary file for testing

            length = bytes_to_int(ber_len_bytes)

        # Read the value (length bytes)
        value = pipe.read(length)
        if len(value) < length:
            break  # Break the loop when length is too small
        f.write(value)  # Write data to binary file for testing

        klv_data = key + len_byte + ber_len_bytes + value  # Concatenate key length and data
        klv_data_as_bytes_io = BytesIO(klv_data)  # Wrap klv_data with BytesIO (before parsing)

        # Parse the KLV data
        for packet in klvdata.StreamParser(klv_data_as_bytes_io): 
            metadata = packet.MetadataList()
            for key, value in metadata.items():
                print(key, value)
                
            print("\n") # New line

# Execute FFmpeg as sub-process
# Map the video to stderr and map the data to stdout
process = sp.Popen(shlex.split('ffmpeg -hide_banner -loglevel quiet '                        # Set loglevel to quiet for disabling the prints ot stderr
                               '-i "rtp://192.168.0.141:11024" '                                        # Input video "Day Flight.mpg"
                               '-map 0:v -c:v rawvideo -pix_fmt bgr24 -f:v rawvideo pipe:2 ' # rawvideo format is mapped to stderr pipe (raw video codec with bgr24 pixel format)
                               '-map 0:d -c copy -copy_unknown -f:d data pipe:1 '            # Copy the data without ddecoding.
                               '-report'),                                                   # Create a log file (because we can't the statuses that are usually printed to stderr).
                                stdout=sp.PIPE, stderr=sp.PIPE)


# Start video reader thread (pass stderr pipe as argument).
video_thread = threading.Thread(target=video_reader, args=(process.stderr,))
video_thread.start()

# Start data reader thread (pass stdout pipe as argument).
data_thread = threading.Thread(target=data_reader, args=(process.stdout,))
data_thread.start()


# Wait for threads (and process) to finish.
video_thread.join()
data_thread.join()
process.wait()



  


  With the above script, I was facing two issues :

  


  

  1. The second thread resulted in an attribute error

  


  


  Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "video_data_extraction.py", line 97, in data_reader
    print(packet.MetadataList())
AttributeError: 'UnknownElement' object has no attribute 'MetadataList'



  


  

  2. With this though I continuously able to see following output on the terminal regarding reading the images

  


  


  read image
image showed on window
read image
image showed on window
read image
image showed on window
read image
image showed on window
read image
image showed on window
read image
image showed on window


  


  The imshow windows wasnt updating properly ! It seemed stuck after a few frames.

  


  Further diving into the lane with the help of following command, I concluded that the video stream that I am reading has H264 encoding

  


  ffprobe -i rtp://192.168.0.141:11024 -show_streams -show_formats


  


  Output of the above command :

  


  ffprobe version 4.2.7-0ubuntu0.1 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[rtp @ 0xaaaac81ecce0] PES packet size mismatch
    Last message repeated 62 times
[NULL @ 0xaaaac81f09b0] non-existing PPS 0 referenced
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[rtp @ 0xaaaac81ecce0] PES packet size mismatch
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[rtp @ 0xaaaac81ecce0] PES packet size mismatch
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0xaaaac81f09b0] decode_slice_header error
[h264 @ 0xaaaac81f09b0] no frame!
[rtp @ 0xaaaac81ecce0] PES packet size mismatch
    Last message repeated 187 times
Input #0, rtp, from 'rtp://192.168.0.141:11024':
  Duration: N/A, start: 1317.040656, bitrate: N/A
  Program 1 
    Stream #0:1: Video: h264 (Constrained Baseline) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1280x720, 25 fps, 25 tbr, 90k tbn
    Stream #0:0: Data: klv (KLVA / 0x41564C4B)
Unsupported codec with id 100356 for input stream 0
[STREAM]
index=0
codec_name=klv
codec_long_name=SMPTE 336M Key-Length-Value (KLV) metadata
profile=unknown
codec_type=data
codec_tag_string=KLVA
codec_tag=0x41564c4b
id=N/A
r_frame_rate=0/0
avg_frame_rate=0/0
time_base=1/90000
start_pts=118533659
start_time=1317.040656
duration_ts=N/A
duration=N/A
bit_rate=N/A
max_bit_rate=N/A
bits_per_raw_sample=N/A
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
[/STREAM]
[STREAM]
index=1
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=Constrained Baseline
codec_type=video
codec_time_base=1/50
codec_tag_string=[27][0][0][0]
codec_tag=0x001b
width=1280
height=720
coded_width=1280
coded_height=720
has_b_frames=0
sample_aspect_ratio=N/A
display_aspect_ratio=N/A
pix_fmt=yuv420p
level=31
color_range=unknown
color_space=unknown
color_transfer=unknown
color_primaries=unknown
chroma_location=left
field_order=progressive
timecode=N/A
refs=1
is_avc=false
nal_length_size=0
id=N/A
r_frame_rate=25/1
avg_frame_rate=25/1
time_base=1/90000
start_pts=118533659
start_time=1317.040656
duration_ts=N/A
duration=N/A
bit_rate=N/A
max_bit_rate=N/A
bits_per_raw_sample=8
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
[/STREAM]
[FORMAT]
filename=rtp://192.168.0.141:11024
nb_streams=2
nb_programs=1
format_name=rtp
format_long_name=RTP input
start_time=1317.040656
duration=N/A
size=N/A
bit_rate=N/A
probe_score=100
[/FORMAT]


  


  Further, in the log output, I see a lot of statements in regard to missed packets and PES packet mismatch

  


  [rtp @ 0xaaaaf31896c0] max delay reached. need to consume packet
[rtp @ 0xaaaaf31896c0] RTP: missed 98 packets
[rtp @ 0xaaaaf31896c0] Continuity check failed for pid 40 expected 14 got 10
[rtp @ 0xaaaaf31896c0] PES packet size mismatch
rtp://192.168.0.141:11024: corrupt input packet in stream 0
frame=  124 fps=2.6 q=-0.0 size=  334800kB time=00:00:05.32 bitrate=515406.0kbits/s dup=97 drop=0 speed=0.111x 


  


  What arguments do I provide to ffmpeg and in what order because my stream 0 is metadata and stream 1 is video so as to display image frame by frame with opencv ?
I would be grateful for any help that you could provide.

  


  Further, I also have a query regarding how does ffmpeg know to that it has to first convert the rtp packets into mpeg2 TS packets before segregating video stream and data stream ?

  


• Concat video/audio files downloaded from mpd dash manifest

  15 mai 2022, par Bloworlf Mathurin

  I have this manifest I fetched from an url :

  


  &lt;?xml version="1.0"?>&#xA;<mpd type="dynamic" xmlns="urn:mpeg:dash:schema:mpd:2011" profiles="urn:mpeg:dash:profile:isoff-live:2011" availabilitystarttime="2022-05-14T21:18:50-07:00" availabilityendtime="2022-05-14T22:35:36-07:00" timeshiftbufferdepth="PT20S" suggestedpresentationdelay="PT2S" minbuffertime="PT1S" publishtime="2022-05-14T21:18:50-07:00" minimumupdateperiod="PT1S" validationerrors="" currentservertimems="0" firstavtimems="1652588331877" lastvideoframets="0" loapstreamid="17944405669984029" publishframetime="939">&#xA;    <period start="PT0S">&#xA;        <adaptationset segmentalignment="true" maxwidth="432" maxheight="766" maxframerate="30">&#xA;            <representation mimetype="video/mp4" codecs="avc1.4d401e" width="432" height="766" framerate="30" startwithsap="1" bandwidth="38894" maxbandwidth="46772" playbackresolutionmos="432:82.84,720:65.43,216:70.39" qualityclass="sd" qualitylabel="432p">&#xA;                <segmenttemplate presentationtimeoffset="0" timescale="1000" initialization="some/url/some_id-init.m4v?ms=m_C&amp;amp;ccb=2-4" media="some/url/some_id-$Time$.m4v?ms=m_C&amp;amp;ccb=2-4">&#xA;                    <segmenttimeline>&#xA;                        <s t="4092613" d="2000"></s>&#xA;                        <s t="4094613" d="2000"></s>&#xA;                        <s t="4096613" d="2000"></s>&#xA;                        <s t="4098613" d="2000"></s>&#xA;                        <s t="4100613" d="2000"></s>&#xA;                        <s t="4102613" d="2000"></s>&#xA;                        <s t="4104613" d="2000"></s>&#xA;                        <s t="4106613" d="2000"></s>&#xA;                        <s t="4108613" d="2000"></s>&#xA;                        <s t="4110613" d="2000"></s>&#xA;                    </segmenttimeline>&#xA;                </segmenttemplate>&#xA;            </representation>&#xA;        </adaptationset>&#xA;        <adaptationset segmentalignment="true">&#xA;            <representation mimetype="audio/mp4" codecs="mp4a.40.2" audiosamplingrate="44100" startwithsap="1" bandwidth="48979" maxbandwidth="57732">&#xA;                <segmenttemplate presentationtimeoffset="0" timescale="1000" initialization="some/url/some_id-init.m4a?ms=m_C&amp;amp;ccb=2-4" media="some/url/some_id-$Time$.m4a?ms=m_C&amp;amp;ccb=2-4">&#xA;                    <segmenttimeline>&#xA;                        <s t="4092613" d="2000"></s>&#xA;                        <s t="4094613" d="2000"></s>&#xA;                        <s t="4096613" d="2000"></s>&#xA;                        <s t="4098613" d="2000"></s>&#xA;                        <s t="4100613" d="2000"></s>&#xA;                        <s t="4102613" d="2000"></s>&#xA;                        <s t="4104613" d="2000"></s>&#xA;                        <s t="4106613" d="2000"></s>&#xA;                        <s t="4108613" d="2000"></s>&#xA;                        <s t="4110613" d="2000"></s>&#xA;                    </segmenttimeline>&#xA;                </segmenttemplate>&#xA;            </representation>&#xA;        </adaptationset>&#xA;    </period>&#xA;</mpd>&#xA;

  


  I manage to download all the files (init file + 10 segment files) and put them in an array

  


  File[] files = downloadSegments();

  


  So I have :
[0] -> file-0.m4v (which is the init file)
[1] -> file-1.m4v (1st segment)
... and so on.

  


  My question is how can I concat/append all these files into another file (final_segment.m4v) ?

  


  I looked around a lot and am now using 'com.arthenica:ffmpeg-kit-video:4.5.1-1'
I've last tried :

  


  FFmpegSession session = FFmpegKit.execute("-i file-0.m4v -i file-1.m4v ... -c copy final_segment.m4v");


  


  Is there something I'm missing ?

  


  Also I assume that I'll have to do the same for the audio segments. So I will have 2 files (final_segment.m4v and final_segment.m4a) that I will have to merge/mix together.

  


  If you could help me with some piece of code, that would be great.

  


• Ffmpeg input seeking - "Invalid NAL unit size"

  30 janvier 2024, par Dimitris

  I'm trying to use ffmpeg to get a 10-second clip from the middle of a video. The execution time of the command is important, that's why I've decided to use combined input & output seeking (as illustrated here).
The input video file is a CMAF with fragmented MP4, duration of 10 minutes.

  


  I'm testing on a Mac, Ffmpeg version is 6.1.1.

  


  This is the command that I'm using :

  


  ffmpeg -nostdin -y -ss 290 -copyts -start_at_zero -i https://devcdn.flowplayer.com/5f07362e-c358-41d0-857a-c64302a3fcc9/cmaf/17bdb16d-71d1-414c-a291-a028bd45b9ec/playlist_360.m3u8 -ss 300.0 -t 10 -vcodec libwebp -lossless 0 -quality 60 -compression_level 2 -loop 0 -an -sn output.webp


  


  Result : no output file is created.

  


  From what I understand it fails to seek position "290" in the video, probably due to "Invalid NAL unit size" errors.

  


  Here's the output :

  


  ffmpeg version N-106797-g580fb6a8c9-tessus Copyright (c) 2000-2022 the FFmpeg developersbuilt with Apple clang version 11.0.0 (clang-1100.0.33.17)configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplaylibavutil      57. 24.101 / 57. 24.101libavcodec     59. 27.100 / 59. 27.100libavformat    59. 23.100 / 59. 23.100libavdevice    59.  6.100 / 59.  6.100libavfilter     8. 37.100 /  8. 37.100libswscale      6.  6.100 /  6.  6.100libswresample   4.  6.100 /  4.  6.100libpostproc    56.  5.100 / 56.  5.100Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'https://devcdn.flowplayer.com/5f07362e-c358-41d0-857a-c64302a3fcc9/cmaf/17bdb16d-71d1-414c-a291-a028bd45b9ec/playlist_360.cmfv':Metadata:major_brand     : isomminor_version   : 1compatible_brands: isomavc1dashcmfccreation_time   : 2024-01-30T07:41:03.000000ZDuration: 00:09:56.54, start: 0.083333, bitrate: 458 kb/sStream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 640x360 [SAR 1:1 DAR 16:9], 1 kb/s, 24 fps, 24 tbr, 90k tbn (default)Metadata:creation_time   : 2024-01-30T07:41:03.000000Zhandler_name    : ETI ISO Video Media Handlervendor_id       : [0][0][0]ffmpeg version 6.1.1-tessus  https://evermeet.cx/ffmpeg/  Copyright (c) 2000-2023 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.17)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
[hls @ 0x7fc7bb904280] Skip ('#EXT-X-VERSION:6')
[hls @ 0x7fc7bb904280] Opening 'https://devcdn.flowplayer.com/5f07362e-c358-41d0-857a-c64302a3fcc9/cmaf/17bdb16d-71d1-414c-a291-a028bd45b9ec/playlist_360.cmfv' for reading
    Last message repeated 2 times
Input #0, hls, from '[**]/playlist_360.m3u8':
  Duration: 00:09:56.46, start: 0.083333, bitrate: 0 kb/s
  Program 0 
    Metadata:
      variant_bitrate : 0
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 640x360 [SAR 1:1 DAR 16:9], 1 kb/s, 24 fps, 24 tbr, 90k tbn (default)
    Metadata:
      variant_bitrate : 0
      compatible_brands: isomavc1dashcmfc
      handler_name    : ETI ISO Video Media Handler
      vendor_id       : [0][0][0][0]
      encoder         : Elemental H.264
      major_brand     : isom
      minor_version   : 1
      creation_time   : 2024-01-30T07:41:03.000000Z
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> webp (libwebp))
[hls @ 0x7fc7bb904280] Opening 'https://devcdn.flowplayer.com/5f07362e-c358-41d0-857a-c64302a3fcc9/cmaf/17bdb16d-71d1-414c-a291-a028bd45b9ec/playlist_360.cmfv' for reading
    Last message repeated 2 times
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (1772342253 > 1534).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 1538
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (-1977545460 > 1481).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 1485
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (1694403391 > 1582).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 1586
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (-1404850266 > 1661).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 1665
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (703351242 > 1680).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 1684
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (-836978648 > 1751).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 1755
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (752797651 > 1867).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 1871
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (-1831058223 > 1833).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 1837
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (-1238958831 > 2067).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2071
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (435683248 > 2090).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2094
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (2136335178 > 2229).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2233
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (-1468707300 > 2203).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2207
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (482758774 > 2402).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2406
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (-1079612217 > 2417).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2421
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (-608087491 > 2546).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2550
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (-1457748625 > 2527).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2531
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (1933919710 > 2734).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2738
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (1004643870 > 2803).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2807
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (-207765435 > 2988).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2992
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (-196888537 > 2306).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2310
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (1118966683 > 2620).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2624
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (1325583054 > 2715).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2719
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (-2003602869 > 2906).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2910
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (1666330272 > 3085).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 3089
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (-742329993 > 2593).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2597
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (1326266794 > 2347).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2351
[NULL @ 0x7fc7bb804f40] Invalid NAL unit size (2459776 > 2155).
[NULL @ 0x7fc7bb804f40] missing picture in access unit with size 2159
[https @ 0x7fc7ba022a00] Opening 'https://devcdn.flowplayer.com/5f07362e-c358-41d0-857a-c64302a3fcc9/cmaf/17bdb16d-71d1-414c-a291-a028bd45b9ec/playlist_360.cmfv' for reading
[...]
[vost#0:0/libwebp @ 0x7fc7bbb05780] No filtered frames for output stream, trying to initialize anyway.
Output #0, webp, to 'output.webp':
  Metadata:
    encoder         : Lavf60.16.100
  Stream #0:0(und): Video: webp, yuv420p(progressive), 640x360 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 24 fps, 1k tbn (default)
    Metadata:
      variant_bitrate : 0
      compatible_brands: isomavc1dashcmfc
      handler_name    : ETI ISO Video Media Handler
      vendor_id       : [0][0][0][0]
      creation_time   : 2024-01-30T07:41:03.000000Z
      major_brand     : isom
      minor_version   : 1
      encoder         : Lavc60.31.102 libwebp
[out#0/webp @ 0x7fc7bbb04900] video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[out#0/webp @ 0x7fc7bbb04900] Output file is empty, nothing was encoded(check -ss / -t / -frames parameters if used)
frame=    0 fps=0.0 q=0.0 Lsize=       0kB time=N/A bitrate=N/A speed=N/A 


  


  What I've tried so far :

  


  


  1. Downloaded the input file to a local directory and used it as input to ffmpeg - same results.

     


  


  2. Used the mp4 file from the playlist directly as an input to ffmpeg - worked but execution time is very slow

     


  


  3. Emmited the input seeking part (-ss 290 -copyts -start_at_zero) from the command - worked but also very slow in terms of execution time

     


  


  


  Any ideas on why I'm getting "Invalid NAL unit size" and how to make the command work with input seeking ?