Recherche avancée

Sur d’autres sites (11148)

• ffmpeg lags when streaming video+audio from RPi Zero W with Logitech C920

  7 janvier 2021, par Ema

  I've been trying to setup a baby monitor with a Raspberry Pi Zero and a Logitech C920 webcam. I does work with VLC (cvlc) but it lags too much and gets worse over time.

  


  So I am playing around with ffmpeg and I am getting some better results. This is what I've done so far.

  


  First I set the webcam to output h264 1080p natively (the Pi Zero W can't afford to do any transcoding).

  


  v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=1


  


  Now, if I stream audio only with

  


  ffmpeg \
-f alsa \
-i hw:1,0 \
-vn \
-flags +global_header \
-acodec aac \
-ac 1 \
-ar 16000 \
-ab 16k \
-f rtp rtp://192.168.0.10:5002 > audio.sdp


  


  it works great and the lag is about 1 second (definitely acceptable).

  


  If I stream video only with

  


  ffmpeg \
-f v4l2 \
-vcodec h264 \
-i /dev/video0 \
-an \
-vcodec copy \
-pix_fmt yuv420p \
-r 30 \
-b:v 512k \
-flags +global_header \
-f rtp rtp://192.168.0.10:5000 > video.sdp


  


  same result, very little lag (for some reason the first -vcodec is necessary to force the webcam to output h264).

  


  However, when I stream both with

  


  ffmpeg \
-f v4l2 \
-vcodec h264 \
-i /dev/video0 \
-f alsa \
-i hw:1,0 \
-an \
-preset ultrafast \
-tune zerolatency \
-vcodec copy \
-pix_fmt yuv420p \
-r 30 \
-b:v 512k \
-flags +global_header \
-f rtp rtp://192.168.0.10:5000 \
-vn \
-flags +global_header \
-acodec aac \
-ac 1 \
-ar 16000 \
-ab 16k \
-f rtp rtp://192.168.0.10:5002 > both.sdp


  


  the lag ramps up to 10 seconds and audio and video are out of sync. Does anybody know why ?

  


  I've tried UDP and TCP instead of RTP but then the lag is always high, even with audio/video only.

  


  Any suggestion is much appreciated.

  


  P.S. On the client side (MacOS) I'm receiving with

  


  ffplay -protocol_whitelist file,rtp,udp -i file.sdp


  


• How to use audio frame after decode mp3 file using pyav, ffmpeg, python

  2 janvier 2021, par Long Tran Dai

  I am using using python with pyav, ffmpeg to decode mp3 in the memory. I know there are some other way to do it, like pipe ffmpeg command. However, I would like to explore pyav and ffmpeg API. So I have the following code. It works but the sound is very noisy, although hearable :

  


  import numpy as np&#xA;import av # to convert mp3 to wav using ffmpeg&#xA;import pyaudio # to play music&#xA;&#xA;mp3_path = &#x27;D:/MyProg/python/SauTimThiepHong.mp3&#x27;&#xA;&#xA;def decodeStream(mp3_path):&#xA;  # Run NOT OK&#xA;  &#xA;  container = av.open(mp3_path)&#xA;  stream = next(s for s in container.streams if s.type == &#x27;audio&#x27;)&#xA;  frame_count = 0&#xA;  data = bytearray()&#xA;  for packet in container.demux(stream):&#xA;    # <class>&#xA;    # We need to skip the "flushing" packets that `demux` generates.&#xA;    #if frame_count == 5000 : break         &#xA;    if packet.dts is None:&#xA;        continue&#xA;    for frame in packet.decode():   &#xA;        #&#xA;        # type(frame) : <class>&#xA;        #frame.samples = 1152 : 1152 diem du lieu : Number of audio samples (per channel)&#xA;        # moi frame co size = 1152 (diem) * 2 (channels) * 4 (bytes / diem) = 9216 bytes&#xA;        # 11021 frames&#xA;        #arr = frame.to_ndarray() # arr.nbytes = 9216&#xA;&#xA;        #channels = []  &#xA;        channels = frame.to_ndarray().astype("float16")&#xA;        #for plane in frame.planes:&#xA;            #channels.append(plane.to_bytes()) #plane has 4 bytes / sample, but audio has only 2 bytes&#xA;        #    channels.append(np.frombuffer(plane, dtype=np.single).astype("float16"))&#xA;            #channels.append(np.frombuffer(plane, dtype=np.single)) # kieu np.single co 4 bytes&#xA;        if not frame.is_corrupt:&#xA;            #data.extend(np.frombuffer(frame.planes[0], dtype=np.single).astype("float16")) # 1 channel: noisy&#xA;            # type(planes) : <class>&#xA;            frame_count &#x2B;= 1&#xA;            #print( &#x27;>>>> %04d&#x27; % frame_count, frame)   &#xA;            #if frame_count == 5000 : break     &#xA;            # mix channels:&#xA;            for i in range(frame.samples):                &#xA;                for ch in channels: # dec_ctx->channels&#xA;                    data.extend(ch[i]) #noisy&#xA;                    #fwrite(frame->data[ch] &#x2B; data_size*i, 1, data_size, outfile)&#xA;  return bytes(data)&#xA;</class></class></class>

  


  I use pipe ffmpeg to get decoded data to compare and find they are different :

  


  def RunFFMPEG(mp3_path, target_fs = "44100"):
    # Run OK
    import subprocess
    # init command
    ffmpeg_command = ["ffmpeg", "-i", mp3_path,
                   "-ab", "128k", "-acodec", "pcm_s16le", "-ac", "0", "-ar", target_fs, "-map",
                   "0:a", "-map_metadata", "-1", "-sn", "-vn", "-y",
                   "-f", "wav", "pipe:1"]
    # excute ffmpeg command
    pipe = subprocess.run(ffmpeg_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize= 10**8)
    # debug
    #print(pipe.stdout, pipe.stderr)
    # read signal as numpy array and assign sampling rate
    #audio_np = np.frombuffer(buffer=pipe.stdout, dtype=np.uint16, offset=44)
    #audio_np = np.frombuffer(buffer=pipe.stdout, dtype=np.uint16)
    #sig, fs  = audio_np, target_fs
    #return audio_np
    return pipe.stdout[78:]     


  


  Then I use pyaudio to play data and find it very noisy

  


  p = pyaudio.PyAudio()
streamOut = p.open(format=pyaudio.paInt16, channels=2, rate= 44100, output=True)
#streamOut = p.open(format=pyaudio.paInt16, channels=1, rate= 44100, output=True)

mydata = decodeStream(mp3_path)
print("bytes of mydata = ", len(mydata))
#print("bytes of mydata = ", mydata.nbytes)

ffMpegdata = RunFFMPEG(mp3_path)
print("bytes of ffMpegdata = ", len(ffMpegdata)) 
#print("bytes of ffMpegdata = ", ffMpegdata.nbytes)

minlen = min(len(mydata), len(ffMpegdata))
print("mydata == ffMpegdata", mydata[:minlen] == ffMpegdata[:minlen]) # ffMpegdata.tobytes()[:minlen] )

#bytes of mydata =  50784768
#bytes of ffMpegdata =  50784768
#mydata == ffMpegdata False

streamOut.write(mydata)
streamOut.write(ffMpegdata)
streamOut.stop_stream()
streamOut.close()
p.terminate()


  


  Please help me to understand decoded frame of pyav api (after for frame in packet.decode() :). Should it be processed more ? or I have some error ?

  


  It makes me crazy for 3 days. I could not guess where to go.

  


  Thank you very much.

  


• Merge image, audio, video with no audio, video with audio, with ffmpeg

  17 février 2021, par Basj

  Similarly to Merge videos and images using ffmpeg (which is not a duplicate for the reasons explained below), I'd like to merge multiple inputs which can be either :

  


  

  • image only,

  


  • audio only,

  


  • video with audio,

  


  • video without audio

  


  


  into one output video, with stereo audio.

  


  Note : If multiple audio channels are playing at the same time, they should be mixed ; idem for video : the images from multiple sources should overlap.

  


  I tried this (comments added here) :

  


  ffmpeg 
  -i tmp/%04d.png       # [0]
  -f lavfi -t 0.1 -i anullsrc   # [1], if needed for inputs without sound?
  -i a.mp3              # [2], we keep 1 sec. from it; should start at 0'05" in output video
  -i b.mp3              # [3], we keep 2 sec. from it; should start at 0'06" in output video
  -i with_sound.mp4     # [4], we keep 3 sec. from it; should start at 0'07" in output video
  -i without_sound.mp4  # [5], we keep 4 sec. from it; should start at 0'08" in output video
  -filter_complex 
    [2]atrim=start=0:duration=1.0,asetpts=PTS-STARTPTS[s2];[s2]adelay=5000|5000[t2];
    [3]atrim=start=0:duration=2.0,asetpts=PTS-STARTPTS[s3];[s3]adelay=6000|6000[t3];
    [4]atrim=start=0:duration=3.0,asetpts=PTS-STARTPTS[s4];[s4]adelay=7000|7000[t4];
    [5]atrim=start=0:duration=4.0,asetpts=PTS-STARTPTS[s5];[s5]adelay=8000|8000[t5];
    [0][1][t2][t3][t4][t5]concat=n=6:a=1:v=1:unsafe=1[outv][outa]
  -map [outv] -map [outa] out.mp4


  


  I tried with various values concat=n=5, n=6, etc. and added unsafe=1, but I always get similar errors :

  


  


  [Parsed_adelay_2 @ 00000000006e8140] Media type mismatch between the 'Parsed_adelay_2' filter output pad 0 (audio) and the 'Parsed_concat_6' filter input pad 2 (video)
  
[AVFilterGraph @ 00000000006923c0] Cannot create the link adelay:0 -> concat:2

  


  


  or for the times I got it nearly working, the videos were added one after another and not merged / mixed.

  


  Also, I'm looking for a syntax that would work even if I don't know in advance if the input videos have or don't have audio (I'm doing a script and I don't know in advance if the videos have audio channels).

  


  ---

  


  TL ;DR :

  


  Question : How to mix/merge multiple inputs (image, audio, video with-or-without-sound) with ffmpeg, with a precise starting timestamp for each, into a single video output ?