Recherche avancée

Sur d’autres sites (5940)

• ffmpeg "Manifest too large" when downloading youtube video

  22 avril 2024, par ynn

  TL ;DR

  


  I'm trying to programmatically download a part of a YouTube video. The widely known procedure doesn't work for some videos and I'd like to overcome this situation.

  


  ---

  


  Context

  


  I'm trying to programmatically download a part of a YouTube video. As described in How to download portion of video with youtube-dl command ?, you can achieve this by the following commands.

  


  #Converts a human-readable URL to longer URLs for internal use.
~ $ youtube-dl --get-url 'https://www.youtube.com/watch?v=POrfo478HRI'
https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=299&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C298%2C299%2C302%2C303&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=video%2Fmp4&gir=yes&clen=9205054989&dur=17931.000&lmt=1597347724488304&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7316222&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgYDqTm0CnLfG0kxsYNrAdtAUYB7alnowiaBArU8R5bBwCIQDeCRDJHFO_PWcnbeFaJvip80deuboN4Pi1x3eRhJBxlQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes
https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=251&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=audio%2Fwebm&gir=yes&clen=268945126&dur=17931.001&lmt=1597348111385641&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7311222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAK8XGu3vHucHwqERZ7TQniuvKEd-NTbMkwZbu8EXJ5E3AiEAiS7OYFOcsIrD2xp-AJNzucj1H9ZKMlmkCl_sU7__dZo%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes

#Stores the result to a variable.
~ $ url_video_part='https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=299&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C298%2C299%2C302%2C303&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=video%2Fmp4&gir=yes&clen=9205054989&dur=17931.000&lmt=1597347724488304&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7316222&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgYDqTm0CnLfG0kxsYNrAdtAUYB7alnowiaBArU8R5bBwCIQDeCRDJHFO_PWcnbeFaJvip80deuboN4Pi1x3eRhJBxlQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes'

#Stores the result to a variable.
~ $ url_audio_part='https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=251&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=audio%2Fwebm&gir=yes&clen=268945126&dur=17931.001&lmt=1597348111385641&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7311222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAK8XGu3vHucHwqERZ7TQniuvKEd-NTbMkwZbu8EXJ5E3AiEAiS7OYFOcsIrD2xp-AJNzucj1H9ZKMlmkCl_sU7__dZo%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes'

#Downloads the video for one minute from 12:45.
~ $ ffmpeg    -ss 12:45 -i "${url_video_part}"    -ss 12:45 -i "${url_audio_part}"    -t 1:00 output.mkv
frame= 3600 fps= 49 q=-1.0 Lsize=   27047kB time=00:00:59.99 bitrate=3692.9kbits/s speed=0.813x


  


  This does work perfectly.

  


  ---

  


  Problem

  


  For some videos^†1, the internal URLs are of the form https://manifest.googlevideo.com/... instead of https://RANDOM-STRING.googlevideo.com/... as in the example above.

  


  ~ $ youtube-dl --get-url 'https://www.youtube.com/watch?v=KRMfnvLODZQ'
https://manifest.googlevideo.com/api/manifest/dash/expire/1597593866/ei/qgQ5X5maN8-ilQSu15W4CQ/ip/10.100.238.99/id/KRMfnvLODZQ.1/source/yt_live_broadcast/requiressl/yes/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhANYKQvKTN8BRmhp7--4tQp9aqRp_qIGjh472BTYLyJ62AiAautFF86xGDHOe7mYlYaZ1W86EM4k2e6UiJS9Q91dMog%3D%3D
https://manifest.googlevideo.com/api/manifest/dash/expire/1597593866/ei/qgQ5X5maN8-ilQSu15W4CQ/ip/10.100.238.99/id/KRMfnvLODZQ.1/source/yt_live_broadcast/requiressl/yes/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhANYKQvKTN8BRmhp7--4tQp9aqRp_qIGjh472BTYLyJ62AiAautFF86xGDHOe7mYlYaZ1W86EM4k2e6UiJS9Q91dMog%3D%3D


  


  And downloading such videos always fails.

  


  ~ $ ffmpeg    -ss 12:45 -i "${url_video_part}"    -ss 12:45 -i "${url_audio_part}"    -t 1:00 output.mkv
[dash @ 0x55fa6a1a4400] Manifest too large: 3776722
https://manifest.googlevideo.com/...: Invalid data found when processing input


  


  Please note downloading full of such videos using youtube-dl alone succeeds.

  


  _{†1 : As far as we tested, recently (e.g. yesterday) uploaded videos have such URLs and when we retry the same youtube-dl --get-url command after a day or so, we get "innocent" URLs like in the first example.}

  


  ---

  


  Questions

  


  How can I download parts of such videos

  


  


  • by specifying additional options for ffmpeg,

    


  


  • by somehow forcefully retrieving internal URLs of the form https://RANDOM-STRING.googlevideo.com/...,

    


  


  • or by something which I have no idea at all ?

    


  


  


  My current workaround is just to wait for a day or two until the internal URLs change, which is much slower than downloading the full video and then cropping it.

  


  ---

  


  Environments

  


  ~ $ lsb_release -d
Description:    Arch Linux

~ $ ffmpeg -version
ffmpeg version n4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 10.1.0 (GCC)
configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100

~ $ youtube-dl --version
2020.07.28


  


• Live streaming Channel Multiview used ffmpeg X11grab and shaka packager

  18 août 2020, par jgkim0518

  I have been trying to live stream channel multiview used xvfb, firefox, vlc, ffmpeg and shaka packager.
1). Making virtual display used Xvfb.
2). Popping up a web page used fireforx on 1).
3). Drawing nine video(each channel) used vlc plugin on 2).
4). Capturing 2) used ffmpeg library, x11grab.
5). Transcoding 4) to FHD(HEVC)+10 Audio(each channel sound + dummy), FHD(H.264)+no sound, HD(H.264)+no sound, SD(H.264)+no sound, nHD(H.264)+no sound
6). packaging 5) used Shaka Packager

  


  This is my command. shell script

  


  function exec-xvfb () {
$XVFB :$display_port -br -noreset -nolisten tcp -screen 0 1920x1080x24 &}

function exec-firefox () {
    AUDIO_CTRL_PATH="$audio_ctrl_port:/TEST/vlcenv/.sync.dat" VLC_INFO_PATH=/TEST/vlcenv/vlc_info.dat AUDIO_SHARE_KEY="$AUDIO_SHARE_KEY" AUDIO_SHARE_PATH=/TEST/vlcenv/.share.shm MOSAIC_META_PATH=/TEST/meta_bin.dat DISPLAY=":$display_port" $FIREFOX --display=:$display_port --no remote --profile /TEST/browser -width 1920 -height 1080 http://127.0.0.1:8021/template/best_channel_9_1/page/index.html &
}

function transcoder () {
    AUDIO_CTRL_PATH="$audio_ctrl_port:/TEST/vlcenv/.sync.dat" VLC_INFO_PATH=/TEST/vlcenv/vlc_info.dat 
AUDIO_SHARE_KEY="$AUDIO_SHARE_KEY" AUDIO_SHARE_PATH=/TEST/vlcenv/.share.shm 
MOSAIC_META_PATH=/TEST/meta_bin.dat DISPLAY=":$display_port" 
LD_LIBRARY_PATH=/transcoder/lib:/transcoder/lib/cuda /transcoder/gpu/transcoder -y -re \
    -analyzeduration 0 -f s16le -ar 48000 -ac 2 -pkt_size 128000 -i syc://@:$trc_sync_port?dummy=1 \
    -analyzeduration 0 -f s16le -ar 48000 -ac 2 -pkt_size 128000 -i syc://@:$(($trc_sync_port+1)) \
    -analyzeduration 0 -f s16le -ar 48000 -ac 2 -pkt_size 128000 -i syc://@:$(($trc_sync_port+2)) \
    -analyzeduration 0 -f s16le -ar 48000 -ac 2 -pkt_size 128000 -i syc://@:$(($trc_sync_port+3)) \
    -analyzeduration 0 -f s16le -ar 48000 -ac 2 -pkt_size 128000 -i syc://@:$(($trc_sync_port+4)) \
    -analyzeduration 0 -f s16le -ar 48000 -ac 2 -pkt_size 128000 -i syc://@:$(($trc_sync_port+5)) \
    -analyzeduration 0 -f s16le -ar 48000 -ac 2 -pkt_size 128000 -i syc://@:$(($trc_sync_port+6)) \
    -analyzeduration 0 -f s16le -ar 48000 -ac 2 -pkt_size 128000 -i syc://@:$(($trc_sync_port+7)) \
    -analyzeduration 0 -f s16le -ar 48000 -ac 2 -pkt_size 128000 -i syc://@:$(($trc_sync_port+8)) \
    -analyzeduration 0 -f s16le -ar 48000 -ac 2 -pkt_size 128000 -i syc://@:$(($trc_sync_port+9)) \
    -analyzeduration 0 -f x11grab -draw_mouse 0 -video_size 1920x1080 -framerate 29.97 -i :$display_port.0+0,0,nomouse \
    -c:a libfdk_aac -ar:a 48000 -ac:a 2 -b:a 128000 -vf hwupload_cuda=device=$gpu_no,scale_npp=1920:1080:format=yuv420p -gpu $gpu_no -c:v hevc_nvenc -preset:v fast -profile:v main -r 29.97 -g 15 -b:v 6000000 -bufsize 14000000 -maxrate 7000000 \
    -map 0:a -map 1:a -map 2:a -map 3:a -map 4:a -map 5:a -map 6:a -map 7:a -map 8:a -map 9:a -map 10:v -f mpegts udp://127.0.0.1:$trc_output_port?pkt_size=1316 \
    -an -vf hwupload_cuda=device=$gpu_no,scale_npp=1920:1080:format=yuv420p -gpu $gpu_no -c:v h264_nvenc -preset:v fast -profile:v main -r 29.97 -g 15 -b:v 4000000 -bufsize 8000000 -maxrate 4000000 -f mpegts udp://127.0.0.1:$(($trc_output_port+1))?pkt_size=1316 \
    -an -vf hwupload_cuda=device=$gpu_no,scale_npp=1280:720:format=yuv420p -gpu $gpu_no -c:v h264_nvenc -preset:v fast -profile:v main -r 29.97 -g 15 -b:v 2000000 -bufsize 4000000 -maxrate 2000000 -f mpegts udp://127.0.0.1:$(($trc_output_port+2))?pkt_size=1316 \
    -an -vf hwupload_cuda=device=$gpu_no,scale_npp=720:480:format=yuv420p -gpu $gpu_no -c:v h264_nvenc -preset:v fast -profile:v main -r 29.97 -g 15 -b:v 1500000 -bufsize 3000000 -maxrate 1500000 -f mpegts udp://127.0.0.1:$(($trc_output_port+3))?pkt_size=1316 \
    -an -vf hwupload_cuda=device=$gpu_no,scale_npp=640:360:format=yuv420p -gpu $gpu_no -c:v h264_nvenc -preset:v fast -profile:v main -r 29.97 -g 15 -b:v 800000 -bufsize 1600000 -maxrate 800000 -f mpegts udp://127.0.0.1:$(($trc_output_port+4))?pkt_size=1316 \main &
}

function packager () { 
    rm -rf $OUTPUT/*
    /packager/packager \ 

    "in=udp://127.0.0.1:$trc_output_port,stream=0,init_segment=$OUTPUT/audio/0/audio_4.mp4,segment_template=$OUTPUT/audio/0/\$Time\$.m4s" \                 "in=udp://127.0.0.1:$trc_output_port,stream=1,init_segment=$OUTPUT/audio/1/audio_4.mp4,segment_template=$OUTPUT/audio/1/\$Time\$.m4s" \
    "in=udp://127.0.0.1:$trc_output_port,stream=2,init_segment=$OUTPUT/audio/2/audio_4.mp4,segment_template=$OUTPUT/audio/2/\$Time\$.m4s" \
    "in=udp://127.0.0.1:$trc_output_port,stream=3,init_segment=$OUTPUT/audio/3/audio_4.mp4,segment_template=$OUTPUT/audio/3/\$Time\$.m4s" \
    "in=udp://127.0.0.1:$trc_output_port,stream=4,init_segment=$OUTPUT/audio/4/audio_4.mp4,segment_template=$OUTPUT/audio/4/\$Time\$.m4s" \
    "in=udp://127.0.0.1:$trc_output_port,stream=5,init_segment=$OUTPUT/audio/5/audio_4.mp4,segment_template=$OUTPUT/audio/5/\$Time\$.m4s" \
    "in=udp://127.0.0.1:$trc_output_port,stream=6,init_segment=$OUTPUT/audio/6/audio_4.mp4,segment_template=$OUTPUT/audio/6/\$Time\$.m4s" \
    "in=udp://127.0.0.1:$trc_output_port,stream=7,init_segment=$OUTPUT/audio/7/audio_4.mp4,segment_template=$OUTPUT/audio/7/\$Time\$.m4s" \
    "in=udp://127.0.0.1:$trc_output_port,stream=8,init_segment=$OUTPUT/audio/8/audio_4.mp4,segment_template=$OUTPUT/audio/8/\$Time\$.m4s" \
    "in=udp://127.0.0.1:$trc_output_port,stream=9,init_segment=$OUTPUT/audio/9/audio_4.mp4,segment_template=$OUTPUT/audio/9/\$Time\$.m4s" \
    "in=udp://127.0.0.1:$trc_output_port,stream=video,init_segment=$OUTPUT/video/0/video_4.mp4,segment_template=$OUTPUT/video/0/\$Time\$.m4s" \
    "in=udp://127.0.0.1:$(($trc_output_port+1)),stream=video,init_segment=$OUTPUT/video/1/video_4.mp4,segment_template=$OUTPUT/video/1/\$Time\$.m4s" \
    "in=udp://127.0.0.1:$(($trc_output_port+2)),stream=video,init_segment=$OUTPUT/video/2/video_4.mp4,segment_template=$OUTPUT/video/2/\$Time\$.m4s" \
    "in=udp://127.0.0.1:$(($trc_output_port+3)),stream=video,init_segment=$OUTPUT/video/3/video_4.mp4,segment_template=$OUTPUT/video/3/\$Time\$.m4s" \
    "in=udp://127.0.0.1:$(($trc_output_port+4)),stream=video,init_segment=$OUTPUT/video/4/video_4.mp4,segment_template=$OUTPUT/video/4/\$Time\$.m4s" \
    --segment_duration 10 --fragment_duration 10 --minimum_update_period 10 --allow_codec_switching --preserved_segments_outside_live_window 24 --time_shift_buffer_depth 40 \
    --mpd_output $OUTPUT/$output_mpd.mpd
}


  


  vlc warning and error messages when duration 7

  


  ...
adaptive debug: Timeline
adaptive debug: Element #1 d=669670 r=0 @t=6321195
adaptive debug: Element #2 d=576576 r=0 @t=6990865
adaptive debug: Element #3 d=669670 r=0 @t=7567441
adaptive debug: Element #4 d=672673 r=0 @t=8237111
adaptive debug: Element #5 d=561561 r=0 @t=8909784
adaptive debug: Element #6 d=654655 r=0 @t=9471345
adaptive debug: Element #7 d=630631 r=0 @t=10126000
adaptive debug: Updated MPD, next update in 7s
main warning: picture is too late to be displayed (missing 223 ms)
main warning: picture is too late to be displayed (missing 157 ms)
main warning: picture is too late to be displayed (missing 91 ms)
main warning: picture is too late to be displayed (missing 24 ms)
adaptive debug: Retrieving http://192.168.0.181:81/300/audio/5/9451440.m4s @0
adaptive debug: Retrieving http://192.168.0.181:81/300/video/0/9475185.m4s @0
adaptive debug: Retrieving http://192.168.0.181:81/300/300.mpd @0
main debug: creating access: http://192.168.0.181:81/300/300.mpd
main debug: (path: \\192.168.0.181:81\300\300.mpd)
main debug: looking for access module matching "http": 26 candidates
http debug: resolving 192.168.0.181 ...
http debug: outgoing request: GET /300/300.mpd HTTP/1.1 Host:192.168.0.181:81 Accept: */* Accept-Language: en_US User-Agent: VLC/3.0.11 LibVLC/3.0.11 Range: bytes=0- 
http debug: incoming response: HTTP/1.1 206 Partial Content Server: nginx Date: Mon, 17 Aug 2020 09:10:08 GMT Content-Type: application/octet-stream Content-Length: 10406 Last-Modified: Mon, 17 Aug 2020 09:10:08 GMT Connection: keep-alive ETag: "5f3a4970-28a6" Content-Range: bytes 0-10405/10406 
main debug: using access module "access"
main debug: looking for stream_filter module matching "prefetch,cache_block": 24 candidates
prefetch debug: using 10406 bytes buffer, 10406 bytes read
main debug: using stream_filter module "prefetch"
main debug: looking for stream_filter module matching "any": 24 candidates
main debug: no stream_filter modules matched
main debug: looking for stream_filter module matching "inflate": 24 candidates
main debug: no stream_filter modules matched
prefetch debug: end of stream
main debug: removing module "prefetch"
main debug: removing module "access"
main debug: looking for xml reader module matching "any": 1 candidates
main debug: using xml reader module "xml"
adaptive debug: MPD profile=urn:mpeg:dash:profile:isoff-live:2011 mediaPresentationDuration=0 minBufferTime=2
adaptive debug: BaseUrl=http://192.168.0.181:81/300/
adaptive debug: Period
adaptive debug: BaseAdaptationSet 0
adaptive debug: Representation 0 [mp4a]
adaptive debug: InitSegmentTemplate #0 url=http://192.168.0.181:81/300/audio/0/audio_4.mp4 duration 0
adaptive debug: SegmentTemplate #12 url=http://192.168.0.181:81/300/audio/0/$Time$.m4s duration 0
adaptive debug: Timeline
adaptive debug: Element #1 d=629760 r=1 @t=6930480
adaptive debug: Element #3 d=631680 r=0 @t=8190000
adaptive debug: Element #4 d=629760 r=3 @t=8821680
adaptive debug: Representation 1 [mp4a]
adaptive debug: InitSegmentTemplate #0 url=http://192.168.0.181:81/300/audio/1/audio_4.mp4 duration 0
adaptive debug: SegmentTemplate #12 url=http://192.168.0.181:81/300/audio/1/$Time$.m4s duration 0
...
adaptive debug: Timeline
adaptive debug: Element #1 d=576576 r=0 @t=6990865
adaptive debug: Element #2 d=669670 r=0 @t=7567441
adaptive debug: Element #3 d=672673 r=0 @t=8237111
adaptive debug: Element #4 d=561561 r=0 @t=8909784
adaptive debug: Element #5 d=654655 r=0 @t=9471345
adaptive debug: Element #6 d=630631 r=0 @t=10126000
adaptive debug: Element #7 d=666666 r=0 @t=10756631
adaptive debug: Updated MPD, next update in 7s
main error: ES_OUT_SET_(GROUP_)PCR is called too late (pts_delay increased to 2168 ms)
main debug: ES_OUT_RESET_PCR called
main debug: Buffering 0%
main debug: Buffering 1%
main debug: Buffering 2%
main debug: Received first picture
mmdevice debug: state changed: 0
main debug: Buffering 4%
...
main debug: Buffering 15%
wasapi debug: reset
main debug: Buffering 17%
...
main debug: Buffering 98%
main debug: Stream buffering done (2176 ms in 16 ms)
main debug: Decoder wait done in 0 ms
main warning: playback too early (-110003): down-sampling
mmdevice debug: state changed: 1
adaptive debug: Retrieving http://192.168.0.181:81/300/video/0/10129840.m4s @0
adaptive debug: Retrieving http://192.168.0.181:81/300/audio/5/10081200.m4s @0
adaptive debug: Retrieving http://192.168.0.181:81/300/300.mpd @0
main debug: creating access: http://192.168.0.181:81/300/300.mpd
main debug: (path: \\192.168.0.181:81\300\300.mpd)
main debug: looking for access module matching "http": 26 candidates
http debug: resolving 192.168.0.181 ...
http debug: outgoing request: GET /300/300.mpd HTTP/1.1 Host: 192.168.0.181:81 Accept: */* Accept-Language: en_US User-Agent: VLC/3.0.11 LibVLC/3.0.11 Range: bytes=0- 
http debug: incoming response: HTTP/1.1 206 Partial Content Server: nginx Date: Mon, 17 Aug 2020 09:10:15 GMT Content-Type: application/octet-stream Content-Length: 10387 Last-Modified: Mon, 17 Aug 2020 09:10:15 GMT Connection: keep-alive ETag: "5f3a4977-2893" Content-Range: bytes 0-10386/10387 
main debug: using access module "access"
main debug: looking for stream_filter module matching "prefetch,cache_block": 24 candidates
prefetch debug: using 10387 bytes buffer, 10387 bytes read
main debug: using stream_filter module "prefetch"
main debug: looking for stream_filter module matching "any": 24 candidates
main debug: no stream_filter modules matched
main debug: looking for stream_filter module matching "inflate": 24 candidates
main debug: no stream_filter modules matched
prefetch debug: end of stream
main debug: removing module "prefetch"
main debug: removing module "access"
main debug: looking for xml reader module matching "any": 1 candidates
main debug: using xml reader module "xml"
adaptive debug: MPD profile=urn:mpeg:dash:profile:isoff-live:2011 mediaPresentationDuration=0 minBufferTime=2
adaptive debug: BaseUrl=http://192.168.0.181:81/300/
adaptive debug: Period
adaptive debug: BaseAdaptationSet 0
adaptive debug: Representation 0 [mp4a]
adaptive debug: InitSegmentTemplate #0 url=http://192.168.0.181:81/300/audio/0/audio_4.mp4 duration 0
adaptive debug: SegmentTemplate #13 url=http://192.168.0.181:81/300/audio/0/$Time$.m4s duration 0
...


  


  keep getting a similar message

  


  vlc warning and error messages when duration 5

  


  adaptive debug: Timeline
adaptive debug: Element #1 d=456457 r=0 @t=9501375
adaptive debug: Element #2 d=453453 r=0 @t=9957832
adaptive debug: Element #3 d=471472 r=0 @t=10411285
adaptive debug: Element #4 d=381381 r=0 @t=10882757
adaptive debug: Element #5 d=477478 r=0 @t=11264138
adaptive debug: Element #6 d=468468 r=0 @t=11741616
adaptive debug: Element #7 d=474475 r=0 @t=12210084
adaptive debug: Element #8 d=378378 r=0 @t=12684559
adaptive debug: Element #9 d=468468 r=0 @t=13062937
adaptive debug: Element #10 d=474475 r=0 @t=13531405
adaptive debug: Updated MPD, next update in 5s
adaptive debug: Retrieving http://192.168.0.181:81/300/audio/5/8101680.m4s @0
main error: ES_OUT_SET_(GROUP_)PCR is called too late (pts_delay increased to 4794 ms)
main warning: buffer too late (-588333 us): dropped
main debug: ES_OUT_RESET_PCR called
main debug: Buffering 0%
main debug: Buffering 1%
main debug: Buffering 2%
main debug: Buffering 3%
...
main debug: Buffering 15%
main debug: Received first picture
main debug: Buffering 16%
...
main debug: Buffering 99%
main debug: Stream buffering done (4804 ms in 122 ms)
main debug: Decoder wait done in 0 ms
main debug: inserting 3246 zeroes
mmdevice debug: state changed: 0
wasapi debug: reset
main warning: playback too early (-85958): down-sampling
mmdevice debug: state changed: 1
adaptive debug: Retrieving http://192.168.0.181:81/300/video/0/8114825.m4s @0
adaptive debug: Retrieving http://192.168.0.181:81/300/300.mpd @0
main debug: creating access: http://192.168.0.181:81/300/300.mpd
main debug: (path: \\192.168.0.181:81\300\300.mpd)
main debug: looking for access module matching "http": 26 candidates
http debug: resolving 192.168.0.181 ...
http debug: outgoing request: GET /300/300.mpd HTTP/1.1 Host: 192.168.0.181:81 Accept: */* Accept-Language: en_US User-Agent: VLC/3.0.11 LibVLC/3.0.11 Range: bytes=0- 
http debug: incoming response: HTTP/1.1 206 Partial Content Server: nginx Date: Mon, 17 Aug 2020 09:18:28 GMT Content-Type: application/octet-stream Content-Length: 12632 Last-Modified: Mon, 17 Aug 2020 09:18:28 GMT Connection: keep-alive ETag: "5f3a4b64-3158" Content-Range: bytes 0-12631/12632 
main debug: using access module "access"
main debug: looking for stream_filter module matching "prefetch,cache_block": 24 candidates
prefetch debug: using 12632 bytes buffer, 12632 bytes read
main debug: using stream_filter module "prefetch"
main debug: looking for stream_filter module matching "any": 24 candidates
main debug: no stream_filter modules matched
main debug: looking for stream_filter module matching "inflate": 24 candidates
main debug: no stream_filter modules matched
prefetch debug: end of stream
main debug: removing module "prefetch"
main debug: removing module "access"
main debug: looking for xml reader module matching "any": 1 candidates
main debug: using xml reader module "xml"
adaptive debug: MPD profile=urn:mpeg:dash:profile:isoff-live:2011 mediaPresentationDuration=0 minBufferTime=2
adaptive debug: BaseUrl=http://192.168.0.181:81/300/
adaptive debug: Period
adaptive debug: BaseAdaptationSet 0
adaptive debug: Representation 0 [mp4a]
adaptive debug: InitSegmentTemplate #0 url=http://192.168.0.181:81/300/audio/0/audio_4.mp4 duration 0
adaptive debug: SegmentTemplate #12 url=http://192.168.0.181:81/300/audio/0/$Time$.m4s duration 0
...
adaptive debug: Timeline
adaptive debug: Element #1 d=453453 r=0 @t=9957832
adaptive debug: Element #2 d=471472 r=0 @t=10411285
adaptive debug: Element #3 d=381381 r=0 @t=10882757
adaptive debug: Element #4 d=477478 r=0 @t=11264138
adaptive debug: Element #5 d=468468 r=0 @t=11741616
adaptive debug: Element #6 d=474475 r=0 @t=12210084
adaptive debug: Element #7 d=378378 r=0 @t=12684559
adaptive debug: Element #8 d=468468 r=0 @t=13062937
adaptive debug: Element #9 d=474475 r=0 @t=13531405
adaptive debug: Element #10 d=459459 r=0 @t=14005880
adaptive debug: Updated MPD, next update in 5s
adaptive debug: Retrieving http://192.168.0.181:81/300/audio/5/8550960.m4s @0
main warning: picture is too late to be displayed (missing 383 ms)
main warning: picture is too late to be displayed (missing 317 ms)
main warning: picture is too late to be displayed (missing 251 ms)
main warning: picture is too late to be displayed (missing 184 ms)
main warning: picture is too late to be displayed (missing 117 ms)
adaptive debug: Retrieving http://192.168.0.181:81/300/300.mpd @0
main debug: creating access: http://192.168.0.181:81/300/300.mpd
main debug: (path: \\192.168.0.181:81\300\300.mpd)
main debug: looking for access module matching "http": 26 candidates
http debug: resolving 192.168.0.181 ...
http debug: outgoing request: GET /300/300.mpd HTTP/1.1 Host: 192.168.0.181:81 Accept: */* Accept-Language: en_US User-Agent: VLC/3.0.11 LibVLC/3.0.11 Range: bytes=0- 
http debug: incoming response: HTTP/1.1 206 Partial Content Server: nginx Date: Mon, 17 Aug 2020 09:18:33 GMT Content-Type: application/octet-stream Content-Length: 12596 Last-Modified: Mon, 17 Aug 2020 09:18:33 GMT Connection: keep-alive ETag: "5f3a4b69-3134" Content-Range: bytes 0-12595/12596 
main debug: using access module "access"
main debug: looking for stream_filter module matching "prefetch,cache_block": 24 candidates
prefetch debug: using 12596 bytes buffer, 12596 bytes read
main debug: using stream_filter module "prefetch"
main debug: looking for stream_filter module matching "any": 24 candidates
main debug: no stream_filter modules matched
main debug: looking for stream_filter module matching "inflate": 24 candidates
main debug: no stream_filter modules matched
prefetch debug: end of stream
main debug: removing module "prefetch"
main debug: removing module "access"
main debug: looking for xml reader module matching "any": 1 candidates
main debug: using xml reader module "xml"
adaptive debug: MPD profile=urn:mpeg:dash:profile:isoff-live:2011 mediaPresentationDuration=0 minBufferTime=2
adaptive debug: BaseUrl=http://192.168.0.181:81/300/
adaptive debug: Period
adaptive debug: BaseAdaptationSet 0
adaptive debug: Representation 0 [mp4a]
adaptive debug: InitSegmentTemplate #0 url=http://192.168.0.181:81/300/audio/0/audio_4.mp4 duration 0
adaptive debug: SegmentTemplate #13 url=http://192.168.0.181:81/300/audio/0/$Time$.m4s duration 0
...


  


  keep getting a similar message

  


  This is played on vlc. this is duration 10 sec. but I want to live streaming duration 5 under.
cuda, framerate, GOP size and bitrate are fixed.

  


  I know that is inefficient. but now, I have to live stream channel multiview duration 5 under. How can I achieve ? please help me.

  


• Very slow writes on MySQL 8 - waiting for handler commit

  23 mai 2023, par Akshat Goel

  I have MySQL 8 docker installation installed on an edge device which has the following two tables to write to

  


  video_paths | CREATE TABLE `video_paths` (
  `entry` int(11) NOT NULL AUTO_INCREMENT,
  `timestamp` bigint(20) NOT NULL,
  `duration` int(11) NOT NULL,
  `path` varchar(255) NOT NULL,
  `motion` int(11) NOT NULL DEFAULT '0',
  `cam_id` varchar(255) NOT NULL DEFAULT '',
  `hd` tinyint(1) NOT NULL DEFAULT '0',
  PRIMARY KEY (`entry`),
  KEY `cam_id` (`cam_id`),
  KEY `timestamp` (`timestamp`)
) ENGINE=InnoDB AUTO_INCREMENT=7342309 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci


  


  AND

  


  CREATE TABLE `tracker` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `table_name` varchar(255) NOT NULL,
  `primary_key_name` varchar(255) NOT NULL,
  `pointer` int(11) NOT NULL DEFAULT '0',
  PRIMARY KEY (`id`),
  UNIQUE KEY `table_name` (`table_name`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci


  


  The following queries are run every few secs for up to 32 cameras and are taking a lot of time as indicated by the slow query log.

  


  UPDATE tracker SET pointer = 7342046 WHERE table_name = 'video_paths'

INSERT INTO video_paths (timestamp,duration,path,cam_id,hd) VALUES (1597548365000,5000,'/s/ss/x-0/v/2020-08-16/3/1.ts','x-1',1)



  


  Most of the time is spent in the waiting for handler commit state

  


  The total size of my data (tables + index) is 1GB and I have the following settings enabled to optimise for write

  


  skip-log-bin - Disabled the bin log because I don't have a replica and therefore no use for it
innodb_flush_log_at_trx_commit =2 - I am Optimising for performance rather than consistency here.
range_optimizer_max_mem_size =0 As mention in this question, I have allowed max memory to range optimiser.
inndo_buffer_pool_size= 512Mb - This should be enough for my data ?.
  
innodb_log_file_size= 96Mb *2 files

  


  I am seeing queries that are taking up to 90-100 secs sometimes.

  


  SET timestamp=1597549337;
INSERT INTO video_paths (timestamp,duration,path,cam_id,hd) VALUES (1597548365000,5000,'/s/ss/x-0/v/2020-08-16/3/1.ts','x-1',1);
# Time: 2020-08-16T03:42:24.533408Z
# Query_time: 96.712976  Lock_time: 0.000033 Rows_sent: 0  Rows_examined: 0


  


  ---UPDATE---
Here's the complete my.cnf file

  


  my.cnf

[mysqld]
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
datadir         = /var/lib/mysql
secure-file-priv= NULL
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

skip-log-bin
innodb_buffer_pool_size=536870912
innodb_log_file_size=100663296

# Custom config should go here
!includedir /etc/mysql/conf.d/

conf.d/docker.cnf 
[mysqld]
skip-host-cache
skip-name-resolve 


  


  The docker container is using the host mode so complete 15GB memory is available to the container.

  


  --- UPDATE 2 ---
After increasing the innodb_buffer_pool_size to 2GB as suggested by @fyrye, the statements have now started getting stuck on STATE = UPDATE instead of waiting for handler commit.

  


  ---- UPDATE 3 ---
Looks like the CPU is causing the bottleneck


  


  ** ---- UPDATE 4 ---- **
Additional info

  


  

  1. Ram Size

  


  


                total        used        free      shared  buff/cache   available
Mem:          15909        1711        9385        2491        4813       11600
Swap:             0           0           0


  


  

  2. No SSD/NVMe devices attached

  


  3. SHOW GLOBAL STATUS - https://pastebin.com/vtWi0PUq

  


  4. SHOW GLOBAL VARIABLES - https://pastebin.com/MUZeG959

  


  5. SHOW FULL PROCESSLIST - https://pastebin.com/eebEcYk7

  


  6. htop - htop here is for the edge system which has 4 other containers running which include the main app, ffmpeg, mqtt, etc.


  


  7. ulimit -a :

  


  


  core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 62576
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 62576
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


  


  

  8. opstat -xm 5 4

  


  


  Linux 4.15.0-106-generic (xxxx)    08/18/2020  _x86_64_    (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
         26.97    0.00   22.36   22.53    0.00   28.14

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
loop0             0.00     0.00    0.00    0.00     0.00     0.00     3.20     0.00    2.40    2.40    0.00   0.00   0.00
sda              13.78     9.89   32.24   11.44     0.37     4.10   209.51    47.52 1079.07   44.07 3994.87  22.39  97.81

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
         19.71    0.00   27.85   40.87    0.00   11.57

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
loop0             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    1.40    4.60     0.03     2.71   934.93   142.66 24221.33  666.29 31390.26 166.67 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
         20.16    0.00   26.77   28.30    0.00   24.77

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
loop0             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    8.80    5.60     0.03     3.45   496.11   141.28 12507.78  194.00 31858.00  69.44 100.00


  


  

  9. mpstat -P ALL 5 3

  


  


  Linux 4.15.0-106-generic (sn-1f0ce8)    08/18/2020  _x86_64_    (4 CPU)

02:15:47 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:15:52 PM  all   21.48    0.00   20.40   29.01    0.00    7.94    0.00    0.00    0.00   21.17
02:15:52 PM    0   24.95    0.00   20.86    5.32    0.00    0.61    0.00    0.00    0.00   48.26
02:15:52 PM    1   17.59    0.00   18.81   57.67    0.00    5.93    0.00    0.00    0.00    0.00
02:15:52 PM    2   21.28    0.00   17.36    0.21    0.00   24.79    0.00    0.00    0.00   36.36
02:15:52 PM    3   22.34    0.00   24.59   52.46    0.00    0.61    0.00    0.00    0.00    0.00

02:15:52 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:15:57 PM  all   20.56    0.00   20.00   28.26    0.00    7.08    0.00    0.00    0.00   24.10
02:15:57 PM    0   24.44    0.00   18.89   12.32    0.00    0.21    0.00    0.00    0.00   44.15
02:15:57 PM    1   17.73    0.00   15.46   33.20    0.00    4.95    0.00    0.00    0.00   28.66
02:15:57 PM    2   18.93    0.00   22.22   12.35    0.00   22.84    0.00    0.00    0.00   23.66
02:15:57 PM    3   21.06    0.00   23.31   55.21    0.00    0.41    0.00    0.00    0.00    0.00

02:15:57 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:16:02 PM  all   21.81    0.00   18.32   26.42    0.00    7.03    0.00    0.00    0.00   26.42
02:16:02 PM    0   26.43    0.00   19.67    0.20    0.00    0.41    0.00    0.00    0.00   53.28
02:16:02 PM    1   20.57    0.00   17.11   45.21    0.00    5.30    0.00    0.00    0.00   11.81
02:16:02 PM    2   19.67    0.00   16.74    0.21    0.00   21.97    0.00    0.00    0.00   41.42
02:16:02 PM    3   20.45    0.00   19.84   58.91    0.00    0.81    0.00    0.00    0.00    0.00

Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
Average:     all   21.28    0.00   19.57   27.90    0.00    7.35    0.00    0.00    0.00   23.90
Average:       0   25.27    0.00   19.81    5.94    0.00    0.41    0.00    0.00    0.00   48.57
Average:       1   18.63    0.00   17.13   45.39    0.00    5.39    0.00    0.00    0.00   13.45
Average:       2   19.96    0.00   18.78    4.28    0.00   23.20    0.00    0.00    0.00   33.77
Average:       3   21.28    0.00   22.57   55.54    0.00    0.61    0.00    0.00    0.00    0.00