Recherche avancée

Sur d’autres sites (11819)

• ffmpeg "Manifest too large" when downloading youtube video

  22 avril 2024, par ynn

  TL ;DR

  


  I'm trying to programmatically download a part of a YouTube video. The widely known procedure doesn't work for some videos and I'd like to overcome this situation.

  


  ---

  


  Context

  


  I'm trying to programmatically download a part of a YouTube video. As described in How to download portion of video with youtube-dl command ?, you can achieve this by the following commands.

  


  #Converts a human-readable URL to longer URLs for internal use.
~ $ youtube-dl --get-url 'https://www.youtube.com/watch?v=POrfo478HRI'
https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=299&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C298%2C299%2C302%2C303&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=video%2Fmp4&gir=yes&clen=9205054989&dur=17931.000&lmt=1597347724488304&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7316222&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgYDqTm0CnLfG0kxsYNrAdtAUYB7alnowiaBArU8R5bBwCIQDeCRDJHFO_PWcnbeFaJvip80deuboN4Pi1x3eRhJBxlQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes
https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=251&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=audio%2Fwebm&gir=yes&clen=268945126&dur=17931.001&lmt=1597348111385641&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7311222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAK8XGu3vHucHwqERZ7TQniuvKEd-NTbMkwZbu8EXJ5E3AiEAiS7OYFOcsIrD2xp-AJNzucj1H9ZKMlmkCl_sU7__dZo%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes

#Stores the result to a variable.
~ $ url_video_part='https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=299&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C298%2C299%2C302%2C303&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=video%2Fmp4&gir=yes&clen=9205054989&dur=17931.000&lmt=1597347724488304&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7316222&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgYDqTm0CnLfG0kxsYNrAdtAUYB7alnowiaBArU8R5bBwCIQDeCRDJHFO_PWcnbeFaJvip80deuboN4Pi1x3eRhJBxlQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes'

#Stores the result to a variable.
~ $ url_audio_part='https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=251&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=audio%2Fwebm&gir=yes&clen=268945126&dur=17931.001&lmt=1597348111385641&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7311222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAK8XGu3vHucHwqERZ7TQniuvKEd-NTbMkwZbu8EXJ5E3AiEAiS7OYFOcsIrD2xp-AJNzucj1H9ZKMlmkCl_sU7__dZo%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes'

#Downloads the video for one minute from 12:45.
~ $ ffmpeg    -ss 12:45 -i "${url_video_part}"    -ss 12:45 -i "${url_audio_part}"    -t 1:00 output.mkv
frame= 3600 fps= 49 q=-1.0 Lsize=   27047kB time=00:00:59.99 bitrate=3692.9kbits/s speed=0.813x


  


  This does work perfectly.

  


  ---

  


  Problem

  


  For some videos^†1, the internal URLs are of the form https://manifest.googlevideo.com/... instead of https://RANDOM-STRING.googlevideo.com/... as in the example above.

  


  ~ $ youtube-dl --get-url 'https://www.youtube.com/watch?v=KRMfnvLODZQ'
https://manifest.googlevideo.com/api/manifest/dash/expire/1597593866/ei/qgQ5X5maN8-ilQSu15W4CQ/ip/10.100.238.99/id/KRMfnvLODZQ.1/source/yt_live_broadcast/requiressl/yes/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhANYKQvKTN8BRmhp7--4tQp9aqRp_qIGjh472BTYLyJ62AiAautFF86xGDHOe7mYlYaZ1W86EM4k2e6UiJS9Q91dMog%3D%3D
https://manifest.googlevideo.com/api/manifest/dash/expire/1597593866/ei/qgQ5X5maN8-ilQSu15W4CQ/ip/10.100.238.99/id/KRMfnvLODZQ.1/source/yt_live_broadcast/requiressl/yes/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhANYKQvKTN8BRmhp7--4tQp9aqRp_qIGjh472BTYLyJ62AiAautFF86xGDHOe7mYlYaZ1W86EM4k2e6UiJS9Q91dMog%3D%3D


  


  And downloading such videos always fails.

  


  ~ $ ffmpeg    -ss 12:45 -i "${url_video_part}"    -ss 12:45 -i "${url_audio_part}"    -t 1:00 output.mkv
[dash @ 0x55fa6a1a4400] Manifest too large: 3776722
https://manifest.googlevideo.com/...: Invalid data found when processing input


  


  Please note downloading full of such videos using youtube-dl alone succeeds.

  


  _{†1 : As far as we tested, recently (e.g. yesterday) uploaded videos have such URLs and when we retry the same youtube-dl --get-url command after a day or so, we get "innocent" URLs like in the first example.}

  


  ---

  


  Questions

  


  How can I download parts of such videos

  


  


  • by specifying additional options for ffmpeg,

    


  


  • by somehow forcefully retrieving internal URLs of the form https://RANDOM-STRING.googlevideo.com/...,

    


  


  • or by something which I have no idea at all ?

    


  


  


  My current workaround is just to wait for a day or two until the internal URLs change, which is much slower than downloading the full video and then cropping it.

  


  ---

  


  Environments

  


  ~ $ lsb_release -d
Description:    Arch Linux

~ $ ffmpeg -version
ffmpeg version n4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 10.1.0 (GCC)
configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100

~ $ youtube-dl --version
2020.07.28


  


• Combining three or more videos with ffmpeg and the xfade filter

  25 février 2021, par sil

  As of 2020 ffmpeg has the xfade filter which can combine videos with a transition. Combining two videos is easy enough :

  


  ffmpeg -i vid1.mp4 -i vid2.mp4 \
  -filter_complex [0][1]xfade=transition=pixelize:duration=1:offset=4,format=yuv420p" \
  out.mp4


  


  But I don't understand how to combine three videos (so that v1 fades into v2 and v2 then fades into v3. I tried something like this :

  


  ffmpeg -i vid1.mp4 -i vid2.mp4 -i vid3.mp4 \
  -filter_complex [0][1]xfade=transition=pixelize:duration=1:offset=4,format=yuv420p[0n1];[0n1][2]xfade=transition=pixelize:duration=1:offset=9,format=yuv420p" \
  out.mp4


  


  but that doesn't work. My idea was that 0 and 1, or vid1 and vid2, would be combined into a [0n1] stream with a transition by xfade, and then that 0n1 stream could be combined with vid3 with another filter. As far as I can tell, this includes the first two videos but not the third. What this of course means is that I don't understand how to specify a filtergraph correctly !

  


  How should I use xfade to combine 3 or more videos with transitions between them ?

  


  A full example is as follows. Here I'll use three images (because then issues with combining videos at different frame rates are avoided), and smash them all to 500x500 for ease (in the final version they would be letterboxed to keep resolution and so on).

  


  ffmpeg \
  -loop 1 -t 5 -i tests/p1.jpg \
  -loop 1 -t 5 -i tests/p2.jpg \
  -loop 1 -t 5 -i tests/p3.jpg \
  -filter_complex "[0]scale=500:500[s0];[1]scale=500:500[s1];[2]scale=500:500[s2];[s0][s1]xfade=transition=pixelize:duration=1:offset=4,format=yuv420p[s01];[s01][s2]xfade=transition=pixelize:duration=1:offset=9,format=yuv420p" out.mp4


  


  I would expect this to create a video which was :

  


  

  • 4 seconds of p1.jpg

  


  • a pixelise transition into p2.jpg lasting 1 second

  


  • 4 seconds of p2.jpg

  


  • a pixelise transition into p3.jpg lasting 1 second

  


  • 4 seconds of p3.jpg

  


  


  but what I actually get is

  


  

  • 4 seconds of p1.jpg

  


  • a pixelise transition into p2.jpg lasting 1 second

  


  • 4 seconds of p2.jpg

  


  


  and then the video ends. p3 is not included at all.

  


  The output is as follows :

  


  ffmpeg version N-53546-g5eb4405fc5-static https://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (Debian 8.3.0-6)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
  libavutil      56. 56.100 / 56. 56.100
  libavcodec     58. 97.100 / 58. 97.100
  libavformat    58. 49.100 / 58. 49.100
  libavdevice    58. 11.101 / 58. 11.101
  libavfilter     7. 87.100 /  7. 87.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, image2, from 'tests/p1.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 44845 kb/s
    Stream #0:0: Video: mjpeg (Baseline), yuvj444p(pc, bt470bg/unknown/unknown), 820x1270 [SAR 150:150 DAR 82:127], 25 fps, 25 tbr, 25 tbn, 25 tbc
Input #1, image2, from 'tests/p2.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 22325 kb/s
    Stream #1:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 960x600 [SAR 1:1 DAR 8:5], 25 fps, 25 tbr, 25 tbn, 25 tbc
Input #2, image2, from 'tests/p3.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 15266 kb/s
    Stream #2:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 728x669 [SAR 96:96 DAR 728:669], 25 fps, 25 tbr, 25 tbn, 25 tbc
File 'out.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 (mjpeg) -> scale
  Stream #1:0 (mjpeg) -> scale
  Stream #2:0 (mjpeg) -> scale
  format -> Stream #0:0 (libx264)
Press [q] to stop, [?] for help
[swscaler @ 0x8228040] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x8258640] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x827df40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x829f800] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x82c13c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x82e8340] deprecated pixel format used, make sure you did set range correctly
[libx264 @ 0x77b7600] using SAR=82/127
[libx264 @ 0x77b7600] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x77b7600] profile High, level 3.0, 4:2:0, 8-bit
[libx264 @ 0x77b7600] 264 - core 161 r3018 db0d417 - H.264/MPEG-4 AVC codec - Copyleft 2003-2020 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'out.mp4':
  Metadata:
    encoder         : Lavf58.49.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 500x500 [SAR 82:127 DAR 82:127], q=-1--1, 25 fps, 12800 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc58.97.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame=  101 fps=100 q=28.0 size=       0kB time=00:00:01.92 bitrate=   0.2kbits/s speed=[swscaler @ 0x8291d80] deprecated pixel format used, make sure you did set range correctly
    Last message repeated 2 times
[swscaler @ 0x82b3000] deprecated pixel format used, make sure you did set range correctly
    Last message repeated 2 times
[swscaler @ 0x82fc200] deprecated pixel format used, make sure you did set range correctly
    Last message repeated 2 times
frame=  153 fps=101 q=28.0 size=       0kB time=00:00:04.00 bitrate=   0.1kbits/s speed=[swscaler @ 0x82fc200] deprecated pixel format used, make sure you did set range correctly
    Last message repeated 2 times
frame=  225 fps=111 q=28.0 size=     256kB time=00:00:06.88 bitrate= 304.9kbits/s dup=0 frame=  225 fps= 95 q=-1.0 Lsize=     267kB time=00:00:08.88 bitrate= 245.9kbits/s dup=0 drop=125 speed=3.74x    
video:263kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.293847%
[libx264 @ 0x77b7600] frame I:2     Avg QP:21.03  size: 19928
[libx264 @ 0x77b7600] frame P:66    Avg QP:21.18  size:  2453
[libx264 @ 0x77b7600] frame B:157   Avg QP:29.72  size:   427
[libx264 @ 0x77b7600] consecutive B-frames:  5.3%  4.4%  1.3% 88.9%
[libx264 @ 0x77b7600] mb I  I16..4: 29.0% 27.8% 43.2%
[libx264 @ 0x77b7600] mb P  I16..4: 10.8%  3.2%  7.5%  P16..4:  2.6%  0.9%  0.3%  0.0%  0.0%    skip:74.8%
[libx264 @ 0x77b7600] mb B  I16..4:  1.1%  0.5%  1.6%  B16..8:  1.0%  0.5%  0.1%  direct: 0.3%  skip:94.9%  L0:37.4% L1:43.5% BI:19.1%
[libx264 @ 0x77b7600] 8x8 transform intra:16.5% inter:55.3%
[libx264 @ 0x77b7600] coded y,uvDC,uvAC intra: 31.4% 71.7% 31.8% inter: 1.1% 1.4% 0.1%
[libx264 @ 0x77b7600] i16 v,h,dc,p: 45% 46%  9%  1%
[libx264 @ 0x77b7600] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 30% 27%  6%  1%  1%  2%  1%  7%
[libx264 @ 0x77b7600] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 56% 31%  9%  1%  1%  1%  1%  1%  1%
[libx264 @ 0x77b7600] i8c dc,h,v,p: 46% 27% 23%  4%
[libx264 @ 0x77b7600] Weighted P-Frames: Y:10.6% UV:7.6%
[libx264 @ 0x77b7600] ref P L0: 61.9% 17.1% 13.9%  6.8%  0.3%
[libx264 @ 0x77b7600] ref B L0: 76.3% 22.8%  0.9%
[libx264 @ 0x77b7600] ref B L1: 97.6%  2.4%
[libx264 @ 0x77b7600] kb/s:238.88


  


• Detect if "faststart" has been applied to video

  26 juin 2024, par Will Ashworth

  When running "qt-faststart" on an MP4, you're essentially taking the meta data from the back and placing it at the front of the file. In my case, so Flash can properly start playing the video before it's fully done loading.

  



  I have a large number of videos that I'm running through a shell script to encode overnight. When I upload, however, I won't necessarily know in my final folder of videos (on the server) have "qt-faststart" run on it.

  



  My goal is to find the straggler videos and run qt-faststart on them manually, but I'll need a way to compile a list. Is there any way to check for this meta data information with PHP or something ? Wondering how I can tell if a video has already had qt-faststart run on it or not.