Recherche avancée

Sur d’autres sites (3532)

• ffmpeg "Manifest too large" when downloading youtube video

  22 avril 2024, par ynn

  TL ;DR

  


  I'm trying to programmatically download a part of a YouTube video. The widely known procedure doesn't work for some videos and I'd like to overcome this situation.

  


  ---

  


  Context

  


  I'm trying to programmatically download a part of a YouTube video. As described in How to download portion of video with youtube-dl command ?, you can achieve this by the following commands.

  


  #Converts a human-readable URL to longer URLs for internal use.
~ $ youtube-dl --get-url 'https://www.youtube.com/watch?v=POrfo478HRI'
https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=299&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C298%2C299%2C302%2C303&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=video%2Fmp4&gir=yes&clen=9205054989&dur=17931.000&lmt=1597347724488304&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7316222&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgYDqTm0CnLfG0kxsYNrAdtAUYB7alnowiaBArU8R5bBwCIQDeCRDJHFO_PWcnbeFaJvip80deuboN4Pi1x3eRhJBxlQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes
https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=251&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=audio%2Fwebm&gir=yes&clen=268945126&dur=17931.001&lmt=1597348111385641&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7311222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAK8XGu3vHucHwqERZ7TQniuvKEd-NTbMkwZbu8EXJ5E3AiEAiS7OYFOcsIrD2xp-AJNzucj1H9ZKMlmkCl_sU7__dZo%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes

#Stores the result to a variable.
~ $ url_video_part='https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=299&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C298%2C299%2C302%2C303&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=video%2Fmp4&gir=yes&clen=9205054989&dur=17931.000&lmt=1597347724488304&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7316222&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgYDqTm0CnLfG0kxsYNrAdtAUYB7alnowiaBArU8R5bBwCIQDeCRDJHFO_PWcnbeFaJvip80deuboN4Pi1x3eRhJBxlQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes'

#Stores the result to a variable.
~ $ url_audio_part='https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=251&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=audio%2Fwebm&gir=yes&clen=268945126&dur=17931.001&lmt=1597348111385641&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7311222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAK8XGu3vHucHwqERZ7TQniuvKEd-NTbMkwZbu8EXJ5E3AiEAiS7OYFOcsIrD2xp-AJNzucj1H9ZKMlmkCl_sU7__dZo%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes'

#Downloads the video for one minute from 12:45.
~ $ ffmpeg    -ss 12:45 -i "${url_video_part}"    -ss 12:45 -i "${url_audio_part}"    -t 1:00 output.mkv
frame= 3600 fps= 49 q=-1.0 Lsize=   27047kB time=00:00:59.99 bitrate=3692.9kbits/s speed=0.813x


  


  This does work perfectly.

  


  ---

  


  Problem

  


  For some videos^†1, the internal URLs are of the form https://manifest.googlevideo.com/... instead of https://RANDOM-STRING.googlevideo.com/... as in the example above.

  


  ~ $ youtube-dl --get-url 'https://www.youtube.com/watch?v=KRMfnvLODZQ'
https://manifest.googlevideo.com/api/manifest/dash/expire/1597593866/ei/qgQ5X5maN8-ilQSu15W4CQ/ip/10.100.238.99/id/KRMfnvLODZQ.1/source/yt_live_broadcast/requiressl/yes/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhANYKQvKTN8BRmhp7--4tQp9aqRp_qIGjh472BTYLyJ62AiAautFF86xGDHOe7mYlYaZ1W86EM4k2e6UiJS9Q91dMog%3D%3D
https://manifest.googlevideo.com/api/manifest/dash/expire/1597593866/ei/qgQ5X5maN8-ilQSu15W4CQ/ip/10.100.238.99/id/KRMfnvLODZQ.1/source/yt_live_broadcast/requiressl/yes/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhANYKQvKTN8BRmhp7--4tQp9aqRp_qIGjh472BTYLyJ62AiAautFF86xGDHOe7mYlYaZ1W86EM4k2e6UiJS9Q91dMog%3D%3D


  


  And downloading such videos always fails.

  


  ~ $ ffmpeg    -ss 12:45 -i "${url_video_part}"    -ss 12:45 -i "${url_audio_part}"    -t 1:00 output.mkv
[dash @ 0x55fa6a1a4400] Manifest too large: 3776722
https://manifest.googlevideo.com/...: Invalid data found when processing input


  


  Please note downloading full of such videos using youtube-dl alone succeeds.

  


  _{†1 : As far as we tested, recently (e.g. yesterday) uploaded videos have such URLs and when we retry the same youtube-dl --get-url command after a day or so, we get "innocent" URLs like in the first example.}

  


  ---

  


  Questions

  


  How can I download parts of such videos

  


  


  • by specifying additional options for ffmpeg,

    


  


  • by somehow forcefully retrieving internal URLs of the form https://RANDOM-STRING.googlevideo.com/...,

    


  


  • or by something which I have no idea at all ?

    


  


  


  My current workaround is just to wait for a day or two until the internal URLs change, which is much slower than downloading the full video and then cropping it.

  


  ---

  


  Environments

  


  ~ $ lsb_release -d
Description:    Arch Linux

~ $ ffmpeg -version
ffmpeg version n4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 10.1.0 (GCC)
configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100

~ $ youtube-dl --version
2020.07.28


  


• How to make a L band effect with ffmpeg so that when user start command the video will squeeze in and L band will come from bottom left [closed]

  3 août 2020, par kiara

  Please find this example I want to achieve something like this https://www.youtube.com/watch?v=SsEO_CgSrg0

  


• FFServer streaming H.264 from Logitech C920

  23 octobre 2014, par Ricardo Frederico Leote Mota

  I’m having some trouble trying to implement a web live stream from my Logitech C920 webcam w/ H.264 encoding using ffmpeg/ffserver.

  Here is the configuration file I’m using for the ffserver :

  # Port on which the server is listening. You must select a different
  
  # port from your standard HTTP web server if it is running on the same
  
  # computer.
  
  Port 8090
  
  
  
  # Address on which the server is bound. Only useful if you have
  
  # several network interfaces.
  
  BindAddress 0.0.0.0
  
  
  
  # Number of simultaneous HTTP connections that can be handled. It has
  
  # to be defined *before* the MaxClients parameter, since it defines the
  
  # MaxClients maximum limit.
  
  MaxHTTPConnections 2000
  
  
  
  # Number of simultaneous requests that can be handled. Since FFServer
  
  # is very fast, it is more likely that you will want to leave this high
  
  # and use MaxBandwidth, below.
  
  MaxClients 1000
  
  
  
  # This the maximum amount of kbit/sec that you are prepared to
  
  # consume when streaming to clients.
  
  MaxBandwidth 1000
  
  
  
  # Access log file (uses standard Apache log file format)
  
  # '-' is the standard output.
  
  CustomLog -
  
  
  
  ##################################################################
  
  # Definition of the live feeds. Each live feed contains one video
  
  # and/or audio sequence coming from an ffmpeg encoder or another
  
  # ffserver. This sequence may be encoded simultaneously with several
  
  # codecs at several resolutions.
  
  
  
  <feed>
  
  
  
  # You must use 'ffmpeg' to send a live feed to ffserver. In this
  
  # example, you can type:
  
  #
  
  
  
  File /tmp/feed.ffm
  
  FileMaxSize 100M
  
  
  
  # You could specify
  
  # ReadOnlyFile /saved/specialvideo.ffm
  
  # This marks the file as readonly and it will not be deleted or updated.
  
  
  
  # Only allow connections from localhost to the feed.
  
  ACL allow 127.0.0.1
  
  
  
  </feed>
  
  
  
  
  
  ##################################################################
  
  # Now you can define each stream which will be generated from the
  
  # original audio and video stream. Each format has a filename (here
  
  # 'test1.mpg'). FFServer will send this stream when answering a
  
  # request containing this filename.
  
  
  
  <stream>
  
  
  
  Feed feed.ffm
  
  Format mp4
  
  
  
  # Audio settings
  
  NoAudio
  
  
  
  # Video Settings
  
  #VideoSize 1280x720                      # Video resolution
  
  #VideoFrameRate 30                      # Video FPS
  
  #AVOptionVideo flags +global_header          # Parameters passed to encoder
  
  #AVOptionVideo cpu-used 0
  
  #AVOptionVideo qmin 10
  
  #AVOptionVideo qmax 42
  
  #AVOptionVideo quality good
  
  PreRoll 0
  
  StartSendOnKey
  
  #VideoBitRate 400                      # Video bitrate
  
  
  
  </stream>
  
  
  
  # Server status
  
  
  
  <stream>
  
  Format status
  
  
  
  # Only allow local people to get the status
  
  ACL allow localhost
  
  ACL allow 192.168.0.0 192.168.255.255
  
  
  
  </stream>
  
  
  
  
  
  # Redirect index.html to the appropriate site

  And here is the command I’m issuing through the command line and the respective output :

  ffmpeg -s 1280x720 -f video4linux2 -r 30 -vcodec h264 -re -i /dev/video0 -copyinkf -codec copy h**p://localhost:8090/feed.ffm
  
  ffmpeg version N-67036-g04a4fb8 Copyright (c) 2000-2014 the FFmpeg developers
  
    built on Oct 20 2014 17:39:50 with gcc 4.8 (Ubuntu 4.8.2-19ubuntu1)
  
    configuration: --enable-gpl --enable-libx264
  
    libavutil      54. 10.100 / 54. 10.100
  
    libavcodec     56.  8.102 / 56.  8.102
  
    libavformat    56.  9.101 / 56.  9.101
  
    libavdevice    56.  1.100 / 56.  1.100
  
    libavfilter     5.  2.100 /  5.  2.100
  
    libswscale      3.  1.101 /  3.  1.101
  
    libswresample   1.  1.100 /  1.  1.100
  
    libpostproc    53.  3.100 / 53.  3.100
  
  Input #0, video4linux2,v4l2, from '/dev/video0':
  
    Duration: N/A, start: 12614.859199, bitrate: N/A
  
      Stream #0:0: Video: h264 (Constrained Baseline), yuvj420p(pc), 1280x720 [SAR 1:1 DAR 16:9], -5 kb/s, 30 fps, 30 tbr, 1000k tbn, 60 tbc
  
  Output #0, ffm, to 'h**p://localhost:8090/feed.ffm':
  
    Metadata:
  
      creation_time   : now
  
      encoder         : Lavf56.9.101
  
      Stream #0:0: Video: h264, yuvj420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, -5 kb/s, 30 fps, 1000k tbn, 30 tbc
  
  Stream mapping:
  
    Stream #0:0 -> #0:0 (copy)
  
  Press [q] to stop, [?] for help
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 0, current: -166263; changing to 1. This may result in incorrect timestamps in the output file.
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 1, current: -134365; changing to 2. This may result in incorrect timestamps in the output file.
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 2, current: -102381; changing to 3. This may result in incorrect timestamps in the output file.
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 3, current: -66434; changing to 4. This may result in incorrect timestamps in the output file.
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 4, current: -34548; changing to 5. This may result in incorrect timestamps in the output file.
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 5, current: -2645; changing to 6. This may result in incorrect timestamps in the output file.
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 232857, current: 138514; changing to 232858. This may result in incorrect timestamps in the output file.
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 232858, current: 167942; changing to 232859. This may result in incorrect timestamps in the output file.
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 232859, current: 197415; changing to 232860. This may result in incorrect timestamps in the output file.
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 232860, current: 227037; changing to 232861. This may result in incorrect timestamps in the output file.
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 997959, current: 938599; changing to 997960. This may result in incorrect timestamps in the output file.
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 997960, current: 995052; changing to 997961. This may result in incorrect timestamps in the output file.
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 1240745, current: 1232841; changing to 1240746. This may result in incorrect timestamps in the output file.
  
  [ffm @ 0x28c7ea0] Non-monotonous DTS in output stream 0:0; previous: 1293561, current: 1282157; changing to 1293562. This may result in incorrect timestamps in the output file.
  
  frame=   69 fps= 34 q=-1.0 Lsize=     912kB time=00:00:02.01 bitrate=3702.1kbits/s

  When I try to play the stream with VLC for example - vlc h**p ://localhost:8090/stream.mp4 - i get the following error :

  [00007f5234003948] core stream error : cannot pre fill buffer

  And I can’t watch the stream on the web browser side also.

  Could someone give me an hint on this ?

  Many thanks and your help is much appreciated ! ;)