Recherche avancée

Sur d’autres sites (6286)

• Google’s YouTube Uses FFmpeg

  9 février 2011, par Multimedia Mike — General

  Controversy arose last week when Google accused Microsoft of stealing search engine results for their Bing search engine. It was a pretty novel sting operation and Google did a good job of visually illustrating their side of the story on their official blog.

  This reminds me of the fact that Google’s YouTube video hosting site uses FFmpeg for converting videos. Not that this is in the same league as the search engine shenanigans (it’s perfectly legit to use FFmpeg in this capacity, but to my knowledge, Google/YouTube has never confirmed FFmpeg usage), but I thought I would revisit this item and illustrate it with screenshots. This is not new information— I first empirically tested this fact 4 years ago. However, a lot of people wonder how exactly I can identify FFmpeg on the backend when I claim that I’ve written code that helps power YouTube.

  Short Answer
  How do I know YouTube uses FFmpeg to convert multimedia ? Because :

  1. FFmpeg can decode a number of impossibly obscure multimedia formats using code I wrote
  2. YouTube can transcode many of the same formats
  3. I screwed up when I wrote the code to support some of these weird formats
  4. My mistakes are still present when YouTube transcodes certain fringe formats

  Longer Answer (With Pictures !)
  Let’s take a video format named RoQ, developed by noted game designer Graeme Devine. Originated for use in the FMV-heavy game The 11th Hour, the format eventually found its way into the Quake 3 engine as well as many games derived from the same technology.

  Dr. Tim Ferguson reverse engineered the format (though it would later be open sourced along with the rest of the Q3 engine). I wrote a RoQ playback system for FFmpeg, and I messed up in doing so. I believe my coding error helps demonstrate the case I’m trying to make here.

  Observe what happened when I pushed the jk02.roq sample through YouTube in my original experiment 4 years ago :

  
  
  

  Do you see how the canyon walls bleed into the sky ? That’s not supposed to happen. FFmpeg doesn’t do that anymore but I was able to go back into the source code history to find when it did do that :

  
  
  

  Academic Answer
  FFmpeg fixed this bug in June of 2007 (thanks to Eric Lasota). The problem had to do with premature colorspace conversion in my original decoder.

  Leftovers
  I tried uploading the video again to see if the problem persists in YouTube’s transcoder. First bit of trivia : YouTube detects when you have uploaded the same video twice and rejects the subsequent attempts. So I created a double concatenation of the video and uploaded it. The problem is gone, illustrating that the backend is actually using a newer version of FFmpeg. This surprises me for somewhat esoteric reasons.

  Here’s another interesting bit of trivia for those who don’t do a lot of YouTube uploading— YouTube reports format details when you upload a video :

  
  
  

  So, yep, RoQ format. And you can wager that this will prompt me to go back through the litany of unusual formats that FFmpeg supports to see how YouTube responds.

• bash script to install / update ffmpeg static builds

  1er mars 2017, par Joakim

  Hi I’m trying to make my first "real" bash script to do some real work than can be executed both manually and through cron jobs.

  The script should download and install the ffmpeg-static-build from https://johnvansickle.com/ffmpeg

  Here is what I’ve got so far :

  #!/bin/bash
  
  # Still to come, see if the script runs with root privileges else exit
  
  #Download FFMPEG static daily build and it's md5
  
  cd ~
  
  wget https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-64bit-static.tar.xz | tar -xf
  
  #wget https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-64bit-static.tar.xz.md5
  
  curl -L https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-64bit-static.tar.xz.md5 | tar -xf | tee -a https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-64bit-static.tar.xz | cut -d\  -f1 | md5sum > md5sum.txt
  
  
  
  
  
  #Chech the md5 is currect
  
  #md5sum -c MD5SUMS
  
  file1="md5sum.txt"
  
  file2="ffmpeg-git-64bit-static.tar.xz.md5"
  
  
  
  if ! cmp --silent "$file1" "$file2"; then
  
      echo "md5sum doesn't match...\n exit" >&2
  
      exit 1
  
  fi
  
  
  
  
  
  #tar -xf ffmpeg-*.tar.xz
  
  cp ffmpeg-*-static/ff* /usr/bin/
  
  cp ffmpeg-*-static/ff* /usr/local/bin/
  
  cp ffmpeg-*-static/qt-faststart /usr/bin/
  
  cp ffmpeg-*-static/qt-faststart /usr/local/bin/
  
  # Consider change second location to use ln -s
  
  # Remove downloads and do some clean up
  
  
  
  rm -fr ffmpeg-*
  
  
  
  #EOF

  As you can see in the script i would like to add the md5sum check but it fails (got the md5sum check part from Compare md5 sums in bash script)

  If i remove the md5sum part the script is working, but right now the script fails at the | tar -xf | part with this code

  2017-02-28 00:10:24 (617 KB/s) - ‘ffmpeg-git-64bit-static.tar.xz’ saved [17564756/17564756]
  
  
  
  tar: option requires an argument -- 'f'
  
  Try 'tar --help' or 'tar --usage' for more information.
  
  tee: 'https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-64bit-static.tar.xz': No such file or directory
  
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
  
                                   Dload  Upload   Total   Spent    Left  Speed
  
  100    65  100    65    0     0     87      0 --:--:-- --:--:-- --:--:--    87
  
  (23) Failed writing body
  
  md5sum doesn't match...
  
   exit

  As this is my first, I would appreciate any "I’m 4 years old" advices and why

  update

  Have made some changes to the script based on the suggestions in this thread, but my problem is I have no output what so ever at this state :( So I think it’s time to ask for the next clue

  #!/bin/bash
  
  
  
  # version 0.3z
  
  
  
  ## Download FFMPEG static daily build and it's md5
  
  ## 
  
  ## To make this script working you might need to change the below values
  
  ## based on whether you are using a 32bit or 64 bit OS
  
  ## to obtain the right links you have to have a look @ https://johnvansickle.com/ffmpeg/
  
  ## and then change those below
  
  ## 
  
  ## If you are running on a shared server or dowsn't have root priviliges you might need to uncomment
  
  ## point 5.
  
  
  
  
  
  # a "~" means running users home folder :) and should be different from destination dir
  
  download_dir=~
  
  
  
  # as this can change if the ffmpeg is to be stored on ex. shared server
  
  dest_dir=/usr/bin/
  
  
  
  # The version it equal the filename from above link
  
  version=ffmpeg-git-64bit-static.tar.xz
  
  
  
  ## Do not change anything below here!!
  
  source_url=https://johnvansickle.com/ffmpeg/builds/${version}
  
  md5_url=https://johnvansickle.com/ffmpeg/builds/${version}.md5
  
  
  
  # Still to come, see if the script runs with write privileges else exit
  
  
  
  # 1. Can we enter download directory else exit
  
  cd ${download_dir} ||
  
  
  
          printf "%s\n" "You can't enter this folder.\nPlease chage download_dir in this script..." >&2
  
          exit 1
  
  
  
  # 2. Check the md5 is correct or have changed from last check
  
  ## instead of downloading ffmpeg-git-64bit-static.tar.xz every time,
  
  ## regardless if it is new or not, I recommend only downloading it
  
  ## if the md5 does not match. Would save John some bandwidth at least
  
  ## thx for the idea to @LordNeckbeard
  
  
  
  
  
  
  
  ## So somehow i'll guess some sed or grep only first part is nessesary :(
  
  ## This is getting more advance than expected for a first time script :/
  
  
  
  if ! diff <(md5sum ${version}) <(curl -s ${md5_url})
  
  
  
      then
  
          printf "%s\n" "md5sum doesn't match...\n
  
                          Downloading new version" >&2
  
          rm -f ${version} >&2
  
          curl ${source_url} -o ${download_dir}/${version} >&2
  
          #exit 2
  
  
  
      elif
  
          diff <(md5sum ${version}) <(curl -s ${md5_url})
  
          printf "%s\n" "Nothing new to download" >&2
  
        exit 3
  
  fi
  
  
  
  # 3. untar
  
  tar -xf ffmpeg-git-*-static.tar.xz
  
  
  
  # 4. Move builds to destination directories
  
  
  
  mv ${download_dir}/ffmpeg-*-static/ff* ${dest_dir}/
  
  mv ${download_dir}/ffmpeg-*-static/qt-faststart ${dest_dir}/
  
  
  
  # 5. Make soft links to static builds
  
  ln -sfn ${dest_dir}/qt-faststart /usr/local/bin/qt-faststart
  
  ln -sfn ${dest_dir}/ffmpeg /usr/local/bin/ffmpeg
  
  ln -sfn ${dest_dir}/ffmpeg-10bit /usr/local/bin/ffmpeg-10bit
  
  ln -sfn ${dest_dir}/ffprobe /usr/local/bin/ffprobe
  
  ln -sfn ${dest_dir}/ffserver /usr/local/bin/ffserver
  
  
  
  # Remove unzipped folder to do some clean up
  
  
  
  rm -fr ffmpeg-git-*-static/
  
  
  
  #EOF

  note : do to some more in depth answering of why not compile from source :
  1. This precompiled is usable on all Linux variations, despite distro and version
  2. It usable on shared hosting servers with ssh access

  UPDATE 2

      #!/bin/bash
  
  
  
  # version 0.4z
  
  
  
  ## Download FFMPEG static daily build and it's md5
  
  ## 
  
  ## To make this script working you might need to change the below values
  
  ## based on whether you are using a 32bit or 64 bit OS
  
  ## to obtain the right links you have to have a look @ https://johnvansickle.com/ffmpeg/
  
  ## and then change those below
  
  ## 
  
  ## Finished and working script should be distributed under GPLv3: free as in freedom
  
  ## 
  
  ## If you are running on a shared server or dowsn't have root priviliges you might need to uncomment
  
  ## point 5.
  
  
  
  
  
  # a "~" means running users home folder :) and should be different from destination dir
  
  download_dir=~
  
  
  
  # as this can change if the ffmpeg is to be stored on ex. shared server
  
  dest_dir=/usr/bin/
  
  
  
  # The version it equal the filename from above link
  
  version=ffmpeg-git-64bit-static.tar.xz
  
  
  
  ## Do not change anything below here!!
  
  source_url=https://johnvansickle.com/ffmpeg/builds/${version}
  
  md5="curl -s https://johnvansickle.com/ffmpeg/builds/${version}.md5"
  
  
  
  # Still to come, see if the script runs with write privileges else exit
  
  
  
  # 1. Can we enter download directory else exit
  
  cd ${download_dir} || 
  
          printf "%s\n" "You can't enter this folder.\nPlease chage download_dir in this script..." >&2
  
          exit 1
  
  
  
  # 2. Check the md5 is correct or have changed from last check
  
  ## instead of downloading ffmpeg-git-64bit-static.tar.xz every time,
  
  ## regardless if it is new or not, I recommend only downloading it
  
  ## if the md5 does not match. Would save John some bandwidth at least
  
  ## thx for the idea to @LordNeckbeard
  
  
  
  ## This is getting more advance than expected for a first time script :/
  
  
  
  if diff <(md5sum ${version}) <(${md5})
  
  
  
      then
  
          printf "%s\n" "No new version availeble" >&2
  
          exit 1
  
  
  
  elif ! diff <(md5sum ${version}) <(${md5})
  
      then
  
          rm -f ${version}
  
          curl ${source_url} > ${version}
  
          exit 0
  
  
  
          #only proceed if downloaded version match it's md5
  
          if ! diff <(md5sum ${version}) <(${md5})
  
              then
  
              rm -f ${version}
  
              printf "%s\n" "Downloaded version is damaged, try later\ndamaged version have been deleted" >&2
  
              exit 1
  
          fi
  
  
  
              # 3. untar
  
              tar -xf ffmpeg-git-*-static.tar.xz
  
  
  
              # 4. Move builds to destination directories
  
              mv ${download_dir}/ffmpeg-*-static/ff* ${dest_dir}/
  
              mv ${download_dir}/ffmpeg-*-static/qt-faststart ${dest_dir}/
  
  
  
              # 5. Make soft links to static builds
  
              ln -sfn ${dest_dir}/qt-faststart /usr/local/bin/qt-faststart
  
              ln -sfn ${dest_dir}/ffmpeg /usr/local/bin/ffmpeg
  
              ln -sfn ${dest_dir}/ffmpeg-10bit /usr/local/bin/ffmpeg-10bit
  
              ln -sfn ${dest_dir}/ffprobe /usr/local/bin/ffprobe
  
              ln -sfn ${dest_dir}/ffserver /usr/local/bin/ffserver
  
  
  
              # Remove unzipped folder to do some clean up
  
              rm -fr ffmpeg-git-*-static/
  
              printf "%s\n" "Going to install new version" >&2
  
              exit 1
  
  fi
  
  #EOF

  But still having some issues :(

  1. Running this script returns : a blanc shell, but I’ve expected one of the printf statements
  2. When I’m trying to narrow down the problem and going back to basic and trying to run the script with only the "if" part it fails with 44 : Syntax error : "(" unexpected

  3. running the very same "if" part typed directly into the shell/terminal itself it’s all happy !!
     if diff <(md5sum ffmpeg-git-64bit-static.tar.xz) <(curl -s "https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-64bit-static.tar.xz.md5"); then printf "%s\n" "No new version availeble" >&2; elif ! diff <(md5sum ffmpeg-git-64bit-static.tar.xz) <(curl -s "https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-64bit-static.tar.xz.md5"); then rm -f ffmpeg-git-64bit-static.tar.xz; curl https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-64bit-static.tar.xz > ffmpeg-git-64bit-static.tar.xz; printf "%s\n" "Going to install new version" >&2; fi

  4. I’m confused

• Further Dreamcast Hacking

  3 février 2011, par Multimedia Mike — Sega Dreamcast

  I’m still haunted by Sega Dreamcast programming, specifically the fact that I used to be able to execute custom programs on the thing (roughly 8-10 years ago) and now I cannot. I’m going to compose a post to describe my current adventures on this front. There are 3 approaches I have been using : Raw, Kallistios, and the almighty Linux.

  
  

  Raw
  What I refer to as "raw" is an assortment of programs that lived in a small number of source files (sometimes just one ASM file) and could be compiled with the most basic SH-4 toolchain. The advantage here is that there aren’t many moving parts and not many things that can possibly go wrong, so it provides a good functional baseline.

  One of the original Dreamcast hackers was Marcus Comstedt, who still has his original DC material hosted at the reasonably easy-to-remember URL mc.pp.se/dc. I can get some of these simple demos to work, but not others.

  I also successfully assembled and ran a pair of 256-byte (!!) demos from this old DC scene page.

  KallistiOS
  KallistiOS (or just KOS) was a real-time OS developed for the DC and was popular among the DC homebrew community. All the programming I did back in the day was based around KOS. Now I can’t get any of it to work. More specifically, KOS can’t seem to make it past a certain point in its system initialization.

  The Linux Option
  I was never that excited about running Linux on my Dreamcast. For some hackers, running Linux on a given piece of consumer electronics is the highest attainable goal. Back in the day, I looked at it from a much more pragmatic perspective— I didn’t see much use in running Linux on the DC, not as much as running KOS which was developed to be a much more appropriate fit.

  However, I was able to burn a CD-R of an old binary image of Linux 2.4.5 compiled for the Dreamcast and boot it some months ago. So I at least have a feeling that this should work. I have never cross-compiled a kernel of my own (though I have compiled many, many x86 kernels in my time, so I’m not a total n00b in this regard). I figured this might be a good time to start.

  The first item that worries me is getting a functional cross-compiling toolchain. Fortunately, a little digging in the Linux kernel documentation pointed me in the direction of a bunch of ready-made toolchains hosted at kernel.org. So I grabbed one of the SH toolchains (gcc-4.3.3-nolibc) and got rolling.

  I’m well familiar with the cycle of 'make menuconfig' in order to pick configuration options, and then 'make' to build a kernel (or usually 'make zImage' or 'make bzImage' to create compressed images). For cross compiling, the primary difference seems to be editing the root Makefile in the Linux source code tree (I’m using 2.6.37, the latest stable as of this writing) and setting a value for the CROSS_COMPILE variable. Then, run 'make menuconfig' followed by 'make' as normal.

  The Linux 2.6 series is supposed to support a range of Renesas (formerly Hitachi) SH processors and board configurations. This includes reasonable defaults for the Sega Dreamcast hardware. I got it all compiling except for a series of .S files. Linus Torvalds once helped me debug a program I work on so I thought I’d see if there was something I could help debug here.

  The first issue was with ASM statements of a form similar to :

  mov #0xffffffe0, r1
  

  Now, the DC’s SH-4 is a RISC CPU. A lot of RISC architectures adopt a fixed instruction size of 32 bits. You can’t encode an entire 32-bit immediate value inside of a 32-bit instruction (there would be no room for the instruction encoding). Further, the SH series encoded instructions with a mere 16 bits. The move immediate data instruction only allows for an 8-bit, sign-extended value.

  I decided that the above statement is equivalent to :

  mov #-32, r1
  

  I’ll give this statement the benefit of the doubt that it used to work with the gcc toolchain somewhere along the line. I assume that the assembler is supposed to know enough to substitute the first form with the second.

  The next problem is that an ’sti’ instruction shows up in a number of spots. Using Intel x86 conventions, this is a "set interrupt flag" instruction (I remember that the 6502 CPU had the same instruction mnemonic, though its interrupt flag’s operation was opposite that of the x86). The SH-4 reference manual lists no ’sti’ instruction. When it gets to these lines, the assembler complains about immediate move instructions with too large data, like the instructions above. I’m guessing they must be macro’d to something else but I failed to find where. I commented out those lines for the time being. Probably not that smart, but I want to keep this moving for now.

  So I got the code to compile into a kernel file called ’vmlinux’. I’ve seen this file many times before but never thought about how to get it to run directly. The process has usually been to compress it and send it over to lilo or grub for loading, as that is the job of the bootloader. I have never even wondered what format the vmlinux file takes until now. It seems that ’vmlinux’ is just a plain old ELF file :

  $ file vmlinux
  vmlinux : ELF 32-bit LSB executable, Renesas SH,
  version 1 (SYSV), statically linked, not stripped
  

  The ’dc-tool’ program that uploads executables to the waiting bootloader on the Dreamcast is perfectly cool accepting ELF files (and S-record files, and raw binary files). After a very lengthy upload process, execution fails (resets the system).

  For the sake of comparison, I dusted off that Linux 2.4.5 bootable Dreamcast CD-ROM and directly uploaded the vmlinux file from that disc. That works just fine (until it’s time to go to the next loading phase, i.e., finding a filesystem). Possible issues here could include the commented ’sti’ instructions (could be that they aren’t just decoration). I’m also trying to understand the memory organization— perhaps the bootloader wants the ELF to be based at a different address. Or maybe the kernel and the bootloader don’t like each other in the first place— in this case, I need to study the bootable Linux CD-ROM to see how it’s done.

  Optimism
  Even though I’m meeting with rather marginal success, this is tremendously educational. I greatly enjoy these exercises if only for the deeper understanding they bring for the lowest-level system details.