Recherche avancée

Sur d’autres sites (13246)

• Subtitling Sierra VMD Files

  1er juin 2016, par Multimedia Mike — Game Hacking

  I was contacted by a game translation hobbyist from Spain (henceforth known as The Translator). He had set his sights on Sierra’s 7-CD Phantasmagoria. This mammoth game was driven by a lot of FMV files and animations that have speech. These require language translation in the form of video subtitling. He’s lucky that he found possibly the one person on the whole internet who has just the right combination of skill, time, and interest to pull this off. And why would I care about helping ? I guess I share a certain camaraderie with game hackers. Don’t act so surprised. You know what kind of stuff I like to work on.

  The FMV format used in this game is VMD, which makes an appearance in numerous Sierra titles. FFmpeg already supports decoding this format. FFmpeg also supports subtitling video. So, ideally, all that’s necessary to support this goal is to add a muxer for the VMD format which can encode raw video and audio, which the format supports. Implement video compression as extra credit.

  The pipeline that I envisioned looks like this :

  
  

  

  VMD Subtitling Process

  
  

  “Trivial !” I surmised. I just never learn, do I ?

  The Plan
  So here’s my initial pitch, outlining the work I estimated that I would need to do towards the stated goal :

  1. Create a new file muxer that produces a syntactically valid VMD file with bogus video and audio data. Make sure it works with both FFmpeg’s playback system as well as the proper Phantasmagoria engine.
  2. Create a new video encoder that essentially operates in pass-through mode while correctly building a palette.
  3. Create a new basic encoder for the video frames.

  A big unknown for me was exactly how subtitle handling operates in FFmpeg. Thanks to this project, I now know. I was concerned because I was pretty sure that font rendering entails anti-aliasing which bodes poorly for keeping the palette count under 256 unique colors.

  Computer Science Puzzle
  When pondering how to process the palette, I was excited for the opportunity to exercise actual computer science. FFmpeg converts frames from paletted frames to full RGB frames. Then it needs to convert them back to paletted frames. I had a vague recollection of solving this problem once before when I was experimenting with a new paletted video codec. I seem to recall that I did the palette conversion in a very naive manner. I just used a static 256-element array and processed each RGB pixel of the frame, seeing if the value already occurred in the table (O(n) lookup) and adding it otherwise.
  

  There are more efficient algorithms, however, such as hash tables and trees. Somewhere along the line, FFmpeg helpfully acquired a rarely-used tree data structure, which was perfect for this project.

  So I was pretty pleased with this optimization. Too bad this wouldn’t survive to the end of the effort.

  Another palette-related challenge was the fact that a group of pictures would be accumulating a new palette but that palette needed to be recorded before the group. Thus, the muxer needed to have extra logic to rewind the file when the video encoder transmitted a palette change.

  Video Compression
  VMD has a few methods in its compression toolbox. It can use interframe differencing, it has some RLE, or it can code a frame raw. It can also use a custom LZ-like format on top of these. For early prototypes, I elected to leave each frame coded raw. After the concept was proved, I implemented the frame differencing.

  
  

  

  
  Top frame compared with the middle frame yields the bottom frame : red pixels indicate changes
  

  Encoding only those red dots in between vast runs of unchanged pixels yielded a vast measurable improvement. The next step was to try wiring up FFmpeg’s existing LZ compression facilities to the encoder. This turned out to be implausible since VMD’s LZ variant has nothing to do with anything FFmpeg already provides. Fortunately, the LZ piece is not absolutely required and the frame differencing + RLE provides plenty of compression.

  Subtitling
  I’ve never done anything, multimedia programming-wise, concerning subtitles. I guess all the entertainment I care about has always been in my native tongue. What a good excuse to program outside of my comfort zone !

  First, I needed to know how to access FFmpeg’s subtitling facilities. Fortunately, The Translator did the legwork on this matter so I didn’t have to figure it out.

  However, I intuitively had misgivings about this phase. I had heard that the subtitling process performs anti-aliasing. That means that the image would need to be promoted to a higher colorspace for this phase and that the anti-aliasing process would likely push the color count way past 256. Some quick tests revealed this to be the case, as the running color count would leap by several hundred colors as soon as the palette accounting algorithm encountered a subtitle.

  So I dug into the subtitle subsystem. I discovered that the subtitle library operates by creating a linked list of subtitle bitmaps that the client app must render. The bitmaps are comprised of 8-bit alpha transparency values that must be composited onto the target frame (i.e., 0 = transparent, 255 = 100% opaque). For example, the letter ‘H’ :

                                    (with 00s removed)
  13 F8 41 00 00 00 00 68 E4  |  13 F8 41             68 E4    
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF DC D0 D0 D0 D0 E4 EC  |  14 FF DC D0 D0 D0 D0 E4 EC
  14 FF 7E 50 50 50 50 9A EC  |  14 FF 7E 50 50 50 50 9A EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  11 E0 3B 00 00 00 00 5E CE  |  11 E0 3B             5E CE
  

  To get around the color explosion problem, I chose a threshold value and quantized values above and below to 255 and 0, respectively. Further, the process chooses an appropriate color from the existing palette rather than introducing any new colors.

  Muxing Matters
  In order to force VMD into a general purpose media framework, a lot of special information needs to be passed around. Like many paletted codecs, the palette needs to be transmitted from the file demuxer to the video decoder via some side channel. For re-encoding, this also implies that the palette needs to make the trip from the video encoder to the file muxer. As if this wasn’t enough, individual VMD frames have even more data that needs to be ferried between the muxer and codec levels, including frame change boundaries. FFmpeg provides methods to do these things, but I could not always rely on the systems to relay the data in all cases. I was probably doing something wrong ; I accept that. Instead, I just packed all the information at the front of an encoded frame and split it apart in the muxer.

  I could not quite figure out how to get the audio and video muxed correctly. As a result, neither FFmpeg nor the Phantasmagoria engine could replay the files correctly.

  Plan B
  Since I was having so much trouble creating an entirely new VMD file, likely due to numerous unknown bits of the file format, I thought of another angle : re-use the existing VMD file. For this approach, I kept the video encoder and file muxer that I created in the initial phase, but modified the file muxer to emit a special intermediate file. Then, I created a Python tool to repackage the original VMD file using compressed video data in the intermediate file.

  For this phase, I also implemented a command line switch for FFmpeg to disable subtitle blending, to make the feature feel like less of an unofficial hack, as though this nonsense would ever have a chance of being incorporated upstream.

  At this point, I was seeing some success with the complete, albeit roundabout, subtitling process. I constructed a subtitle file using “Spanish I Learned From Mexican Telenovelas” and the frames turned out fairly readable :

  
  

  

  “she cheated on him”

  
  

  

  “he’s a scumbag” … these random subtitles could fit surprisingly well !

  
  

  The few files that I tested appeared to work fine. But then I handed off my work to The Translator and he immediately found a bunch of problems. According to my notes, the problems mostly took the form of flashing, solid color frames. Further, I found tiny, mostly imperceptible flaws in my RLE compressor, usually only detectable by running strict comparison tools ; but I wasn’t satisfied.

  At this point, I think I attempted to just encode the entire palette at the front of each frame, as allowed by the format, but that did not seem to fix any problems. My notes are not completely clear on this matter (likely because I was still trying to figure out the exact problem), but I think it had to do with FFmpeg inserting extra video frames in order to even out gaps in the video framerate.

  Sigh, Plan C
  At this point, I was getting tired of trying to force FFmpeg to do this. So I decided to minimize its involvement using lessons learned up to this point.

  The next pitch :

  1. Create a new C program that can open an existing VMD file and output an identical VMD file. I know this sounds easy, but the specific method of copying entails interpreting individual parts of the file and writing those individual parts to the new file. This is in preparation for…
  2. Import the VMD video decoder functions directly into the program to decode the individual video frames and re-encode them, replacing the video frames as the file is rewritten.
  3. Wire up the subtitle system. During the adventure to disable subtitle blending, I accidentally learned enough about interfacing to the subtitle library to just invoke it directly.
  4. Rewrite the RLE method so that it is 100% correct.

  Off to work I went. That part about lifting the existing VMD decoder functions out of their libavcodec nest turned out to not be that straightforward. As an alternative, I modified the decoder to dump the raw frames to an intermediate file. In doing so, I think I was able to avoid the issue of the duplicated frames that plagued the previous efforts.

  Also, remember how I was really pleased with the palette conversion technique in which I was able to leverage computer science big-O theory ? By this stage, I had no reason to convert the paletted video to RGB in the first place ; all of the decoding, subtitling and re-encoding operates in the paletted colorspace.

  This approach seemed to work pretty well. The final program is subtitle-vmd.c. The process is still a little weird. The modifications in my own FFmpeg fork are necessary to create an intermediate file that the new C tool can operate with.

  Next Steps
  The Translator has found some assorted bugs and corner cases that still need to be ironed out. Further, for extra credit, I need find the change windows for each frame to improve compression just a little more. I don’t think I will be trying for LZ compression, though.

  However, almost as soon as I had this whole system working, The Translator informed me that there is another, different movie format in play in the Phantasmagoria engine called ROBOT, with an extension of RBT. Fortunately, enough of the algorithms have been reverse engineered and re-implemented in ScummVM that I was able to sort out enough details for another subtitling project. That will be the subject of a future post.

  See Also :

  • Subtitling Sierra RBT Files : The followup in which I discuss how to scribble text on the other animation format

  The post Subtitling Sierra VMD Files first appeared on Breaking Eggs And Making Omelettes.

• Subtitling Sierra VMD Files

  1er juin 2016, par Multimedia Mike — Game Hacking

  I was contacted by a game translation hobbyist from Spain (henceforth known as The Translator). He had set his sights on Sierra’s 7-CD Phantasmagoria. This mammoth game was driven by a lot of FMV files and animations that have speech. These require language translation in the form of video subtitling. He’s lucky that he found possibly the one person on the whole internet who has just the right combination of skill, time, and interest to pull this off. And why would I care about helping ? I guess I share a certain camaraderie with game hackers. Don’t act so surprised. You know what kind of stuff I like to work on.

  The FMV format used in this game is VMD, which makes an appearance in numerous Sierra titles. FFmpeg already supports decoding this format. FFmpeg also supports subtitling video. So, ideally, all that’s necessary to support this goal is to add a muxer for the VMD format which can encode raw video and audio, which the format supports. Implement video compression as extra credit.

  The pipeline that I envisioned looks like this :

  
  

  

  VMD Subtitling Process

  
  

  “Trivial !” I surmised. I just never learn, do I ?

  The Plan
  So here’s my initial pitch, outlining the work I estimated that I would need to do towards the stated goal :

  1. Create a new file muxer that produces a syntactically valid VMD file with bogus video and audio data. Make sure it works with both FFmpeg’s playback system as well as the proper Phantasmagoria engine.
  2. Create a new video encoder that essentially operates in pass-through mode while correctly building a palette.
  3. Create a new basic encoder for the video frames.

  A big unknown for me was exactly how subtitle handling operates in FFmpeg. Thanks to this project, I now know. I was concerned because I was pretty sure that font rendering entails anti-aliasing which bodes poorly for keeping the palette count under 256 unique colors.

  Computer Science Puzzle
  When pondering how to process the palette, I was excited for the opportunity to exercise actual computer science. FFmpeg converts frames from paletted frames to full RGB frames. Then it needs to convert them back to paletted frames. I had a vague recollection of solving this problem once before when I was experimenting with a new paletted video codec. I seem to recall that I did the palette conversion in a very naive manner. I just used a static 256-element array and processed each RGB pixel of the frame, seeing if the value already occurred in the table (O(n) lookup) and adding it otherwise.
  

  There are more efficient algorithms, however, such as hash tables and trees. Somewhere along the line, FFmpeg helpfully acquired a rarely-used tree data structure, which was perfect for this project.

  So I was pretty pleased with this optimization. Too bad this wouldn’t survive to the end of the effort.

  Another palette-related challenge was the fact that a group of pictures would be accumulating a new palette but that palette needed to be recorded before the group. Thus, the muxer needed to have extra logic to rewind the file when the video encoder transmitted a palette change.

  Video Compression
  VMD has a few methods in its compression toolbox. It can use interframe differencing, it has some RLE, or it can code a frame raw. It can also use a custom LZ-like format on top of these. For early prototypes, I elected to leave each frame coded raw. After the concept was proved, I implemented the frame differencing.

  
  

  

  
  Top frame compared with the middle frame yields the bottom frame : red pixels indicate changes
  

  Encoding only those red dots in between vast runs of unchanged pixels yielded a vast measurable improvement. The next step was to try wiring up FFmpeg’s existing LZ compression facilities to the encoder. This turned out to be implausible since VMD’s LZ variant has nothing to do with anything FFmpeg already provides. Fortunately, the LZ piece is not absolutely required and the frame differencing + RLE provides plenty of compression.

  Subtitling
  I’ve never done anything, multimedia programming-wise, concerning subtitles. I guess all the entertainment I care about has always been in my native tongue. What a good excuse to program outside of my comfort zone !

  First, I needed to know how to access FFmpeg’s subtitling facilities. Fortunately, The Translator did the legwork on this matter so I didn’t have to figure it out.

  However, I intuitively had misgivings about this phase. I had heard that the subtitling process performs anti-aliasing. That means that the image would need to be promoted to a higher colorspace for this phase and that the anti-aliasing process would likely push the color count way past 256. Some quick tests revealed this to be the case, as the running color count would leap by several hundred colors as soon as the palette accounting algorithm encountered a subtitle.

  So I dug into the subtitle subsystem. I discovered that the subtitle library operates by creating a linked list of subtitle bitmaps that the client app must render. The bitmaps are comprised of 8-bit alpha transparency values that must be composited onto the target frame (i.e., 0 = transparent, 255 = 100% opaque). For example, the letter ‘H’ :

                                    (with 00s removed)
  13 F8 41 00 00 00 00 68 E4  |  13 F8 41             68 E4    
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF DC D0 D0 D0 D0 E4 EC  |  14 FF DC D0 D0 D0 D0 E4 EC
  14 FF 7E 50 50 50 50 9A EC  |  14 FF 7E 50 50 50 50 9A EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  11 E0 3B 00 00 00 00 5E CE  |  11 E0 3B             5E CE
  

  To get around the color explosion problem, I chose a threshold value and quantized values above and below to 255 and 0, respectively. Further, the process chooses an appropriate color from the existing palette rather than introducing any new colors.

  Muxing Matters
  In order to force VMD into a general purpose media framework, a lot of special information needs to be passed around. Like many paletted codecs, the palette needs to be transmitted from the file demuxer to the video decoder via some side channel. For re-encoding, this also implies that the palette needs to make the trip from the video encoder to the file muxer. As if this wasn’t enough, individual VMD frames have even more data that needs to be ferried between the muxer and codec levels, including frame change boundaries. FFmpeg provides methods to do these things, but I could not always rely on the systems to relay the data in all cases. I was probably doing something wrong ; I accept that. Instead, I just packed all the information at the front of an encoded frame and split it apart in the muxer.

  I could not quite figure out how to get the audio and video muxed correctly. As a result, neither FFmpeg nor the Phantasmagoria engine could replay the files correctly.

  Plan B
  Since I was having so much trouble creating an entirely new VMD file, likely due to numerous unknown bits of the file format, I thought of another angle : re-use the existing VMD file. For this approach, I kept the video encoder and file muxer that I created in the initial phase, but modified the file muxer to emit a special intermediate file. Then, I created a Python tool to repackage the original VMD file using compressed video data in the intermediate file.

  For this phase, I also implemented a command line switch for FFmpeg to disable subtitle blending, to make the feature feel like less of an unofficial hack, as though this nonsense would ever have a chance of being incorporated upstream.

  At this point, I was seeing some success with the complete, albeit roundabout, subtitling process. I constructed a subtitle file using “Spanish I Learned From Mexican Telenovelas” and the frames turned out fairly readable :

  
  

  

  “she cheated on him”

  
  

  

  “he’s a scumbag” … these random subtitles could fit surprisingly well !

  
  

  The few files that I tested appeared to work fine. But then I handed off my work to The Translator and he immediately found a bunch of problems. According to my notes, the problems mostly took the form of flashing, solid color frames. Further, I found tiny, mostly imperceptible flaws in my RLE compressor, usually only detectable by running strict comparison tools ; but I wasn’t satisfied.

  At this point, I think I attempted to just encode the entire palette at the front of each frame, as allowed by the format, but that did not seem to fix any problems. My notes are not completely clear on this matter (likely because I was still trying to figure out the exact problem), but I think it had to do with FFmpeg inserting extra video frames in order to even out gaps in the video framerate.

  Sigh, Plan C
  At this point, I was getting tired of trying to force FFmpeg to do this. So I decided to minimize its involvement using lessons learned up to this point.

  The next pitch :

  1. Create a new C program that can open an existing VMD file and output an identical VMD file. I know this sounds easy, but the specific method of copying entails interpreting individual parts of the file and writing those individual parts to the new file. This is in preparation for…
  2. Import the VMD video decoder functions directly into the program to decode the individual video frames and re-encode them, replacing the video frames as the file is rewritten.
  3. Wire up the subtitle system. During the adventure to disable subtitle blending, I accidentally learned enough about interfacing to the subtitle library to just invoke it directly.
  4. Rewrite the RLE method so that it is 100% correct.

  Off to work I went. That part about lifting the existing VMD decoder functions out of their libavcodec nest turned out to not be that straightforward. As an alternative, I modified the decoder to dump the raw frames to an intermediate file. In doing so, I think I was able to avoid the issue of the duplicated frames that plagued the previous efforts.

  Also, remember how I was really pleased with the palette conversion technique in which I was able to leverage computer science big-O theory ? By this stage, I had no reason to convert the paletted video to RGB in the first place ; all of the decoding, subtitling and re-encoding operates in the paletted colorspace.

  This approach seemed to work pretty well. The final program is subtitle-vmd.c. The process is still a little weird. The modifications in my own FFmpeg fork are necessary to create an intermediate file that the new C tool can operate with.

  Next Steps
  The Translator has found some assorted bugs and corner cases that still need to be ironed out. Further, for extra credit, I need find the change windows for each frame to improve compression just a little more. I don’t think I will be trying for LZ compression, though.

  However, almost as soon as I had this whole system working, The Translator informed me that there is another, different movie format in play in the Phantasmagoria engine called ROBOT, with an extension of RBT. Fortunately, enough of the algorithms have been reverse engineered and re-implemented in ScummVM that I was able to sort out enough details for another subtitling project. That will be the subject of a future post.

  See Also :

  • Subtitling Sierra RBT Files : The followup in which I discuss how to scribble text on the other animation format

• ALSA buffer xrun induced by low quality source in ffmpeg capture

  24 juin 2015, par Peter Becich

  I am attempting to transfer some old Video 8 tapes to my computer, though an EasyCap USB stick and the motherboard’s sound line-in, on Ubuntu. I believe the arguments are correctly laid out below to capture from two independent streams, and encode them both into the output MP4 file.

  Edit :

  I can simplify the question a bit, now.

  ALSA buffer overrun (or underrun ?) is induced by the unreliable/noisy audio source. For instance, if ffmpeg captures the beginning of tape playback, this causes "buffer xrun" far beyond when the tape gets up to speed and playback should be normal.

  It is interesting that the bitrate shown in the ffmpeg log shoots up higher than normal when it’s producing a garbage output ! (Is this bitrate a sum of of audio and video bitrates ?)

  I’ve tried a couple of audio encoding codecs, and had the same problem.

  Using libfdk_aac :

    Metadata:
  
      encoder         : Lavf56.15.102
  
      Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv422p, 640x480, q=-1--1, 29.97 fps, 11988 tbn, 29.97 tbc
  
      Metadata:
  
        encoder         : Lavc56.13.100 libx264
  
      Stream #0:1: Audio: aac (libfdk_aac) ([64][0][0][0] / 0x0040), 48000 Hz, mono, s16, 128 kb/s
  
      Metadata:
  
        encoder         : Lavc56.13.100 libfdk_aac
  
  Stream mapping:
  
    Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
  
    Stream #1:0 -> #0:1 (pcm_s16le (native) -> aac (libfdk_aac))
  
  [alsa @ 0x22038a0] ALSA buffer xrun.  0kB time=00:00:00.00 bitrate=N/A    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.1934kB time=00:00:02.76 bitrate=5723.5kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.4795kB time=00:00:05.49 bitrate=7150.1kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.7668kB time=00:00:08.21 bitrate=7646.1kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.1475kB time=00:00:10.94 bitrate=8588.9kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.3822kB time=00:00:13.66 bitrate=8289.0kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.5388kB time=00:00:16.38 bitrate=7695.0kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.6896kB time=00:00:19.10 bitrate=7244.0kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.8980kB time=00:00:21.84 bitrate=7118.8kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.2032kB time=00:00:24.55 bitrate=7349.3kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.4612kB time=00:00:27.27 bitrate=7391.1kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.6660kB time=00:00:29.98 bitrate=7284.6kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.9123kB time=00:00:32.68 bitrate=7299.3kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.0641kB time=00:00:35.39 bitrate=7091.7kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.2601kB time=00:00:38.13 bitrate=7002.6kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.5828kB time=00:00:40.87 bitrate=7181.0kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.8481kB time=00:00:43.60 bitrate=7229.9kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.1461kB time=00:00:46.34 bitrate=7328.0kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.3982kB time=00:00:49.06 bitrate=7342.7kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.6565kB time=00:00:51.77 bitrate=7367.8kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.9718kB time=00:00:54.51 bitrate=7471.3kbits/s    
  
  [alsa @ 0x22038a0] ALSA buffer xrun.2341kB time=00:00:57.25 bitrate=7489.2kbits/s    
  
  ^Cframe= 1760 fps= 29 q=-1.0 Lsize=   53946kB time=00:01:00.04 bitrate=7360.3kbits/s    
  
  video:53880kB audio:53kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.022994%
  
  [libx264 @ 0x2217ac0] frame I:8     Avg QP:24.00  size: 55686
  
  [libx264 @ 0x2217ac0] frame P:1752  Avg QP:27.66  size: 31237
  
  [libx264 @ 0x2217ac0] mb I  I16..4: 100.0%  0.0%  0.0%
  
  [libx264 @ 0x2217ac0] mb P  I16..4: 15.0%  0.0%  0.0%  P16..4: 80.2%  0.0%  0.0%  0.0%  0.0%    skip: 4.8%
  
  [libx264 @ 0x2217ac0] coded y,uvDC,uvAC intra: 45.3% 86.6% 59.4% inter: 65.7% 81.3% 11.5%
  
  [libx264 @ 0x2217ac0] i16 v,h,dc,p: 40% 25% 26%  9%
  
  [libx264 @ 0x2217ac0] i8c dc,h,v,p: 45% 24% 19% 12%
  
  [libx264 @ 0x2217ac0] kb/s:7516.07
  
  Received signal 2: terminating.

  Using libvorbis :

    Metadata:
  
      encoder         : Lavf56.15.102
  
      Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv422p, 640x480, q=-1--1, 29.97 fps, 11988 tbn, 29.97 tbc
  
      Metadata:
  
        encoder         : Lavc56.13.100 libx264
  
      Stream #0:1: Audio: vorbis (libvorbis) ([221][0][0][0] / 0x00DD), 48000 Hz, mono, fltp, 128 kb/s
  
      Metadata:
  
        encoder         : Lavc56.13.100 libvorbis
  
  Stream mapping:
  
    Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
  
    Stream #1:0 -> #0:1 (pcm_s16le (native) -> vorbis (libvorbis))
  
  [alsa @ 0x1a948a0] ALSA buffer xrun.  0kB time=00:00:00.00 bitrate=N/A    
  
  [alsa @ 0x1a948a0] ALSA buffer xrun. 402kB time=00:00:04.37 bitrate= 752.3kbits/s    
  
  [alsa @ 0x1a948a0] ALSA buffer xrun.4122kB time=00:00:08.80 bitrate=3833.0kbits/s    
  
  [alsa @ 0x1a948a0] ALSA buffer xrun.8722kB time=00:00:13.14 bitrate=5436.3kbits/s    
  
  [alsa @ 0x1a948a0] ALSA buffer xrun.3903kB time=00:00:17.51 bitrate=6502.2kbits/s    
  
  [alsa @ 0x1a948a0] ALSA buffer xrun.6625kB time=00:00:21.89 bitrate=6221.4kbits/s    
  
  [alsa @ 0x1a948a0] ALSA buffer xrun.9548kB time=00:00:26.28 bitrate=6092.5kbits/s    
  
  ^Cframe=  851 fps= 26 q=-1.0 Lsize=   22018kB time=00:00:30.69 bitrate=5875.3kbits/s    
  
  video:21996kB audio:12kB subtitle:0kB other streams:0kB global headers:4kB muxing overhead: 0.044897%
  
  [libx264 @ 0x1aa8ac0] frame I:4     Avg QP:23.50  size: 62405
  
  [libx264 @ 0x1aa8ac0] frame P:847   Avg QP:25.58  size: 26297
  
  [libx264 @ 0x1aa8ac0] mb I  I16..4: 100.0%  0.0%  0.0%
  
  [libx264 @ 0x1aa8ac0] mb P  I16..4: 13.2%  0.0%  0.0%  P16..4: 72.0%  0.0%  0.0%  0.0%  0.0%    skip:14.8%
  
  [libx264 @ 0x1aa8ac0] coded y,uvDC,uvAC intra: 40.6% 81.0% 58.6% inter: 58.8% 72.7% 8.6%
  
  [libx264 @ 0x1aa8ac0] i16 v,h,dc,p: 41% 28% 22%  9%
  
  [libx264 @ 0x1aa8ac0] i8c dc,h,v,p: 54% 19% 16% 11%
  
  [libx264 @ 0x1aa8ac0] kb/s:6345.60
  
  Received signal 2: terminating.

  ---

  ffmpeg’s detection of the ALSA stream is seemingly goofed up by the inconsistencies of the tape. In the failure case, only short blips of the tapes audio exist in the output MP4. The audio bitrate of the output file is less than 10 kbps, averaged out across the whole file. The output video seems to be fine, even though the low frames-per-second in the failure case log below.

  The audio and video streams can be captured fine for short amounts of time before a source error occurs ; this provides the success case log. The failure case log was created by intentionally making an error in the source streams — turning on the camera makes a brief noisy signal.

  Is there a setting that needs to be forced to keep ffmpeg recording the audio stream, even when the tape is blank or noisy ?

  Could it be that the libfdk_aac audio encoder is tripped up by the low quality source ?

  The relevant line ; rawvideo stream is piped to this in script at bottom :

  ffmpeg -pixel_format uyvy422 -s:v 720x480 -framerate 29.97 -f rawvideo \
  
  -i $PIPE  -f alsa -i hw:0,0 -vf scale=w=720:h=540 -vcodec libx264 \
  
  -preset ultrafast -shortest -c:a libfdk_aac -b:a 128k -af pan=1:c0=c0  \
  
  -ar 96000 $OUTFILE

  The ar argument was one attempt to force recording.

  ffmpeg log file for (short-lived) success ; high frames-per-second captured :

  ffmpeg version 2.5.3 Copyright (c) 2000-2015 the FFmpeg developers
  
    built on Jan 11 2015 17:53:45 with gcc 4.8 (Ubuntu 4.8.2-19ubuntu1)
  
    configuration: --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-avresample --disable-debug --enable-nonfree --enable-gpl --enable-version3 --enable-libpulse --enable-libopencore-amrnb --enable-libopencore-amrwb --disable-decoder=amrnb --disable-decoder=amrwb --enable-libx264 --enable-libx265 --enable-libfdk-aac --enable-libvorbis --enable-libmp3lame --enable-libopus --enable-libvpx --enable-libspeex --enable-libass --enable-avisynth --enable-libsoxr --enable-libxvid --enable-libvo-aacenc --enable-libvidstab
  
    libavutil      54. 15.100 / 54. 15.100
  
    libavcodec     56. 13.100 / 56. 13.100
  
    libavformat    56. 15.102 / 56. 15.102
  
    libavdevice    56.  3.100 / 56.  3.100
  
    libavfilter     5.  2.103 /  5.  2.103
  
    libavresample   2.  1.  0 /  2.  1.  0
  
    libswscale      3.  1.101 /  3.  1.101
  
    libswresample   1.  1.100 /  1.  1.100
  
    libpostproc    53.  3.100 / 53.  3.100
  
  Input #0, rawvideo, from '/tmp/somagic-pipe':
  
    Duration: N/A, start: 0.000000, bitrate: 165722 kb/s
  
      Stream #0:0: Video: rawvideo (UYVY / 0x59565955), uyvy422, 720x480, 165722 kb/s, 29.97 tbr, 29.97 tbn, 29.97 tbc
  
  Home directory not accessible: Permission denied
  
  Guessed Channel Layout for  Input Stream #1.0 : stereo
  
  Input #1, alsa, from 'hw:0,0':
  
    Duration: N/A, start: 1423202268.577088, bitrate: 1536 kb/s
  
      Stream #1:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
  
  No pixel format specified, yuv422p for H.264 encoding chosen.
  
  Use -pix_fmt yuv420p for compatibility with outdated media players.
  
  [Parsed_pan_0 @ 0x3335d60] This syntax is deprecated. Use '|' to separate the list items.
  
  Single channel layout '1' is interpreted as a number of channels, switch to the syntax '1c' otherwise it will be interpreted as a channel layout number in a later version
  
  [Parsed_pan_0 @ 0x3335d60] Pure channel mapping detected: 0
  
  [libx264 @ 0x3364bc0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.1 Cache64
  
  [libx264 @ 0x3364bc0] profile High 4:2:2, level 3.1, 4:2:2 8-bit
  
  [libx264 @ 0x3364bc0] 264 - core 142 r2389 956c8d8 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=3 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
  
  Output #0, mp4, to '/home/peterbecich/easycap/Videos/fpv_video_02_05_2015_21_57_48.mp4':
  
    Metadata:
  
      encoder         : Lavf56.15.102
  
      Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv422p, 720x540, q=-1--1, 29.97 fps, 11988 tbn, 29.97 tbc
  
      Metadata:
  
        encoder         : Lavc56.13.100 libx264
  
      Stream #0:1: Audio: aac (libfdk_aac) ([64][0][0][0] / 0x0040), 96000 Hz, mono, s16, 128 kb/s
  
      Metadata:
  
        encoder         : Lavc56.13.100 libfdk_aac
  
  Stream mapping:
  
    Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
  
    Stream #1:0 -> #0:1 (pcm_s16le (native) -> aac (libfdk_aac))
  
  Press [q] to stop, [?] for help
  
  frame=   13 fps=0.0 q=26.0 size=     187kB time=00:00:00.30 bitrate=5102.7kbits/s    
  
  frame=   29 fps= 29 q=26.0 size=     469kB time=00:00:00.83 bitrate=4607.6kbits/s    
  
  frame=   44 fps= 29 q=26.0 size=     755kB time=00:00:01.33 bitrate=4635.2kbits/s    
  
  frame=   59 fps= 29 q=26.0 size=    1024kB time=00:00:01.83 bitrate=4572.1kbits/s    
  
  frame=   74 fps= 29 q=26.0 size=    1279kB time=00:00:02.33 bitrate=4486.5kbits/s    
  
  frame=   89 fps= 29 q=26.0 size=    1516kB time=00:00:02.83 bitrate=4378.0kbits/s    
  
  frame=  104 fps= 29 q=26.0 size=    1752kB time=00:00:03.33 bitrate=4301.0kbits/s    
  
  frame=  119 fps= 29 q=26.0 size=    1991kB time=00:00:03.83 bitrate=4251.1kbits/s    
  
  frame=  135 fps= 30 q=26.0 size=    2245kB time=00:00:04.37 bitrate=4207.5kbits/s    
  
  frame=  150 fps= 30 q=26.0 size=    2524kB time=00:00:04.87 bitrate=4245.0kbits/s    
  
  frame=  165 fps= 30 q=26.0 size=    2808kB time=00:00:05.37 bitrate=4282.0kbits/s    
  
  frame=  180 fps= 30 q=26.0 size=    3091kB time=00:00:05.87 bitrate=4311.5kbits/s    
  
  [rawvideo @ 0x3350640] Invalid buffer size, packet size 65536 < expected frame_size 691200
  
  Error while decoding stream #0:0: Invalid argument
  
  frame=  183 fps= 29 q=-1.0 Lsize=    3247kB time=00:00:06.11 bitrate=4351.5kbits/s    
  
  video:3142kB audio:96kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.254788%
  
  [libx264 @ 0x3364bc0] frame I:1     Avg QP:20.00  size:  2289
  
  [libx264 @ 0x3364bc0] frame P:182   Avg QP:25.99  size: 17664
  
  [libx264 @ 0x3364bc0] mb I  I16..4: 100.0%  0.0%  0.0%
  
  [libx264 @ 0x3364bc0] mb P  I16..4: 78.5%  0.0%  0.0%  P16..4: 20.2%  0.0%  0.0%  0.0%  0.0%    skip: 1.4%
  
  [libx264 @ 0x3364bc0] coded y,uvDC,uvAC intra: 84.1% 71.5% 18.9% inter: 51.9% 63.5% 0.4%
  
  [libx264 @ 0x3364bc0] i16 v,h,dc,p: 15%  8% 69%  8%
  
  [libx264 @ 0x3364bc0] i8c dc,h,v,p: 50% 19% 24%  7%
  
  [libx264 @ 0x3364bc0] kb/s:4215.02

  ffmpeg log for failure ; low FPS captured :

  ffmpeg version 2.5.3 Copyright (c) 2000-2015 the FFmpeg developers
  
    built on Jan 11 2015 17:53:45 with gcc 4.8 (Ubuntu 4.8.2-19ubuntu1)
  
    configuration: --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-avresample --disable-debug --enable-nonfree --enable-gpl --enable-version3 --enable-libpulse --enable-libopencore-amrnb --enable-libopencore-amrwb --disable-decoder=amrnb --disable-decoder=amrwb --enable-libx264 --enable-libx265 --enable-libfdk-aac --enable-libvorbis --enable-libmp3lame --enable-libopus --enable-libvpx --enable-libspeex --enable-libass --enable-avisynth --enable-libsoxr --enable-libxvid --enable-libvo-aacenc --enable-libvidstab
  
    libavutil      54. 15.100 / 54. 15.100
  
    libavcodec     56. 13.100 / 56. 13.100
  
    libavformat    56. 15.102 / 56. 15.102
  
    libavdevice    56.  3.100 / 56.  3.100
  
    libavfilter     5.  2.103 /  5.  2.103
  
    libavresample   2.  1.  0 /  2.  1.  0
  
    libswscale      3.  1.101 /  3.  1.101
  
    libswresample   1.  1.100 /  1.  1.100
  
    libpostproc    53.  3.100 / 53.  3.100
  
  Input #0, rawvideo, from '/tmp/somagic-pipe':
  
    Duration: N/A, start: 0.000000, bitrate: 165722 kb/s
  
      Stream #0:0: Video: rawvideo (UYVY / 0x59565955), uyvy422, 720x480, 165722 kb/s, 29.97 tbr, 29.97 tbn, 29.97 tbc
  
  Home directory not accessible: Permission denied
  
  Guessed Channel Layout for  Input Stream #1.0 : stereo
  
  Input #1, alsa, from 'hw:0,0':
  
    Duration: N/A, start: 1423201999.226455, bitrate: 1536 kb/s
  
      Stream #1:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
  
  No pixel format specified, yuv422p for H.264 encoding chosen.
  
  Use -pix_fmt yuv420p for compatibility with outdated media players.
  
  [Parsed_pan_0 @ 0x21cad60] This syntax is deprecated. Use '|' to separate the list items.
  
  Single channel layout '1' is interpreted as a number of channels, switch to the syntax '1c' otherwise it will be interpreted as a channel layout number in a later version
  
  [Parsed_pan_0 @ 0x21cad60] Pure channel mapping detected: 0
  
  [libx264 @ 0x21f9bc0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.1 Cache64
  
  [libx264 @ 0x21f9bc0] profile High 4:2:2, level 3.1, 4:2:2 8-bit
  
  [libx264 @ 0x21f9bc0] 264 - core 142 r2389 956c8d8 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=3 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
  
  Output #0, mp4, to '/home/peterbecich/easycap/Videos/fpv_video_02_05_2015_21_53_18.mp4':
  
    Metadata:
  
      encoder         : Lavf56.15.102
  
      Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv422p, 720x540, q=-1--1, 29.97 fps, 11988 tbn, 29.97 tbc
  
      Metadata:
  
        encoder         : Lavc56.13.100 libx264
  
      Stream #0:1: Audio: aac (libfdk_aac) ([64][0][0][0] / 0x0040), 96000 Hz, mono, s16, 128 kb/s
  
      Metadata:
  
        encoder         : Lavc56.13.100 libfdk_aac
  
  Stream mapping:
  
    Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
  
    Stream #1:0 -> #0:1 (pcm_s16le (native) -> aac (libfdk_aac))
  
  Press [q] to stop, [?] for help
  
  frame=    1 fps=0.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A    
  
  frame=    1 fps=1.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A    
  
  frame=    1 fps=0.7 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A    
  
  frame=    1 fps=0.5 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A    
  
  frame=    1 fps=0.4 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A    
  
  frame=    1 fps=0.3 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A    
  
  frame=    1 fps=0.3 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A    
  
  frame=    1 fps=0.2 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A    
  
  frame=    1 fps=0.2 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A    
  
  frame=    1 fps=0.2 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A    
  
  frame=    1 fps=0.2 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A    
  
  [alsa @ 0x21e5ac0] ALSA buffer xrun.
  
  frame=    8 fps=1.3 q=19.0 size=      12kB time=00:00:06.03 bitrate=  15.9kbits/s    
  
  frame=   23 fps=3.5 q=18.0 size=      12kB time=00:00:06.03 bitrate=  16.4kbits/s    
  
  frame=   38 fps=5.4 q=18.0 size=      12kB time=00:00:06.03 bitrate=  16.7kbits/s    
  
  frame=   53 fps=7.0 q=18.0 size=      12kB time=00:00:06.03 bitrate=  16.9kbits/s    
  
  frame=   68 fps=8.4 q=26.0 size=     146kB time=00:00:06.03 bitrate= 198.8kbits/s    
  
  frame=   83 fps=9.7 q=26.0 size=     375kB time=00:00:06.03 bitrate= 510.0kbits/s    
  
  frame=   98 fps= 11 q=26.0 size=     608kB time=00:00:06.03 bitrate= 826.5kbits/s    
  
  frame=  114 fps= 12 q=26.0 size=     875kB time=00:00:06.03 bitrate=1189.1kbits/s    
  
  frame=  128 fps= 13 q=26.0 size=    1091kB time=00:00:06.03 bitrate=1481.6kbits/s    
  
  frame=  144 fps= 14 q=26.0 size=    1339kB time=00:00:06.03 bitrate=1819.2kbits/s    
  
  frame=  159 fps= 14 q=26.0 size=    1571kB time=00:00:06.03 bitrate=2134.6kbits/s    
  
  frame=  174 fps= 15 q=26.0 size=    1796kB time=00:00:06.03 bitrate=2440.1kbits/s    
  
  [alsa @ 0x21e5ac0] ALSA buffer xrun.
  
  frame=  189 fps= 16 q=26.0 size=    2015kB time=00:00:12.04 bitrate=1370.4kbits/s    
  
  frame=  204 fps= 16 q=26.0 size=    2238kB time=00:00:12.04 bitrate=1522.3kbits/s    
  
  frame=  219 fps= 17 q=26.0 size=    2490kB time=00:00:12.04 bitrate=1694.2kbits/s    
  
  frame=  235 fps= 17 q=26.0 size=    2728kB time=00:00:12.04 bitrate=1855.8kbits/s    
  
  frame=  250 fps= 18 q=26.0 size=    2973kB time=00:00:12.04 bitrate=2022.4kbits/s    
  
  [rawvideo @ 0x21e5640] Invalid buffer size, packet size 65536 < expected frame_size 691200
  
  Error while decoding stream #0:0: Invalid argument
  
  frame=  261 fps= 18 q=-1.0 Lsize=    3269kB time=00:00:12.06 bitrate=2220.1kbits/s    
  
  video:3263kB audio:4kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.081101%
  
  [libx264 @ 0x21f9bc0] frame I:2     Avg QP:21.50  size: 21342
  
  [libx264 @ 0x21f9bc0] frame P:259   Avg QP:24.22  size: 12734
  
  [libx264 @ 0x21f9bc0] mb I  I16..4: 100.0%  0.0%  0.0%
  
  [libx264 @ 0x21f9bc0] mb P  I16..4: 62.8%  0.0%  0.0%  P16..4: 14.2%  0.0%  0.0%  0.0%  0.0%    skip:22.9%
  
  [libx264 @ 0x21f9bc0] coded y,uvDC,uvAC intra: 77.7% 61.2% 14.1% inter: 19.7% 24.8% 1.6%
  
  [libx264 @ 0x21f9bc0] i16 v,h,dc,p: 17% 10% 65%  8%
  
  [libx264 @ 0x21f9bc0] i8c dc,h,v,p: 52% 18% 24%  6%
  
  [libx264 @ 0x21f9bc0] kb/s:3068.90

  The whole script :

  #!/bin/sh
  
  
  
  PIPE=/tmp/somagic-pipe
  
  OUTFILEDIR=~/easycap/Videos/
  
  LOGDIR=~/.somagic-log/
  
  NOW=`date +"%m_%d_%Y_%H_%M_%S"`
  
  
  
  OUTFILE=${OUTFILEDIR}fpv_video_${NOW}.mp4
  
  
  
  mkdir $LOGDIR
  
  
  
  FFMPEG_LOG=${LOGDIR}ffmpeg.log
  
  SOMAGIC_LOG=${LOGDIR}somagic.log
  
  MPLAYER_LOG=${LOGDIR}mplayer.log
  
  
  
  rm $PIPE >/dev/null 2>&1
  
  rm $OUTFILE >/dev/null 2>&1
  
  
  
  rm $FFMPEG_LOG
  
  rm $SOMAGIC_LOG
  
  rm $MPLAYER_LOG
  
  
  
  mkfifo $PIPE >/dev/null 2>&1
  
  
  
  ffmpeg -pixel_format uyvy422 -s:v 720x480 -framerate 29.97 -f rawvideo \
  
  -i $PIPE  -f alsa -i hw:0,0 -vf scale=w=720:h=540 -vcodec libx264 \
  
  -preset ultrafast -shortest -c:a libfdk_aac -b:a 128k -af pan=1:c0=c0  \
  
  -ar 96000 $OUTFILE > $FFMPEG_LOG 2>&1 &
  
  
  
  somagic-capture --ntsc -c --luminance=2 --lum-aperture=3 2> $SOMAGIC_LOG \
  
  | tee $PIPE | \
  
  mplayer -vf yadif,screenshot -demuxer rawvideo -rawvideo \
  
  "ntsc:format=uyvy:fps=30000/1001" -aspect 4:3 - 2> $MPLAYER_LOG
  
  
  
  
  
  rm $PIPE >/dev/null 2>&1 

  Modified from here : https://gist.github.com/Brick85/0b327ac2d3d45e23ed33