Breaking Eggs And Making Omelettes

Les articles publiés sur le site

• Dreamcast Development Desktop

  28 mars 2011, par Multimedia Mike — Sega Dreamcast

  Some people are curious about what kind of equipment is required to program a Sega Dreamcast. This is my setup:

  
  
  

  It's a bit overcomplicated. The only piece in that picture which doesn't play a role in the Dreamcast development process is the scanner. The Eee PC does the heavy lifting of development (i.e., text editing and cross compilation) and uploads to the Dreamcast via a special serial cable. Those are the most essential parts and are really the only pieces necessary for a lot of algorithmic stuff (things that can be validated via a serial console). But then I have to go up a level where I output video. That's where things get messy.

  
  
  

  The Mac Mini and giant monitor really just act as a glorified TV in this case. Ideally, it will be more than that. The DC outputs audio and video via composite cables to a Canopus DV capture bridge. That's connected via FireWire to the external hard drive underneath the Mac Mini, which is connected to the Mac. Adobe Premiere Pro handles the DV capture / display.

  One day I hope to have something worthwhile to capture.

• Announcing Dreamroq

  20 mars 2011, par Multimedia Mike — Sega Dreamcast

  I have pushed the first public version of my new Dreamroq (we open source types are just so creative in our naming schemes) library up to Github:

  https://github.com/multimediamike/dreamroq

  To review, this is a library for playing RoQ video files on the Sega Dreamcast. It has a lot of problems right now, many of which are listed in the README file. It comes with a Dreamcast/KOS sample player. It also comes with a simple Unix testing utility which decodes a file to a series of PNM files. If anyone is interested in debugging RoQ again after all these years that it has been supported in other open source programs, there is still a bug in the video decoder that produces some notable artifacts.

  Also, here are some samples I generated that appear to meet the data rate requirements laid out in this post. These samples are based on this sample media set from Xiph.org.

• RoQ on Dreamcast

  18 mars 2011, par Multimedia Mike — Sega Dreamcast

  I have been working on that challenge to play back video on the Sega Dreamcast. To review, I asserted that the RoQ format would be a good fit for the Sega Dreamcast hardware. The goal was to play 640x480 video at 30 frames/second. Short version: I have determined that it is possible to decode such video in real time. However, I ran into certain data rate caveats.

  First off: Have you ever wondered if the Dreamcast can read an 80mm optical disc? It can! I discovered this when I only had 60 MB of RoQ samples to burn on a disc and a spindle full of these 210MB-capacity 80mm CD-Rs that I never have occasion to use.

  
  
  

  New RoQ Library
  There are open source RoQ decoders out there but I decided to write a new one. A few reasons: 1) RoQ is so simple that I didn't think it would take too long; 2) it would be nice to have a RoQ library that is license-compatible (BSD-like) with the rest of the KallistiOS distribution; 3) the idroq.tar.gz distribution, while license-compatible, has enough issues that I didn't want to correct it.

  Thankfully, I was correct about the task not being too difficult: I put together a new RoQ decoder in short order. I'm a bit embarrassed to admit that the part I had the most trouble with was properly converting YUV -> RGB.
  
  About the approach I took: While the original idroq.tar.gz decoder maintains YUV 4:2:0 codebooks (which led to chroma bugs during motion compensation) and FFmpeg's decoder maintains YUV 4:4:4 codebooks, this decoder is built to convert the YUV 4:2:0 vectors into RGB565 vectors during the vector unpacking phase. Thus, the entire frame is rendered in RGB565 -- no lengthy YUV -> RGB conversion after decoding -- and all pixels are shuffled around as 16-bit units (minor speedup vs. shuffling everything as bytes).

  I also entertained the idea of maintaining YUYV codebooks (since the DC supports that colorspace as a texture format). But I scrapped that idea when I remembered it would lead to the same chroma bleeding problem seen in the original idroq.tar.gz decoder.

  Onto The Dreamcast
  I developed the library on a Linux computer, allowing it to output a series of PNM files for visual verification and debugging. Dropping it into a basic DC/KOS-compatible program was trivial and the first order of business was profiling.

  At first, I profiled the entire decode operation: open file, then read and decode each chunk while tossing away the results. I was roundly disappointed to see that, e.g., an 8.5-second RoQ sample needed a little more than 20 seconds to complete. Not real time. I performed a series of optimizations on the decoding library that netted notable performance gains when profiling on Linux. When I brought these same optimizations over to the DC, decoding time didn't improve at all. This was my first suspicion that perhaps my assumptions regarding the DC's optical drive's data rate were not correct.

  Dreamcast Data Rate Profiling
  Let's start with some definitions: In terms of data rate, an 'X', i.e., 1X is the minimum data rate needed to read CD quality audio from a disc. At that speed, a drive should be able to stream 75 sectors each second. When reading mode 1/form 1 CD-ROM data, each sector has 2048 bytes (2 kbytes), so a single-speed data rate should achieve 150 kbytes/sec.

  The Dreamcast is supposed to possess a 12X optical drive. This would imply a maximum data rate of 150 kbytes/sec * 12 = 1800 kbytes/sec.

  Rigging up a trivial experiment using the RoQ samples burned on a few different CD-R discs, the best data rate I can see is about 500-525 kbytes/sec, or around 3.5X.

  Where's the discrepancy? My first theory has to do with the fact that not all optical media is created equal. This is why optical drives often advertise a slew of numbers which refer to the best theoretical speed for reading a CD vs. writing a CD-R vs. writing a CD-RW, etc. Perhaps the DC drive can't read CD-Rs very quickly. To test this theory, I tried streaming a large file from a conventionally mastered CD-ROM. This worked well for the closest CD-ROM I had on hand: I was able to stream data at a rate that works out to about 6.5X.

  I smell a science project for another evening: Profiling read speeds from a mastered CD-ROM, burned CD-R, and also a mastered GD-ROM, on each of the 3 Dreamcast consoles I possess (I've heard that there's variance between optical drives depending on manufacturing run).

  The Good News
  I added a little finer-grained code to profile just the video decoding functions. The good news is that the decoder meets my real time goals: That 8.5-second RoQ sample encoded at 640x480x30fps makes its way through the video decoding functions on the DC in a little less than 5 seconds. If the optical drive can supply the data fast enough, the video decoder can take care of the rest.

  The RoQ encoder included with FFmpeg does not honor any bitrate parameters. Instead, I encoded the same file at 320x240. It reportedly decoded in real time and can be streamed in real time as well.

  I say "reportedly" because I'm simply working from textual output at this point; the next phase is to hook the decoder up to the display hardware.

• Playing Video on a Sega Dreamcast

  9 mars 2011, par Multimedia Mike — Sega Dreamcast

  Here's an honest engineering question: If you were tasked to make compressed video play back on a Sega Dreamcast video game console, what video format would you choose? Personally, I would choose RoQ, the format invented for The 11th Hour computer game and later used in Quake III and other games derived from the same engine. This post explains my reasoning.

  Video Background
  One of the things I wanted to do when I procured a used Sega Dreamcast back in 2001 was turn it into a set-top video playback unit. This is something that a lot of people tried to do, apparently, to varying degrees of success. Interest would wane in a few years as it became easier and easier to crack an Xbox and install XBMC. The Xbox was much better suited to playing codecs that were getting big at the time, most notably MPEG-4 part 2 video (DivX/XviD).

  The Dreamcast, while quite capable when it was released in 1999, was not very well-equipped to deal with an MPEG-type codec. I have recently learned that there are other hackers out there on the internet who are still trying to get the most out of this system. I was contacted for advice about how to make Theora perform better on the Dreamcast.
  
  Interesting thing about consoles and codecs: Since you are necessarily distributing code along with your data, you have far more freedom to use whatever codecs you want for your audio and video data. This is why Vorbis and even Theora have seen quite a bit of use in video games, "internet standards" be darned. Thus, when I realized this application had no hard and fast requirement to use Theora, and that it could use any codec that fit the platform, my mind started churning. When I was programming the DC 10 years ago, I didn't have access to the same wealth of multimedia knowledge that is currently available.

  Requirements Gathering
  What do we need here?

  • Codec needs to run on the Sega Dreamcast; this eliminates codecs for which only binary decoder implementations are available
  • Must decode 320x240 video at 30 fps; higher resolutions up to 640x480 would be desirable
  • Must deliver decent quality at 12X optical read speeds (DC drive speed)
  • There must be some decent, preferably free, encoder readily available; speed of encoding, however, is not important; i.e., "take as long as you need, encoder"

  Theora was the go-to codec because it's just commonly known as "the free, open source video codec". But clearly it's not suitable for, well... any purpose, really (sorry, easy target; OW! stop throwing things!). VP8/WebM -- Theora's heir apparent -- would not qualify either, as my prior experiments have already demonstrated.

  Candidates
  What did the big boys use for video on the Dreamcast? A lot of games relied on CRI's Sofdec middleware which was MPEG-1 video and a custom ADPCM format. I don't know if I have ever seen DC games that used MPEG-1 video at a higher resolution than 320x240 (though I have not searched exhaustively). The fact that CRI used a custom ADPCM format for this application may indicate that there wasn't enough CPU power left over to decode a perceptual, transform-based audio codec alongside the 320x240 video.

  A few other DC games used 4X Technologies' 4XM format. The most notable licensee was Alone in the Dark: The New Nightmare (DC version only; PC version used Bink). This codec was DCT-based but incorporated 16-bit RGB colorspace into its design, presumably to optimize for applications like game consoles that couldn't directly handle planar YUV. AITD:TNN's videos were 640x360, a marked improvement over the typical Sofdec fare. I was about to write off 4XM as a contender due to lack of encoder, but the encoding tools are preserved on our samples site. A few other issues, though: The FFmpeg decoder doesn't seem to work correctly as of this writing (and nobody has noticed yet, even though it's tested via FATE).

  What ideas do I have? Right off the bat, I'm thinking vector quantizer (VQ). Vector quantizers are notoriously slow to compress but are blazingly fast to decompress which is why they were popular in the early days of video compression. First, there's Cinepak. I fear that might be too simple for this application. Plus, I don't know if existing (binary-only) compressors are very decent. It seems that they only ever had to handle small videos and I've heard that they can really fall over if anything more is demanded of them.

  Sorenson Video 1 is another contender. FFmpeg has an encoder (which some allege is better than Sorenson's original compressor). However, I fear that the wonky algorithm and colorspace might not mesh well with the Dreamcast.

  My thinking quickly converged on RoQ. This was designed to run fullscreen (640x480) video on i486-class hardware. While RoQ fundamentally operates in a YUV colorspace, it's trivial to convert it to any other colorspace during decoding and the image will be rendered in that colorspace. Plus, there are open source encoders available for the format (namely, several versions of Eric Lasota's Switchblade encoder, one of which lives natively in FFmpeg), as well as the original proprietary encoder.

  Which Library?
  There are several code choices here: FFmpeg (LGPL), Switchblade (GPL), and the original Quake 3 source code (GPL). There is one more option that I think might be easiest, which is the decoder Dr. Tim created when he reverse engineered the format in the first place. That has a very liberal "do whatever you like, but be nice and give me credit" license (probably qualifies as BSD).

  This code is no longer at its original home but the Wayback Machine still had a copy, which I have now mirrored (idroq.tar.gz).

  Adaptation
  Dr. Tim's code still compiles and runs great on Linux (64-bit!) with SDL output. I would like to get it ported to the Dreamcast using the same SDL output, which KallistiOS supports. Then, there is the matter of fixing the longstanding chroma bug in the original sample decoder (described here). The decoder also needs to be modified to natively render RGB565 data, as that will work best with the DC's graphics hardware.

  After making the code work, I want to profile it and test whether it can handle full-frame 640x480 playback at 30 frames/second. I will need to contrive a sample to achieve this.

  Unfortunately, things went off the rails pretty quickly when I tried to get the RoQ decoder ported to DC/KOS. It looks like there's a bug in KallistiOS's minimalistic standard C library, or at least a discrepancy with my desktop Linux system. When you read to the end of a file and then seek backwards to someplace that isn't the end, is the file still in EOF state?

  According to my Linux desktop:

  open file;          feof() = 0
  seek to end;        feof() = 0
  read one more byte; feof() = 1
  seek back to start; feof() = 0
  

  According to KallistiOS:

  open file;          feof() = 0
  seek to end;        feof() = 0
  read one more byte; feof() = 1
  seek back to start; feof() = 1
  

  Here's the seek-test.c program I used to test this issue:

  PLAIN TEXT

  C:

  1. #include <stdio .h>
  2.  
  3. int main()
  4. {
  5.     FILE *f;
  6.     unsigned char byte;
  7.  
  8.     f = fopen("seek_test.c", "r");
  9.     printf("open file;          feof() = %d\n", feof(f));
  10.     fseek(f, 0, SEEK_END);
  11.     printf("seek to end;        feof() = %d\n", feof(f));
  12.     fread(&byte, 1, 1, f);
  13.     printf("read one more byte; feof() = %d\n", feof(f));
  14.     fseek(f, 0, SEEK_SET);
  15.     printf("seek back to start; feof() = %d\n", feof(f));
  16.     fclose(f);
  17.  
  18.     return 0;
  19. }

  EOF
  Speaking of EOF, I'm about done for this evening.

  What codec would you select for this task, given the requirements involved?

• Samples Archive Mirror

  27 février 2011, par Multimedia Mike — General

  I'm pleased to announce the first (to my knowledge) public mirror of the world-famous MPlayerHQ multimedia samples archive, hosted right here at multimedia.cx (can you think of a better place?):

  http://samples.multimedia.cx/

  Here's hoping the load doesn't crush my simple hosting plan (which actually claims quite a bit of bandwidth per month, but we'll see).