Recherche avancée

Sur d’autres sites (5946)

• Parsing The Clue Chronicles

  30 décembre 2018, par Multimedia Mike — Game Hacking

  A long time ago, I procured a 1999 game called Clue Chronicles : Fatal Illusion, based on the classic board game Clue, a.k.a. Cluedo. At the time, I was big into collecting old, unloved PC games so that I could research obscure multimedia formats.

  
  
  

  Surveying the 3 CD-ROMs contained in the box packaging revealed only Smacker (SMK) videos for full motion video which was nothing new to me or the multimedia hacking community at the time. Studying the mix of data formats present on the discs, I found a selection of straightforward formats such as WAV for audio and BMP for still images. I generally find myself more fascinated by how computer games are constructed rather than by playing them, and this mix of files has always triggered a strong “I could implement a new engine for this !” feeling in me, perhaps as part of the ScummVM project which already provides the core infrastructure for reimplementing engines for 2D adventure games.

  Tying all of the assets together is a custom high-level programming language. I have touched on this before in a blog post over a decade ago. The scripts are in a series of files bearing the extension .ini (usually reserved for configuration scripts, but we’ll let that slide). A representative sample of such a script can be found here :

  clue-chronicles-scarlet-1.txt

  What Is This Language ?
  At the time I first analyzed this language, I was still primarily a C/C++-minded programmer, with a decent amount of Perl experience as a high level language, and had just started to explore Python. I assessed this language to be “mildly object oriented with C++-type comments (‘//’) and reliant upon a number of implicit library functions”. Other people saw other properties. When I look at it nowadays, it reminds me a bit more of JavaScript than C++. I think it’s sort of a Rorschach test for programming languages.

  Strangely, I sort of had this fear that I would put a lot of effort into figuring out how to parse out the language only for someone to come along and point out that it’s a well-known yet academic language that already has a great deal of supporting code and libraries available as open source. Google for “spanish dolphins far side comic” for an illustration of the feeling this would leave me with.

  It doesn’t matter in the end. Even if such libraries exist, how easy would they be to integrate into something like ScummVM ? Time to focus on a workable approach to understanding and processing the format.

  Problem Scope
  So I set about to see if I can write a program to parse the language seen in these INI files. Some questions :

  1. How large is the corpus of data that I need to be sure to support ?
  2. What parsing approach should I take ?
  3. What is the exact language format ?
  4. Other hidden challenges ?

  To figure out how large the data corpus is, I counted all of the INI files on all of the discs. There are 138 unique INI files between the 3 discs. However, there are 146 unique INI files after installation. This leads to a hidden challenge described a bit later.

  What parsing approach should I take ? I worried a bit too much that I might not be doing this the “right” way. I’m trying to ignore doubts like this, like how “SQL Shame” blocked me on a task for a little while a few years ago as I concerned myself that I might not be using the purest, most elegant approach to the problem. I know I covered language parsing a lot time ago in university computer science education and there is a lot of academic literature to the matter. But sometimes, you just have to charge in and experiment and prototype and see what falls out. In doing so, I expect to have a better understanding of the problems that need to solved and the right questions to ask, not unlike that time that I wrote a continuous integration system from scratch because I didn’t actually know that “continuous integration” was the keyword I needed.

  Next, what is the exact language format ? I realized that parsing the language isn’t the first and foremost problem here– I need to know exactly what the language is. I need to know what the grammar are keywords are. In essence, I need to reverse engineer the language before I write a proper parser for it. I guess that fits in nicely with the historical aim of this blog (reverse engineering).

  Now, about the hidden challenges– I mentioned that there are 8 more INI files after the game installs itself. Okay, so what’s the big deal ? For some reason, all of the INI files are in plaintext on the CD-ROM but get compressed (apparently, according to file size ratios) when installed to the hard drive. This includes those 8 extra INI files. I thought to look inside the CAB installation archive file on the CD-ROM and the files were there… but all in compressed form. I suspect that one of the files forms the “root” of the program and is the launching point for the game.

  Parsing Approach
  I took a stab at parsing an INI file. My approach was to first perform lexical analysis on the file and create a list of 4 types : symbols, numbers, strings, and language elements ([]{}()=., :). Apparently, this is the kind of thing that Lex/Flex are good at. This prototyping tool is written in Python, but when I port this to ScummVM, it might be useful to call upon the services of Lex/Flex, or another lexical analyzer, for there are many. I have a feeling it will be easier to use better tools when I understand the full structure of the language based on the data available.
  
  The purpose of this tool is to explore all the possibilities of the existing corpus of INI files. To that end, I ran all 138 of the plaintext files through it, collected all of the symbols, and massaged the results, assuming that the symbols that occurred most frequently are probably core language features. These are all the symbols which occur more than 1000 times among all the scripts :

     6248 false
     5734 looping
     4390 scripts
     3877 layer
     3423 sequentialscript
     3408 setactive
     3360 file
     3257 thescreen
     3239 true
     3008 autoplay
     2914 offset
     2599 transparent
     2441 text
     2361 caption
     2276 add
     2205 ge
     2197 smackanimation
     2196 graphicscript
     2196 graphic
     1977 setstate
     1642 state
     1611 skippable
     1576 desc
     1413 delayscript
     1298 script
     1267 seconds
     1019 rect
  

  About That Compression
  I have sorted out at least these few details of the compression :

  bytes 0-3    "COMP" (a pretty strong sign that this is, in fact, compressed data)
  bytes 4-11   unknown
  bytes 12-15  size of uncompressed data
  bytes 16-19  size of compressed data (filesize - 20)
  bytes 20-    compressed payload
  

  The compression ratios are on the same order of gzip. I was hoping that it was stock zlib data. However, I have been unable to prove this. I wrote a Python script that scrubbed through the first 100 bytes of payload data and tried to get Python’s zlib.decompress to initialize– no luck. It’s frustrating to know that I’ll have to reverse engineer a compression algorithm that deals with just 8 total text files if I want to see this effort through to fruition.

  Update, January 15, 2019
  Some folks expressed interest in trying to sort out the details of the compression format. So I have posted a followup in which I post some samples and go into deeper details about things I have tried :

  Reverse Engineering Clue Chronicles Compression

  The post Parsing The Clue Chronicles first appeared on Breaking Eggs And Making Omelettes.

• Unable to integrate ffmpeg as a native dependency for an Android NDK project

  28 février 2019, par mmark

  I have an Android NDK project which consists entirely of C/C++ code, and it basically processes images without using any external libraries.

  I’m using Android Studio + Gradle NDK Experimental plugin (0.7.0-alpha1).

  Now, I need to integrate ffmpeg as a native library to use it from the C/C++ code in order to decode a H.264 video frame.

  These are the questions I’ve found here regarding this issue :

  Android - Integrating ffmpeg and android-ndk-r9c

  Android NDK w/ ffmpeg library - error running project

  Using FFmpeg native libraries with Android-NDK

  Can not build with prebuilt static libraries using gradle-experimental

  Here is my build.gradle file :

  apply plugin: 'com.android.model.application'
  
  
  
  def ffmpeg_path = file(project(':ffmpeg').projectDir).absolutePath + "/ffmpeg-android"
  
  
  
  model {
  
  
  
      repositories {
  
          libs(PrebuiltLibraries) {
  
              libavcodec {
  
                  headers.srcDir "${ffmpeg_path}/include"
  
                  binaries.withType(StaticLibraryBinary) {
  
                      staticLibraryFile = file("${ffmpeg_path}/${targetPlatform.getName()}/lib/libavcodec.a")
  
                  }
  
              }
  
              libavutil {
  
                  headers.srcDir "${ffmpeg_path}/include"
  
                  binaries.withType(StaticLibraryBinary) {
  
                      staticLibraryFile = file("${ffmpeg_path}/${targetPlatform.getName()}/lib/libavutil.a")
  
                  }
  
              }
  
              libswresample {
  
                  headers.srcDir "${ffmpeg_path}/include"
  
                  binaries.withType(StaticLibraryBinary) {
  
                      staticLibraryFile = file("${ffmpeg_path}/${targetPlatform.getName()}/lib/libswresample.a")
  
                  }
  
              }
  
              libswscale {
  
                  headers.srcDir "${ffmpeg_path}/include"
  
                  binaries.withType(StaticLibraryBinary) {
  
                      staticLibraryFile = file("${ffmpeg_path}/${targetPlatform.getName()}/lib/libswscale.a")
  
                  }
  
              }
  
              libavformat {
  
                  headers.srcDir "${ffmpeg_path}/include"
  
                  binaries.withType(StaticLibraryBinary) {
  
                      staticLibraryFile = file("${ffmpeg_path}/${targetPlatform.getName()}/lib/libavformat.a")
  
                  }
  
              }
  
              libavdevice {
  
                  headers.srcDir "${ffmpeg_path}/include"
  
                  binaries.withType(StaticLibraryBinary) {
  
                      staticLibraryFile = file("${ffmpeg_path}/${targetPlatform.getName()}/lib/libavdevice.a")
  
                  }
  
              }
  
              libavfilter {
  
                  headers.srcDir "${ffmpeg_path}/include"
  
                  binaries.withType(StaticLibraryBinary) {
  
                      staticLibraryFile = file("${ffmpeg_path}/${targetPlatform.getName()}/lib/libavfilter.a")
  
                  }
  
              }
  
              libpostproc {
  
                  headers.srcDir "${ffmpeg_path}/include"
  
                  binaries.withType(StaticLibraryBinary) {
  
                      staticLibraryFile = file("${ffmpeg_path}/${targetPlatform.getName()}/lib/libpostproc.a")
  
                  }
  
              }
  
          }
  
      }
  
  
  
      android {
  
          compileSdkVersion = 23
  
          buildToolsVersion = "23.0.2"
  
  
  
          defaultConfig.with {
  
              applicationId = "com.example.hellojni"
  
              minSdkVersion.apiLevel = 4
  
              targetSdkVersion.apiLevel = 23
  
          }
  
      }
  
  
  
      /*
  
       * native build settings
  
       */
  
      android.ndk {
  
          moduleName = "hello-jni"
  
          platformVersion = 9 //same as minSdkVersion.apiLevel for better compatibility
  
          stl    = "c++_static"
  
          abiFilters.addAll(["armeabi", "armeabi-v7a", "x86"]) //filtering ABIs on the ones Google Play Games library is compatible with.
  
      }
  
      android.buildTypes {
  
          release {
  
              minifyEnabled = false
  
              proguardFiles.add(file('proguard-rules.txt'))
  
          }
  
      }
  
  
  
      android.sources {
  
          main {
  
              jni {
  
                  dependencies {
  
                      library "libavcodec" linkage "static"
  
                      library "libavutil" linkage "static"
  
                      library "libswresample" linkage "static"
  
                      library "libswscale" linkage "static"
  
                      library "libavformat" linkage "static"
  
                      library "libavdevice" linkage "static"
  
                      library "libavfilter" linkage "static"
  
                      library "libpostproc" linkage "static"
  
                  }
  
              }
  
          }
  
      }
  
  
  
  }

  I based on this sample from Google which integrates an external native library.

  This is the ffmpeg project I compiled before importing it into mine :
  https://github.com/WritingMinds/ffmpeg-android

  I’ve added all the .a and .h files and I’m able to import the headers in my C/C++ code, but when I try to compile it I get the following error :

  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(utils.o):utils.c:function try_decode_frame: error: undefined reference to 'avpriv_h264_has_num_reorder_frames'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(utils.o):utils.c:function compute_pkt_fields: error: undefined reference to 'avpriv_h264_has_num_reorder_frames'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(utils.o):utils.c:function parse_packet: error: undefined reference to 'av_parser_parse2'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(utils.o):utils.c:function parse_packet: error: undefined reference to 'av_parser_close'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(utils.o):utils.c:function read_frame_internal: error: undefined reference to 'av_parser_init'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(utils.o):utils.c:function avformat_find_stream_info: error: undefined reference to 'av_parser_init'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(utils.o):utils.c:function avformat_find_stream_info: error: undefined reference to 'avcodec_pix_fmt_to_codec_tag'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(utils.o):utils.c:function avformat_find_stream_info: error: undefined reference to 'avpriv_get_raw_pix_fmt_tags'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(utils.o):utils.c:function ff_stream_add_bitstream_filter: error: undefined reference to 'av_bitstream_filter_init'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(utils.o):utils.c:function ff_stream_add_bitstream_filter: error: undefined reference to 'av_bitstream_filter_close'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(utils.o):utils.c:function av_apply_bitstream_filters: error: undefined reference to 'av_bitstream_filter_filter'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(aadec.o):aadec.c:function aa_read_packet: error: undefined reference to 'av_tea_init'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(aadec.o):aadec.c:function aa_read_packet: error: undefined reference to 'av_tea_crypt'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(aadec.o):aadec.c:function aa_read_header: error: undefined reference to 'av_tea_alloc'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(aadec.o):aadec.c:function aa_read_header: error: undefined reference to 'av_tea_init'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(aadec.o):aadec.c:function aa_read_header: error: undefined reference to 'av_tea_crypt'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(ac3dec.o):ac3dec.c:function ac3_eac3_probe.isra.0: error: undefined reference to 'avpriv_ac3_parse_header'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(adtsenc.o):adtsenc.c:function adts_write_header: error: undefined reference to 'avpriv_mpeg4audio_get_config'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(adtsenc.o):adtsenc.c:function adts_write_header: error: undefined reference to 'avpriv_copy_pce_data'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(asfcrypt.o):asfcrypt.c:function ff_asfcrypt_dec: error: undefined reference to 'av_rc4_alloc'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(asfcrypt.o):asfcrypt.c:function ff_asfcrypt_dec: error: undefined reference to 'av_rc4_init'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(asfcrypt.o):asfcrypt.c:function ff_asfcrypt_dec: error: undefined reference to 'av_rc4_crypt'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(asfcrypt.o):asfcrypt.c:function ff_asfcrypt_dec: error: undefined reference to 'av_rc4_init'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(asfcrypt.o):asfcrypt.c:function ff_asfcrypt_dec: error: undefined reference to 'av_rc4_crypt'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(matroska.o):matroska.c:function ff_mkv_stereo3d_conv: error: undefined reference to 'av_stereo3d_alloc'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(nut.o):nut.c:function ff_nut_add_sp: error: undefined reference to 'av_tree_node_alloc'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(nut.o):nut.c:function ff_nut_add_sp: error: undefined reference to 'av_tree_insert'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(nut.o):nut.c:function ff_nut_free_sp: error: undefined reference to 'av_tree_enumerate'
  
  /Users/marcos/Documents/Android/fdecoder/ffmpeg/ffmpeg-android/armeabi-v7a/lib/libavformat.a(nut.o):nut.c:function ff_nut_free_sp: error: undefined reference to 'av_tree_destroy'
  
  Error:error: ld returned 1 exit status
  
  Error:Execution failed for task ':app:linkHello-jniArmeabi-v7aDebugSharedLibrary'.
  
  > A build operation failed.
  
        Linker failed while linking libhello-jni.so.
  
    See the complete log at: file:///Users/marcos/Documents/Android/fdecoder/app/build/tmp/linkHello-jniArmeabi-v7aDebugSharedLibrary/output.txt
  
  Information:BUILD FAILED
  
  Information:Total time: 14.993 secs
  
  Information:2 errors
  
  Information:0 warnings
  
  Information:See complete output in console

  There’s obviously something I’m not importing properly, but can’t figure exactly what’s missing.

• Dreamcast Operating Systems

  16 septembre 2010, par Multimedia Mike — Sega Dreamcast

  The Sega Dreamcast was famously emblazoned with a logo proudly announcing that it was compatible with Windows CE :

  
  
  

  It’s quite confusing. The console certainly doesn’t boot into some version of Windows to launch games. Apparently, there was a special version of CE developed for the DC and game companies had the option to leverage it. I do recall that some game startup screens would similarly advertise Windows CE.

  Once the homebrew community got ahold of the device, the sky was the limit. I think NetBSD was the first alternative OS to support the Dreamcast. Meanwhile, I have recollections of DC Linux and LinuxDC projects along with more generic Linux-SH and SH-Linux projects.

  
  
  

  DC Evolution hosts a disc image available for download with an unofficial version of DC Linux, assembled by one Adrian O’Grady. I figured out how to burn the disc (burning DC discs is a blog post of its own) and got it working in the console.

  It’s possible to log in directly via the physical keyboard or through a serial terminal provided that you have a coder’s cable. That reminds me– my local Fry’s had a selection of USB-to-serial cables. I think this is another area that is sufficiently commoditized that just about any cable ought to work with Linux out of the box. Or maybe I’m just extrapolating from the experience of having the cheapest cable in the selection (made by io connect) plug and play with Linux.

  
  
  

  Look ! No messy converter box in the middle as in the Belkin case. The reason I went with this cable is that the packaging claimed it was capable of up to 500 Kbits/sec. Most of the cables advertised a max of 115200 bps. I distinctly recall being able to use the DC coder’s cable at 230400 bps a long time ago. Alas, 115200 seems to be the speed limit, even with this new USB cable.

  Anyway, the distribution is based on a 2.4.5 kernel circa 2001. I tried to make PPP work over the serial cable but the kernel doesn’t have support. If you’re interested, here is some basic information about the machine from Linux’s perspective, gleaned from some simple commands. This helps remind us of a simpler time when Linux was able to run comfortably on a computer with 16 MB of RAM.

  Debian GNU/Linux testing/unstable dreamcast ttsc/1
  dreamcast login : root
  
  Linux dreamcast 2.4.5 #27 Thu May 31 07:06:51 JST 2001 sh4 unknown
  Most of the programs included with the Debian GNU/Linux system are
  
  freely redistributable ; the exact distribution terms for each program
  
  are described in the individual files in /usr/share/doc/*/copyright
  Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
  
  permitted by applicable law.
  dreamcast : # uname -a
  
  Linux dreamcast 2.4.5 #27 Thu May 31 07:06:51 JST 2001 sh4 unknown
  dreamcast : # cat /proc/cpuinfo 
  
  cpu family : SH-4
  
  cache size : 8K-byte/16K-byte
  
  bogomips : 199.47
  Machine : dreamcast
  
  CPU clock : 200.00MHz
  
  Bus clock : 100.00MHz
  
  Peripheral module clock : 50.00MHz
  dreamcast : # top -b
   09:14:54 up 14 min,  1 user,  load average : 0.04, 0.03, 0.03
  
  15 processes : 14 sleeping, 1 running, 0 zombie, 0 stopped
  
  CPU states :   1.1% user,   5.8% system,   0.0% nice,  93.1% idle
  
  Mem :     14616K total,    11316K used,     3300K free,     2296K buffers
  
  Swap :        0K total,        0K used,        0K free,     5556K cached
    PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
  
    219 root      18   0  1072 1068   868 R     5.6  7.3   0:00 top
  
      1 root       9   0   596  596   512 S     0.0  4.0   0:01 init
  
      2 root       9   0     0    0     0 SW    0.0  0.0   0:00 keventd
  
      3 root       9   0     0    0     0 SW    0.0  0.0   0:00 kswapd
  
      4 root       9   0     0    0     0 SW    0.0  0.0   0:00 kreclaimd
  
      5 root       9   0     0    0     0 SW    0.0  0.0   0:00 bdflush
  
      6 root       9   0     0    0     0 SW    0.0  0.0   0:00 kupdated
  
      7 root       9   0     0    0     0 SW    0.0  0.0   0:00 kmapled
  
     39 root       9   0   900  900   668 S     0.0  6.1   0:00 devfsd
  
     91 root       8   0   652  652   556 S     0.0  4.4   0:00 pump
  
     96 daemon     9   0   524  524   420 S     0.0  3.5   0:00 portmap
  
    149 root       9   0   944  944   796 S     0.0  6.4   0:00 syslogd
  
    152 root       9   0   604  604   456 S     0.0  4.1   0:00 klogd
  
    187 root       9   0   540  540   456 S     0.0  3.6   0:00 getty
  
    201 root       9   0  1380 1376  1112 S     0.0  9.4   0:01 bash
  

  Note that at this point I had shutdown both gpm and inetd. The rest of the processes, save for bash, are default. The above stats only report about 14 MB of RAM ; where are the other 2 MB ?

  dreamcast : # df -h
  Filesystem            Size  Used Avail Use% Mounted on
  /dev/rd/1             2.0M  560k  1.4M  28% /