Recherche avancée

Sur d’autres sites (7065)

• Parsing The Clue Chronicles

  30 décembre 2018, par Multimedia Mike — Game Hacking

  A long time ago, I procured a 1999 game called Clue Chronicles : Fatal Illusion, based on the classic board game Clue, a.k.a. Cluedo. At the time, I was big into collecting old, unloved PC games so that I could research obscure multimedia formats.

  
  
  

  Surveying the 3 CD-ROMs contained in the box packaging revealed only Smacker (SMK) videos for full motion video which was nothing new to me or the multimedia hacking community at the time. Studying the mix of data formats present on the discs, I found a selection of straightforward formats such as WAV for audio and BMP for still images. I generally find myself more fascinated by how computer games are constructed rather than by playing them, and this mix of files has always triggered a strong “I could implement a new engine for this !” feeling in me, perhaps as part of the ScummVM project which already provides the core infrastructure for reimplementing engines for 2D adventure games.

  Tying all of the assets together is a custom high-level programming language. I have touched on this before in a blog post over a decade ago. The scripts are in a series of files bearing the extension .ini (usually reserved for configuration scripts, but we’ll let that slide). A representative sample of such a script can be found here :

  clue-chronicles-scarlet-1.txt

  What Is This Language ?
  At the time I first analyzed this language, I was still primarily a C/C++-minded programmer, with a decent amount of Perl experience as a high level language, and had just started to explore Python. I assessed this language to be “mildly object oriented with C++-type comments (‘//’) and reliant upon a number of implicit library functions”. Other people saw other properties. When I look at it nowadays, it reminds me a bit more of JavaScript than C++. I think it’s sort of a Rorschach test for programming languages.

  Strangely, I sort of had this fear that I would put a lot of effort into figuring out how to parse out the language only for someone to come along and point out that it’s a well-known yet academic language that already has a great deal of supporting code and libraries available as open source. Google for “spanish dolphins far side comic” for an illustration of the feeling this would leave me with.

  It doesn’t matter in the end. Even if such libraries exist, how easy would they be to integrate into something like ScummVM ? Time to focus on a workable approach to understanding and processing the format.

  Problem Scope
  So I set about to see if I can write a program to parse the language seen in these INI files. Some questions :

  1. How large is the corpus of data that I need to be sure to support ?
  2. What parsing approach should I take ?
  3. What is the exact language format ?
  4. Other hidden challenges ?

  To figure out how large the data corpus is, I counted all of the INI files on all of the discs. There are 138 unique INI files between the 3 discs. However, there are 146 unique INI files after installation. This leads to a hidden challenge described a bit later.

  What parsing approach should I take ? I worried a bit too much that I might not be doing this the “right” way. I’m trying to ignore doubts like this, like how “SQL Shame” blocked me on a task for a little while a few years ago as I concerned myself that I might not be using the purest, most elegant approach to the problem. I know I covered language parsing a lot time ago in university computer science education and there is a lot of academic literature to the matter. But sometimes, you just have to charge in and experiment and prototype and see what falls out. In doing so, I expect to have a better understanding of the problems that need to solved and the right questions to ask, not unlike that time that I wrote a continuous integration system from scratch because I didn’t actually know that “continuous integration” was the keyword I needed.

  Next, what is the exact language format ? I realized that parsing the language isn’t the first and foremost problem here– I need to know exactly what the language is. I need to know what the grammar are keywords are. In essence, I need to reverse engineer the language before I write a proper parser for it. I guess that fits in nicely with the historical aim of this blog (reverse engineering).

  Now, about the hidden challenges– I mentioned that there are 8 more INI files after the game installs itself. Okay, so what’s the big deal ? For some reason, all of the INI files are in plaintext on the CD-ROM but get compressed (apparently, according to file size ratios) when installed to the hard drive. This includes those 8 extra INI files. I thought to look inside the CAB installation archive file on the CD-ROM and the files were there… but all in compressed form. I suspect that one of the files forms the “root” of the program and is the launching point for the game.

  Parsing Approach
  I took a stab at parsing an INI file. My approach was to first perform lexical analysis on the file and create a list of 4 types : symbols, numbers, strings, and language elements ([]{}()=., :). Apparently, this is the kind of thing that Lex/Flex are good at. This prototyping tool is written in Python, but when I port this to ScummVM, it might be useful to call upon the services of Lex/Flex, or another lexical analyzer, for there are many. I have a feeling it will be easier to use better tools when I understand the full structure of the language based on the data available.
  
  The purpose of this tool is to explore all the possibilities of the existing corpus of INI files. To that end, I ran all 138 of the plaintext files through it, collected all of the symbols, and massaged the results, assuming that the symbols that occurred most frequently are probably core language features. These are all the symbols which occur more than 1000 times among all the scripts :

     6248 false
     5734 looping
     4390 scripts
     3877 layer
     3423 sequentialscript
     3408 setactive
     3360 file
     3257 thescreen
     3239 true
     3008 autoplay
     2914 offset
     2599 transparent
     2441 text
     2361 caption
     2276 add
     2205 ge
     2197 smackanimation
     2196 graphicscript
     2196 graphic
     1977 setstate
     1642 state
     1611 skippable
     1576 desc
     1413 delayscript
     1298 script
     1267 seconds
     1019 rect
  

  About That Compression
  I have sorted out at least these few details of the compression :

  bytes 0-3    "COMP" (a pretty strong sign that this is, in fact, compressed data)
  bytes 4-11   unknown
  bytes 12-15  size of uncompressed data
  bytes 16-19  size of compressed data (filesize - 20)
  bytes 20-    compressed payload
  

  The compression ratios are on the same order of gzip. I was hoping that it was stock zlib data. However, I have been unable to prove this. I wrote a Python script that scrubbed through the first 100 bytes of payload data and tried to get Python’s zlib.decompress to initialize– no luck. It’s frustrating to know that I’ll have to reverse engineer a compression algorithm that deals with just 8 total text files if I want to see this effort through to fruition.

  Update, January 15, 2019
  Some folks expressed interest in trying to sort out the details of the compression format. So I have posted a followup in which I post some samples and go into deeper details about things I have tried :

  Reverse Engineering Clue Chronicles Compression

  The post Parsing The Clue Chronicles first appeared on Breaking Eggs And Making Omelettes.

• Statically built FFMPEG binary segmentation fault

  12 février 2020, par stevendesu

  I want to create a custom build of FFMPEG which rips out everything except for the ability to transmux HLS videos to MP4, and I need this build to be 100% static with no external dependencies

  I tried using the following configuration :

  ./configure \
  
      --extra-cflags='-static -static-libstdc++ -static-libgcc' \
  
      --extra-cxxflags='-static -static-libstdc++ -static-libgcc' \
  
      --extra-ldflags='-static -static-libstdc++ -static-libgcc' \
  
      --pkg-config-flags='--static' \
  
      --enable-static \
  
      --disable-shared \
  
      --disable-runtime-cpudetect \
  
      --disable-autodetect \
  
      --disable-ffplay \
  
      --disable-ffprobe \
  
      --disable-doc \
  
      --disable-avdevice \
  
      --disable-swresample \
  
      --disable-swscale \
  
      --disable-postproc \
  
      --disable-pthreads \
  
      --disable-w32threads \
  
      --disable-os2threads \
  
      --enable-network \
  
      --disable-dct \
  
      --disable-dwt \
  
      --disable-error-resilience \
  
      --disable-lsp \
  
      --disable-lzo \
  
      --disable-mdct \
  
      --disable-rdft \
  
      --disable-fft \
  
      --disable-faan \
  
      --disable-pixelutils \
  
      --disable-encoders \
  
      --disable-decoders \
  
      --disable-hwaccels \
  
      --disable-muxers \
  
      --enable-muxer=mov \
  
      --enable-muxer=mp4 \
  
      --disable-demuxers \
  
      --enable-demuxer=hls \
  
      --enable-demuxer=mpegts \
  
      --enable-demuxer=h264 \
  
      --enable-demuxer=aac \
  
      --disable-parsers \
  
      --enable-parser=h264 \
  
      --enable-parser=aac \
  
      --disable-bsfs \
  
      --disable-protocols \
  
      --enable-protocol=tcp \
  
      --enable-protocol=tls \
  
      --enable-protocol=http \
  
      --enable-protocol=https \
  
      --enable-protocol=hls \
  
      --disable-indevs \
  
      --disable-outdevs \
  
      --disable-devices \
  
      --disable-filters \
  
      --disable-alsa \
  
      --disable-appkit \
  
      --disable-avfoundation \
  
      --disable-bzlib \
  
      --disable-coreimage \
  
      --disable-iconv \
  
      --disable-lzma \
  
      --enable-openssl \
  
      --disable-sndio \
  
      --disable-sdl2 \
  
      --disable-securetransport \
  
      --disable-xlib \
  
      --disable-zlib \
  
      --disable-amf \
  
      --disable-audiotoolbox \
  
      --disable-cuda-llvm \
  
      --disable-cuvid \
  
      --disable-d3d11va \
  
      --disable-dxva2 \
  
      --disable-ffnvcodec \
  
      --disable-nvdec \
  
      --disable-nvenc \
  
      --disable-v4l2-m2m \
  
      --disable-vaapi \
  
      --disable-vdpau \
  
      --disable-videotoolbox \
  
      --disable-debug

  This looked about like what I wanted :

  install prefix            /usr/local
  
  source path               .
  
  C compiler                gcc
  
  C library                 glibc
  
  ARCH                      x86 (generic)
  
  big-endian                no
  
  runtime cpu detection     no
  
  standalone assembly       yes
  
  x86 assembler             nasm
  
  MMX enabled               yes
  
  MMXEXT enabled            yes
  
  3DNow! enabled            yes
  
  3DNow! extended enabled   yes
  
  SSE enabled               yes
  
  SSSE3 enabled             yes
  
  AESNI enabled             yes
  
  AVX enabled               yes
  
  AVX2 enabled              yes
  
  AVX-512 enabled           yes
  
  XOP enabled               yes
  
  FMA3 enabled              yes
  
  FMA4 enabled              yes
  
  i686 features enabled     yes
  
  CMOV is fast              yes
  
  EBX available             yes
  
  EBP available             yes
  
  debug symbols             no
  
  strip symbols             yes
  
  optimize for size         no
  
  optimizations             yes
  
  static                    yes
  
  shared                    no
  
  postprocessing support    no
  
  network support           yes
  
  threading support         no
  
  safe bitstream reader     yes
  
  texi2html enabled         no
  
  perl enabled              yes
  
  pod2man enabled           yes
  
  makeinfo enabled          no
  
  makeinfo supports HTML    no
  
  
  
  External libraries:
  
  openssl
  
  
  
  External libraries providing hardware acceleration:
  
  
  
  Libraries:
  
  avcodec                 avfilter                avformat                avutil
  
  
  
  Programs:
  
  ffmpeg
  
  
  
  Enabled decoders:
  
  
  
  Enabled encoders:
  
  
  
  Enabled hwaccels:
  
  
  
  Enabled parsers:
  
  aac                     h264
  
  
  
  Enabled demuxers:
  
  aac                     h264                    hls                     mpegts
  
  
  
  Enabled muxers:
  
  mov                     mp4
  
  
  
  Enabled protocols:
  
  hls                     http                    https                   tcp                     tls
  
  
  
  Enabled filters:
  
  aformat                 anull                   atrim                   format                  hflip                   null                    transpose               trim                    vflip
  
  
  
  Enabled bsfs:
  
  null
  
  
  
  Enabled indevs:
  
  
  
  Enabled outdevs:
  
  
  
  License: LGPL version 2.1 or later

  It included several filters which I won’t ever need or use, but these filters are pulled in automatically if you don’t specify --disable-avfilter, and specifying --disable-avfilter prevents the ffmpeg binary from being produced. So I’m stuck with those.

  Using these parameters and then running make, I received a binary that was about 5.9 MB in size and looked right :

  $> ldd ffmpeg
  
          not a dynamic executable

  But when I try to run it :

  $> ./ffmpeg -version
  
  Segmentation fault

  Using valgrind to try and inspect the cause of the segmentation fault :

  $> valgrind ./ffmpeg -version
  
  .... lots of stuff ...
  
  ==61362== Jump to the invalid address stated on the next line
  
  ==61362==    at 0x0: ???
  
  ==61362==    by 0x70BB1B: ??? (in /src/FFmpeg/ffmpeg)
  
  ==61362==    by 0x70B2E6: ??? (in /src/FFmpeg/ffmpeg)
  
  ==61362==    by 0x4033F9: ??? (in /src/FFmpeg/ffmpeg)
  
  ==61362==    by 0x1FFF000677: ???
  
  ==61362==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
  
  ==61362==
  
  ==61362==
  
  ==61362== Process terminating with default action of signal 11 (SIGSEGV)
  
  ==61362==  Bad permissions for mapped region at address 0x0
  
  ==61362==    at 0x0: ???
  
  ==61362==    by 0x70BB1B: ??? (in /src/FFmpeg/ffmpeg)
  
  ==61362==    by 0x70B2E6: ??? (in /src/FFmpeg/ffmpeg)
  
  ==61362==    by 0x4033F9: ??? (in /src/FFmpeg/ffmpeg)
  
  ==61362==    by 0x1FFF000677: ???
  
  ==61362==
  
  ==61362== HEAP SUMMARY:
  
  ==61362==     in use at exit: 0 bytes in 0 blocks
  
  ==61362==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
  
  ==61362==
  
  ==61362== All heap blocks were freed -- no leaks are possible
  
  ==61362==
  
  ==61362== For counts of detected and suppressed errors, rerun with: -v
  
  ==61362== Use --track-origins=yes to see where uninitialised values come from
  
  ==61362== ERROR SUMMARY: 93 errors from 90 contexts (suppressed: 0 from 0)
  
  Segmentation fault

  Attempting to access memory at location 0x0 sounds like trying to follow a null pointer. But I’m not sure how to fix this.

  gdb backtrace

  When I first ran gdb ./ffmpeg gdb immediately gave me a segmentation fault and I wasn’t kicked into the gdb REPL, so I couldn’t investigate

  After rebuilding ffmpeg I was able to get in this time :

  $> gdb ./ffmpeg
  
  
  
  GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
  
  Copyright (C) 2018 Free Software Foundation, Inc.
  
  License GPLv3+: GNU GPL version 3 or later /gnu.org/licenses/gpl.html>
  
  This is free software: you are free to change and redistribute it.
  
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  
  and "show warranty" for details.
  
  This GDB was configured as "x86_64-linux-gnu".
  
  Type "show configuration" for configuration details.
  
  For bug reporting instructions, please see:
  
  /www.gnu.org/software/gdb/bugs/>.
  
  Find the GDB manual and other documentation resources online at:
  
  /www.gnu.org/software/gdb/documentation/>.
  
  For help, type "help".
  
  Type "apropos word" to search for commands related to "word"...
  
  Reading symbols from ffmpeg...done.
  
  (gdb) r
  
  Starting program: /src/FFmpeg/ffmpeg
  
  warning: Error disabling address space randomization: Operation not permitted
  
  
  
  Program received signal SIGSEGV, Segmentation fault.
  
  0x0000000000000000 in ?? ()
  
  (gdb) bt
  
  #0  0x0000000000000000 in ?? ()
  
  #1  0x0000000000f9a8d5 in __register_frame_info_bases.part.6 ()
  
  #2  0x00000000004445fd in frame_dummy ()
  
  #3  0x0000000000000001 in ?? ()
  
  #4  0x0000000000ebd20c in __libc_csu_init ()
  
  #5  0x0000000000ebc9d7 in __libc_start_main ()
  
  #6  0x000000000044451a in _start ()
  
  (gdb)

  I tried grep’ing the code base for __register_frame_info_bases and found nothing. So I’m not really sure where to go from here

  A fix, but not an explanation

  By randomly removing configuration parameters and rebuilding I discovered that --disable-pthreads was causing the segmentation fault. When I remove this, ffmpeg runs just fine

  I don’t know why this is the case, though. Why would they make it possible to remove something that you need to run ?

• Python - why do I get this ffprobe error ?

  4 juillet 2022, par gasper101

  I am on mac and am working on a project in which my program shoud convert a .mp3 file into a .wav file and than get text from that file (basic speech recognition). For converting a .mp3 file into a .wav file I am using ffmpeg library. I have installed both ffmpeg and ffprobe libraryies using pip and than using homebrew so I have them installed in a folder.

  


  This is my code :

  


  import speech_recognition
from pydub import AudioSegment
from os import path


#get text from original voice file------------------------------------------------- 

sound = AudioSegment.from_mp3("/path/to/file.mp3")
sound.export("/output/path/file.wav", format="wav")

file_audio = speech_recognition.AudioFile('/output/path/file.wav')

r = speech_recognition.Recognizer()
with file_audio as source:
    audio_text = r.record(source)

word = r.recognize_google(audio_text)


  


  And this is the error i get :
FileNotFoundError : [Errno 2] No such file or directory : 'ffprobe'

  


  Full error :
Warning (from warnings module) :
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pydub/utils.py", line 170
warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)
RuntimeWarning : Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work

  


  Warning (from warnings module) :
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pydub/utils.py", line 198
warn("Couldn't find ffprobe or avprobe - defaulting to ffprobe, but may not work", RuntimeWarning)

  


  RuntimeWarning : Couldn't find ffprobe or avprobe - defaulting to ffprobe, but may not work
Traceback (most recent call last) :
File "/Users/marcus/Downloads/pronaunciation_checker/pronaunciation_check.py", line 12, in 
sound = AudioSegment.from_mp3("/Users/marcus/Desktop/think.mp3")
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pydub/audio_segment.py", line 796, in from_mp3
return cls.from_file(file, 'mp3', parameters=parameters)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pydub/audio_segment.py", line 728, in from_file
info = mediainfo_json(orig_file, read_ahead_limit=read_ahead_limit)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pydub/utils.py", line 274, in mediainfo_json
res = Popen(command, stdin=stdin_parameter, stdout=PIPE, stderr=PIPE)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError : [Errno 2] No such file or directory : 'ffprobe'