Recherche avancée

Sur d’autres sites (4474)

• Investigating Steam for Linux

  1er mars 2013, par Multimedia Mike — Game Hacking

  Valve recently released the final, public version of their Steam client for Linux, and the Linux world rejoiced. At least, it probably did. The announcement was 2 weeks ago on Valentine’s Day and I had other things on my mind, so I missed any fanfare. When framed in this manner, the announcement timing becomes suspect– it’s as though Linux enthusiasts would have plenty of time that day or something.

  
  
  

  Taming the Frontier
  Speculation about a Linux Steam client had been kicking around for nearly as long as Steam has existed. However, sometime last year, the rumors became more substantive.

  I naturally wondered how to port something like Steam to Linux. I have some experience with trying to make a necessarily binary-only program that runs on Linux. I’m fairly well-versed in the assorted technical challenges that one might face when attempting such a feat. Because of this, whenever I hear rumors that a company might be entertaining the notion of porting a major piece of proprietary software to Linux, my instinctive reflex is, “What ?! Why, you fools ?! Save yourselves !”

  At least, that’s how it used to be. The proposal of developing a proprietary binary for Linux has been rendered considerably less insane by a few developments, for example :

  1. The rise of Ubuntu Linux as a quasi de facto standard for desktop Linux computing
  2. The increasing homogeneity in personal desktop computing technology

  What I would like to know is how the Steam client runs on Linux. Does it rely on any libraries being present on the system ? Or does it bring its own ? The latter is a trick that proprietary programs can use– transport all of the shared libraries that the main program binary depends upon, install them someplace out of the way on the filesystem, probably in /opt, and then make the main program a shell script which sets a preload path to rely on the known quantity libraries instead of the copies already on the system.

  Downloading and Installing the Client
  For this exercise, I installed x86_64 desktop Ubuntu 12.04 Linux on a l33t gaming rig that was totally top of the line about 5 years ago, and that someone didn’t want anymore and handed down to me recently. So it should be ideal for this project.

  At first, I was blown away– the Linux client is in a .deb package that is less than 2 MB large. I unpacked the steam.deb file and found a bunch of support libraries — mostly X11 and standard C/C++ runtimes. Just as I suspected. Still, I can’t believe how small the thing is. However, my amazement quickly abated when I actually ran Steam and saw this :

  
  
  

  So it turns out steam.db is just the installer program which immediately proceeds to download an additional 160+ MB of data. So there’s actually a lot more information to possibly sift through.
  
  Another component of the installation is to basically run a big ‘apt-get install’ command to make sure a bunch of required packages are installed :

  
  
  

  After all these installation steps, the client was ready to run. However, whenever I tried to do so, I got this dialog which would cause Steam to close when the dialog was dismissed.

  
  
  

  Not a huge deal ; later NVIDIA drivers are fairly straightforward to install on Ubuntu Linux. After a few minutes of downloading, installing and restarting X, Steam ran with minimal complaint (it still had some issue regarding the video drivers but didn’t seem to consider it a deal-breaker).

  Using Steam on Linux

  So here’s Steam running on Linux :

  
  
  

  If you have experience with using Steam on Windows or Mac, you might observe that it looks exactly the same. I don’t have a very expansive library of games (I only started using Steam because purchasing a few computer components a few years ago entitled me to some free Steam downloads of some of the games on the list in the screenshot). I didn’t really expect any of the games to have Linux versions yet, but it turns out that the indie darling FTL : Faster Than Light has been ported to Linux. FTL was a much-heralded Kickstarter success story and sounded like something I wanted to support. I purchased this from Steam shortly after its release last year and was able to download the Linux version at no additional cost with a single click.

  It runs natively on Linux (note the Ubuntu desktop window decorations) :

  
  
  

  You might notice from the main Steam client that, despite purchasing FTL about a 1/2 year ago and starting it up at least a 1/2 dozen times, I haven’t really invested a whole lot of time into it. I only managed to get about 2 minutes further this time :

  
  
  

  What can I say ? This game just bores me to tears. It’s frustrating because I know that this is one of the cool games that all real gamers are supposed to like, but I practically catch myself nodding off every time I try to run through the tutorial. It’s strange to think that I’ve invested far more time into games that offer considerably less stimulation. That’s probably because I had far more free time compared to gaming options during those times.

  But that’s neither here nor there. We’ll file this under “games that aren’t for me.” I’m glad that people like FTL and a little indie underdog has met with such success. And I’m pleased that Steam on Linux works. It’s native and the games are also native, which is all quite laudable (there was speculation that everything would just be running on top of a Wine layer).

  Deeper Analysis
  So I set out wondering how Steam was able to create a proprietary program that would satisfy a large enough cross-section of Linux users (i.e., on different platforms and distros). Answer : well, they didn’t, per the stated requirements. The installation is only tuned to work on Ubuntu 12.04. However, it works on both 32- and 64-bit platforms, the only 2 desktop CPU platforms that matter these days (unless ARM somehow makes inroads on the desktop). The Steam client is quite clearly an x86_32 binary– look at the terminal screenshot above and observe that it’s downloading all :i386 support libraries.

  The file /usr/bin/steam isn’t a binary but a launcher shell script (something you’ll also see if you investigate /usr/bin/firefox on a Linux system). Here’s an interesting tidbit :

  function detect_platform()
  
    # Maybe be smarter someday
    # Right now this is the only platform we have a bootstrap for, so hard-code it.
    echo ubuntu12_32
  
  

  I wager that it’s possible to get Steam running on other distributions, it probably just takes a little more effort (assuming that Steam doesn’t put too much effort into thwarting such attempts).

  As for the FTL game, it comes with binaries and libraries for both x86_32 and x86_64. So, good work to the dev team for creating and testing both versions. FTL also distributes versions of the libraries it expects to work with.

  I suspect that the Steam client overall is largely a WWW rendering engine underneath the covers. That would help explain how Valve is able to achieve such a consistent look and feel, not only across OS platforms, but also through a web browser. When I browse the Steam store through Google Chrome, it looks and feels exactly like the native desktop client. When I first thought of how someone could port Steam to Linux, I immediately wondered about how they would do the UI.

  A little Googling for “steam uses webkit” (just a hunch) confirms my hypothesis.

• Reverse Engineering Italian Literature

  1er juillet 2014, par Multimedia Mike — Reverse Engineering

  Some time ago, Diego “Flameeyes” Pettenò tried his hand at reverse engineering a set of really old CD-ROMs containing even older Italian literature. The goal of this RE endeavor would be to extract the useful literature along with any structural metadata (chapters, etc.) and convert it to a more open format suitable for publication at, e.g., Project Gutenberg or Archive.org.

  Unfortunately, the structure of the data thwarted the more simplistic analysis attempts (like inspecting for blocks of textual data). This will require deeper RE techniques. Further frustrating the effort, however, is the fact that the binaries that implement the reading program are written for the now-archaic Windows 3.1 operating system.

  In pursuit of this RE goal, I recently thought of a way to glean more intelligence using DOSBox.

  Prior Work
  There are 6 discs in the full set (distributed along with 6 sequential issues of a print magazine named L’Espresso). Analysis of the contents of the various discs reveals that many of the files are the same on each disc. It was straightforward to identify the set of files which are unique on each disc. This set of files all end with the extension “LZn”, where n = 1..6 depending on the disc number. Further, the root directory of each disc has a file indicating the sequence number (1..6) of the CD. Obviously, these are the interesting targets.

  The LZ file extensions stand out to an individual skilled in the art of compression– could it be a variation of the venerable LZ compression ? That’s actually unlikely because LZ — also seen as LIZ — stands for Letteratura Italiana Zanichelli (Zanichelli’s Italian Literature).

  The Unix ‘file’ command was of limited utility, unable to plausibly identify any of the files.

  Progress was stalled.

  Saying Hello To An Old Frenemy
  I have been showing this screenshot to younger coworkers to see if any of them recognize it :

  
  
  

  Not a single one has seen it before. Senior computer citizen status : Confirmed.

  I recently watched an Ancient DOS Games video about Windows 3.1 games. This episode showed Windows 3.1 running under DOSBox. I had heard this was possible but that it took a little work to get running. I had a hunch that someone else had probably already done the hard stuff so I took to the BitTorrent networks and quickly found a download that had the goods ready to go– a directory of Windows 3.1 files that just had to be dropped into a DOSBox directory and they would be ready to run.

  Aside : Running OS software procured from a BitTorrent network ? Isn’t that an insane security nightmare ? I’m not too worried since it effectively runs under a sandboxed virtual machine, courtesy of DOSBox. I suppose there’s the risk of trojan’d OS software infecting binaries that eventually leave the sandbox.

  Using DOSBox Like ‘strace’
  strace is a tool available on some Unix systems, including Linux, which is able to monitor the system calls that a program makes. In reverse engineering contexts, it can be useful to monitor an opaque, binary program to see the names of the files it opens and how many bytes it reads, and from which locations. I have written examples of this before (wow, almost 10 years ago to the day ; now I feel old for the second time in this post).

  Here’s the pitch : Make DOSBox perform as strace in order to serve as a platform for reverse engineering Windows 3.1 applications. I formed a mental model about how DOSBox operates — abstracted file system classes with methods for opening and reading files — and then jumped into the source code. Sure enough, the code was exactly as I suspected and a few strategic print statements gave me the data I was looking for.

  Eventually, I even took to running DOSBox under the GNU Debugger (GDB). This hasn’t proven especially useful yet, but it has led to an absurd level of nesting :

  
  
  

  The target application runs under Windows 3.1, which is running under DOSBox, which is running under GDB. This led to a crazy situation in which DOSBox had the mouse focus when a GDB breakpoint was triggered. At this point, DOSBox had all desktop input focus and couldn’t surrender it because it wasn’t running. I had no way to interact with the Linux desktop and had to reboot the computer. The next time, I took care to only use the keyboard to navigate the application and trigger the breakpoint and not allow DOSBox to consume the mouse focus.

  New Intelligence
  
  By instrumenting the local file class (virtual HD files) and the ISO file class (CD-ROM files), I was able to watch which programs and dynamic libraries are loaded and which data files the code cares about. I was able to narrow down the fact that the most interesting programs are called LEGGENDO.EXE (‘reading’) and LEGGENDA.EXE (‘legend’ ; this has been a great Italian lesson as well as RE puzzle). The first calls the latter, which displays this view of the data we are trying to get at :

  
  
  

  When first run, the program takes an interest in a file called DBBIBLIO (‘database library’, I suspect) :

  === Read(’LIZ98\DBBIBLIO.LZ1’) : req 337 bytes ; read 337 bytes from pos 0x0
  === Read(’LIZ98\DBBIBLIO.LZ1’) : req 337 bytes ; read 337 bytes from pos 0x151
  === Read(’LIZ98\DBBIBLIO.LZ1’) : req 337 bytes ; read 337 bytes from pos 0x2A2
  [...]
  

  While we were unable to sort out all of the data files in our cursory investigation, a few things were obvious. The structure of this file looked to contain 336-byte records. Turns out I was off by 1– the records are actually 337 bytes each. The count of records read from disc is equal to the number of items shown in the UI.

  Next, the program is interested in a few more files :

  *** isoFile() : ’DEPOSITO\BLOKCTC.LZ1’, offset 0x27D6000, 2911488 bytes large
  === Read(’DEPOSITO\BLOKCTC.LZ1’) : req 96 bytes ; read 96 bytes from pos 0x0
  *** isoFile() : ’DEPOSITO\BLOKCTX0.LZ1’, offset 0x2A9D000, 17152 bytes large
  === Read(’DEPOSITO\BLOKCTX0.LZ1’) : req 128 bytes ; read 128 bytes from pos 0x0
  === Seek(’DEPOSITO\BLOKCTX0.LZ1’) : seek 384 (0x180) bytes, type 0
  === Read(’DEPOSITO\BLOKCTX0.LZ1’) : req 256 bytes ; read 256 bytes from pos 0x180
  === Seek(’DEPOSITO\BLOKCTC.LZ1’) : seek 1152 (0x480) bytes, type 0
  === Read(’DEPOSITO\BLOKCTC.LZ1’) : req 32 bytes ; read 32 bytes from pos 0x480
  === Read(’DEPOSITO\BLOKCTC.LZ1’) : req 1504 bytes ; read 1504 bytes from pos 0x4A0
  [...]

  Eventually, it becomes obvious that BLOKCTC has the juicy meat. There are 32-byte records followed by variable-length encoded text sections. Since there is no text to be found in these files, the text is either compressed, encrypted, or both. Some rough counting (the program seems to disable copy/paste, which thwarts more precise counting), indicates that the text size is larger than the data chunks being read from disc, so compression seems likely. Encryption isn’t out of the question (especially since the program deems it necessary to disable copy and pasting of this public domain literary data), and if it’s in use, that means the key is being read from one of these files.

  Blocked On Disassembly
  So I’m a bit blocked right now. I know exactly where the data lives, but it’s clear that I need to reverse engineer some binary code. The big problem is that I have no idea how to disassemble Windows 3.1 binaries. These are NE-type executable files. Disassemblers abound for MZ files (MS-DOS executables) and PE files (executables for Windows 95 and beyond). NE files get no respect. It’s difficult (but not impossible) to even find data about the format anymore, and details are incomplete. It should be noted, however, the DOSBox-as-strace method described here lends insight into how Windows 3.1 processes NE-type EXEs. You can’t get any more authoritative than that.

  So far, I have tried the freeware version of IDA Pro. Unfortunately, I haven’t been able to get the program to work on my Windows machine for a long time. Even if I could, I can’t find any evidence that it actually supports NE files (the free version specifically mentions MZ and PE, but does not mention NE or LE).

  I found an old copy of Borland’s beloved Turbo Assembler and Debugger package. It has Turbo Debugger for Windows, both regular and 32-bit versions. Unfortunately, the normal version just hangs Windows 3.1 in DOSBox. The 32-bit Turbo Debugger loads just fine but can’t load the NE file.

  I’ve also wondered if DOSBox contains any advanced features for trapping program execution and disassembling. I haven’t looked too deeply into this yet.

  Future Work
  NE files seem to be the executable format that time forgot. I have a crazy brainstorm about repacking NE files as MZ executables so that they could be taken apart with an MZ disassembler. But this will take some experimenting.

  If anyone else has any ideas about ripping open these binaries, I would appreciate hearing them.

  And I guess I shouldn’t be too surprised to learn that all the literature in this corpus is already freely available and easily downloadable anyway. But you shouldn’t be too surprised if that doesn’t discourage me from trying to crack the format that’s keeping this particular copy of the data locked up.

• What’s new in Piwik 2.16.0 ?

  1er février 2016, par Matthieu Aubry — About, Development, Uncategorized

  In this blog post we showcase what is new in Piwik 2.16.0 and why you should upgrade your Piwik to this latest release !

  Piwik 2.16.0 is currently in Release Candidate and you can already use it by following these simple instructions. Piwik 2.16.0 will be officially released in a few days.

  To see the list of all 250+ changes, read the Piwik 2.16.0 changelog.

  Long Term Support for Piwik 2.X

  Piwik 2.16.0 is our Long Term Support version for Piwik 2. Learn more about Piwik LTS.

  New : Custom Dimensions feature

  With Custom Dimensions you can assign any custom data to your visitors or actions (like pages, events, …) and then visualize the reports of how many visits, conversions, pageviews, etc. there were for each Custom Dimension. They are similar to Custom Variables see the comparison of Custom Dimensions and Custom Variables.

  The new Custom Dimensions plugin is available on the Piwik Marketplace.

  Learn more about creating, tracking, reporting, managing, segmenting custom dimensions in the Custom Dimensions user guide.

  

  

  

  View the list of all visitors who reached a specific goal

  All Goal reports now include a link below the Goal report summary, that lets you see in one click all users that have converted any of your Goals :

  

  Events reports : option to view all columns : min, max and avg values

  By default in Actions > Events report, the columns displayed are “Total events” (Total number of events) and “Total value” (the sum of event values). It is now possible to click on the footer icon to view more detailed columns about event values : minimum event value, maximum event value, and average event value.

  

  Allow zoom to country in realtime visitor map

  In the real time map in Piwik (which displays your users activity appearing in real time !), it is now possible to zoom on a given country :

  

  Export all Records : new ‘all’ option in the Row limit selector

  It is now possible to export all of the data in your reports directly from the user interface ! Select ‘all’ in the row limit selector before exporting your data reports :

  

  New themes on the Marketplace !

  Feel like a change of colors ? Try out the new community submitted Piwik Themes on the Marketplace ! Learn how to install a new theme in Piwik (in only a few clicks).

  

  Let Super User view and edit segments created by other users

  As a Super User of Piwik, it can be very useful to view all Custom Segments created by all users. Starting in Piwik 2.16.0 any Super User can now view all Segments for a given website in the Segment selector :

  • Segments that are shared with All users will now appear below a section “Shared with you :” (a segment can be shared to all users by any admin user by selecting “This segment is visible to All Users” in the Segment editor)
  • Segments that are not shared with anyone are now also visible under the section “Visible to you because you have Super User access :”

  

  New segment ‘deviceBrand’ to segment by Device Brand

  You can now segment by device brands such as Apple, Samsung, LG, Google, Nokia, Sony, Lenovo, Alcatel, etc. The complete list of device brands you can use is listed on the Segmentation developer guide.

  

  New segment ‘actionType’ to view only particular Actions types (pagesviews, downloads, outlinks, events…)

  Use the new “Action type” segment to view only particular actions such as : pageviews, contents, sitesearches, events, outlinks and downloads.

  

  New segment : ‘actionUrl’ to view any actions that matches a given URL

  Use the new “Action URL” segment to view any action that matches a given URL, whether they are Pageviews, Site searches, Contents, Downloads or Events.

  

  New segment operators : “Starts with” and “Ends with”

  The new segment operators “Starts with” and “Ends with” can come handy when creating Custom Segments, and complement well the existing segment operators : Contains, Does not contain, Equals, Not equals, Greater than or equal to, Less than or equal to.

  

  Learn more about Segmentation in our user guides or in the Segmentation developer guide.

  Making Piwik more accessible

  Web accessibility refers to the inclusive practice of removing barriers that prevent interaction with, or access to applications or websites, by people with disabilities. When sites are correctly designed, developed and edited, all users have equal access to information and functionality.

  In Piwik 2.16.0, and with the help of high school students from the Catalyst Academy, we have made several improvements that make Piwik more accessible :

  • Menu now include the ARIA metadata (to make menus easier to navigate and use)
  • Page titles and various key tooltips have been improved (to give a better context to the data)
  • Keyboard access has been much improvement and Piwik is starting to be usable with the keyboard ! (this is very useful as many users are not able to use a mouse, and they should be able to view Piwik reports with the keyboard only)
  • ‘Skip to main content’ link (when pressing the TAB key) now skips the cursor to the page content.
  • Search bar was further improved : try it by pressing the ‘f’ key (useful to open any Piwik report or page without using the mouse)

  New user preference : display times either as 24h format or AM/PM format

  Users can now select whether times should be displayed in 24h format (16:20) or in AM/PM format (4:20pm)

  

  Other admins and Super Users are not visible to users with ‘admin’ permissions anymore

  Piwik 2.16.0 just became even more respectful of your privacy : any user you created and assigned an “admin” permission will not be able to view all other usernames anymore. This change was requested by many Piwik users who rightfully preferred not to disclose all usernames to any ‘admin’ user.

  As a result of this change, when a user with ‘admin’ permission (admin user) wants to give additional permission to another user (target user), the admin user will now need to know the target username and enter it in the Manage Users interface :

  

  New Config file viewer

  A new menu entry Administration > Diagnostics > Config file, lets Super Users view all config values in the UI :

  This new screen serves several purposes :

  • help Piwik administrators discover and learn about all the possible config file settings that Piwik offers.
  • let Super Users and Admins easily view any overridden config.ini.php INI setting without having to open the config file on the server.

  

  New report ‘Personal settings > Custom Variables’ displays custom variable usage

  A new report visible to Super Users lets you see how your Custom Variables slots are being used, which ones are still unused, and how many times a given custom variable name was used.

  

  Learn more about Custom Variables.

  Improvements to Heartbeat feature to accurately count time spent on the last page

  The heartbeat feature which lets you accurately track how long do visitors spend on your website, has been improved in 2.16.0. If you would like to measure how long do people spend on your pages (even if they don’t interact with your website or app), then consider using the heartbeat feature. Learn more in our guide Accurately measure the time spent on each page.

  

  New AnonymousPiwikUsageMeasurement plugin
  

  AnonymousPiwikUsageMeasurement plugin was created for those who would like to measure how their Piwik services is being used. This plugin also sends anonymised tracking data to our demo-anonymous.piwik.org which will help us learn how your Piwik is used and how we could improve the application. Learn more about AnonymousPiwikUsageMeasurement plugin.

  

  Show a warning when Piwik is used as a SuperUser via HTTP

  It is highly recommended to use HTTPS (SSL) for your Piwik server. If you are logged in as a Super User and viewing any Administration pages, you will now see a warning notification message at the top of the screen inviting you to use HTTPS :

  

  MySQL 5.7+ support

  The awesome Piwik community has helped us identify several issues with MySQL 5.7 support and Piwik is now fully compatible with the latest MySQL version 5.7+. Piwik is officially compatible with MySQL >= 4.1 and up to Mysql 5.7 ! Piwik is also compatible with MariaDB.

  

  Better detection of new devices and robots

  Our Device Detector library can now detect many new types of devices : smart TVs, new tablets and mobile phones, and more. New bots were added to our list of bots to keep your analytics report meaningful.

  Learn more : What Device Detector is able to detect, About Device Detector.

  New Referrer Spammers

  Every week we keep updating our community-list of referrer spammers. More than 50 new spammers were added since the last Piwik release !

  Note that the list of referrer spammer is automatically updated every week in your Piwik, so you need to wait at most one week to benefit from the very latest list.

  Learn more about Stopping Referrer Spam in Analytics reports.

  

  Ability to support new SMS providers to send Scheduled Reports by text message

  When you create scheduled reports in Piwik, you can specify one or several email addresses (to send reports by email) and you can also configure phone numbers (so your reports are sent to your phone via text message). So far we only support one SMS Provider “Clockwork SMS” but it is now possible for anyone to create a very simple plugin that adds a new SMS Provider to Piwik.

  A new SMS Provider plugin was released on the Marketplace : FreeMobileMessaging. If you have a favorite SMS Provider, we encourage you to create a new plugin for it !

  Piwik PRO mentions

  A few new mentions of Piwik PRO were added into Piwik to let users know that they can get Professional Help, and additional services and products. For example a new widget was added to the default dashboard. If you provide your Piwik service to your own clients, maybe you would like to remove the Piwik PRO widget and other ads, see this FAQ : How do I remove from Piwik the links to “Piwik PRO” or to “Professional Services” ?

  Stability, Reliability and Performance

  Dozens of other bugs were fixed (including several Major bugs) and many small improvements are included in this release. All these changes make Piwik more stable and reliable, as well as easier to troubleshoot and maintain. We are proud to be able to offer Piwik 2.16.0 as our Long Term Support release !

  If you have any feedback, questions, suggestion, or bug to report, please join our Piwik Community Forums.

  We hope you will love this 2.16.0 release as much as we loved making it.

  Happy Analytics !