Recherche avancée

Sur d’autres sites (7200)

• Parsing The Clue Chronicles

  30 décembre 2018, par Multimedia Mike — Game Hacking

  A long time ago, I procured a 1999 game called Clue Chronicles : Fatal Illusion, based on the classic board game Clue, a.k.a. Cluedo. At the time, I was big into collecting old, unloved PC games so that I could research obscure multimedia formats.

  
  
  

  Surveying the 3 CD-ROMs contained in the box packaging revealed only Smacker (SMK) videos for full motion video which was nothing new to me or the multimedia hacking community at the time. Studying the mix of data formats present on the discs, I found a selection of straightforward formats such as WAV for audio and BMP for still images. I generally find myself more fascinated by how computer games are constructed rather than by playing them, and this mix of files has always triggered a strong “I could implement a new engine for this !” feeling in me, perhaps as part of the ScummVM project which already provides the core infrastructure for reimplementing engines for 2D adventure games.

  Tying all of the assets together is a custom high-level programming language. I have touched on this before in a blog post over a decade ago. The scripts are in a series of files bearing the extension .ini (usually reserved for configuration scripts, but we’ll let that slide). A representative sample of such a script can be found here :

  clue-chronicles-scarlet-1.txt

  What Is This Language ?
  At the time I first analyzed this language, I was still primarily a C/C++-minded programmer, with a decent amount of Perl experience as a high level language, and had just started to explore Python. I assessed this language to be “mildly object oriented with C++-type comments (‘//’) and reliant upon a number of implicit library functions”. Other people saw other properties. When I look at it nowadays, it reminds me a bit more of JavaScript than C++. I think it’s sort of a Rorschach test for programming languages.

  Strangely, I sort of had this fear that I would put a lot of effort into figuring out how to parse out the language only for someone to come along and point out that it’s a well-known yet academic language that already has a great deal of supporting code and libraries available as open source. Google for “spanish dolphins far side comic” for an illustration of the feeling this would leave me with.

  It doesn’t matter in the end. Even if such libraries exist, how easy would they be to integrate into something like ScummVM ? Time to focus on a workable approach to understanding and processing the format.

  Problem Scope
  So I set about to see if I can write a program to parse the language seen in these INI files. Some questions :

  1. How large is the corpus of data that I need to be sure to support ?
  2. What parsing approach should I take ?
  3. What is the exact language format ?
  4. Other hidden challenges ?

  To figure out how large the data corpus is, I counted all of the INI files on all of the discs. There are 138 unique INI files between the 3 discs. However, there are 146 unique INI files after installation. This leads to a hidden challenge described a bit later.

  What parsing approach should I take ? I worried a bit too much that I might not be doing this the “right” way. I’m trying to ignore doubts like this, like how “SQL Shame” blocked me on a task for a little while a few years ago as I concerned myself that I might not be using the purest, most elegant approach to the problem. I know I covered language parsing a lot time ago in university computer science education and there is a lot of academic literature to the matter. But sometimes, you just have to charge in and experiment and prototype and see what falls out. In doing so, I expect to have a better understanding of the problems that need to solved and the right questions to ask, not unlike that time that I wrote a continuous integration system from scratch because I didn’t actually know that “continuous integration” was the keyword I needed.

  Next, what is the exact language format ? I realized that parsing the language isn’t the first and foremost problem here– I need to know exactly what the language is. I need to know what the grammar are keywords are. In essence, I need to reverse engineer the language before I write a proper parser for it. I guess that fits in nicely with the historical aim of this blog (reverse engineering).

  Now, about the hidden challenges– I mentioned that there are 8 more INI files after the game installs itself. Okay, so what’s the big deal ? For some reason, all of the INI files are in plaintext on the CD-ROM but get compressed (apparently, according to file size ratios) when installed to the hard drive. This includes those 8 extra INI files. I thought to look inside the CAB installation archive file on the CD-ROM and the files were there… but all in compressed form. I suspect that one of the files forms the “root” of the program and is the launching point for the game.

  Parsing Approach
  I took a stab at parsing an INI file. My approach was to first perform lexical analysis on the file and create a list of 4 types : symbols, numbers, strings, and language elements ([]{}()=., :). Apparently, this is the kind of thing that Lex/Flex are good at. This prototyping tool is written in Python, but when I port this to ScummVM, it might be useful to call upon the services of Lex/Flex, or another lexical analyzer, for there are many. I have a feeling it will be easier to use better tools when I understand the full structure of the language based on the data available.
  
  The purpose of this tool is to explore all the possibilities of the existing corpus of INI files. To that end, I ran all 138 of the plaintext files through it, collected all of the symbols, and massaged the results, assuming that the symbols that occurred most frequently are probably core language features. These are all the symbols which occur more than 1000 times among all the scripts :

     6248 false
     5734 looping
     4390 scripts
     3877 layer
     3423 sequentialscript
     3408 setactive
     3360 file
     3257 thescreen
     3239 true
     3008 autoplay
     2914 offset
     2599 transparent
     2441 text
     2361 caption
     2276 add
     2205 ge
     2197 smackanimation
     2196 graphicscript
     2196 graphic
     1977 setstate
     1642 state
     1611 skippable
     1576 desc
     1413 delayscript
     1298 script
     1267 seconds
     1019 rect
  

  About That Compression
  I have sorted out at least these few details of the compression :

  bytes 0-3    "COMP" (a pretty strong sign that this is, in fact, compressed data)
  bytes 4-11   unknown
  bytes 12-15  size of uncompressed data
  bytes 16-19  size of compressed data (filesize - 20)
  bytes 20-    compressed payload
  

  The compression ratios are on the same order of gzip. I was hoping that it was stock zlib data. However, I have been unable to prove this. I wrote a Python script that scrubbed through the first 100 bytes of payload data and tried to get Python’s zlib.decompress to initialize– no luck. It’s frustrating to know that I’ll have to reverse engineer a compression algorithm that deals with just 8 total text files if I want to see this effort through to fruition.

  Update, January 15, 2019
  Some folks expressed interest in trying to sort out the details of the compression format. So I have posted a followup in which I post some samples and go into deeper details about things I have tried :

  Reverse Engineering Clue Chronicles Compression

  The post Parsing The Clue Chronicles first appeared on Breaking Eggs And Making Omelettes.

• Why Matomo is a serious alternative to Google Analytics 360

  12 décembre 2018, par Jake Thornton — Marketing
  There’s no doubt about it, the free version of Google Analytics offers great value when it comes to making data-driven decisions for your business. But as your business starts to grow, so does the need for a more powerful web analytics tool.

  Why would I need to use a different web analytics tool ? It’s because Google Analytics (free version) is very limited when it comes to meeting the needs of a fast growing business whose website plays a pivotal role in converting its customers.

  This is where the Google Analytics 360 suite comes in, which is designed to meet the needs of businesses looking to get more accurate and insightful metrics.

  So what’s holding a growing business back from using Google Analytics 360 ?

  While GA360 sounds like a great option when upgrading your web analytics platform, we have found there are three core reasons holding businesses back from taking the leap :

  • Businesses can’t bear to swallow the US$150,000+ price tag (per year !) that comes with upgrading
  • Businesses can’t rely on GA360 to give them all the insights they need
  • Businesses want more control and ownership of their data

  Thankfully there are (only a few) alternatives and as the leading open-source alternative to Google Analytics, we hope to share insights on why Matomo Analytics can be the perfect solution for anyone at this crossroads in their web analytics journey.

  First, what does Google Analytics 360 offer that Google Analytics (free) doesn’t ?

  There’s no doubt about it, the GA360 suite is designed for larger sized businesses with demanding data limits, big budgets to use across the Google Marketing Platform (Google Adwords, DoubleClick etc.) and to get more advanced reporting visualisations and options.
  

  Data Sampling

  Data sampling is the elephant in the room when it comes to comparing GA360 with the freemium version. This is an entire article in its own right but at a basic level, Google Analytics samples your data (makes assumptions based on patterns) once the number of traffic visiting your website reaches a certain limit.

  Google Analytics provides the following information :

  Ad-hoc queries of your data are subject to the following general thresholds for sampling :

  Analytics Standard : 500k sessions at the property level for the date range you are using

  Analytics 360 : 100M sessions at the view level for the date range you are using

  In short, sampled data means inaccurate data. This is why as businesses grow, GA360 becomes a more attractive prospect because there’s no point making data-driven business decisions based on inaccurate data. This is a key weapon Google uses when selling to large businesses, however, this may not seem as concerning if you’re a small business within the sampled data range. For small businesses though, make sure you know the full extent of how this can affect your metrics, for example, your ecommerce data could be sampled, hence your GA reporting not matching your CRM/Ecommerce store data.

  Benefit of using Matomo : There is no data sampling anywhere in Matomo Analytics, that’s why we say 100% Accurate Data reporting across all plans.

  

  Integration with the Google Marketing Platform

  Yes ok, we’ll admit it, GA does a great job at integrating seamlessly with its own products like Google Ads, Google Optimize etc. with a touch of Salesforce integration ; while GA360 takes this to another level compared to it’s freemium version (integration with Google Search 360, Google Display & Video 360 etc.)

  But… what about non-Google advertising platforms ? Well with Google being a dominant leader as a search engine, web browser, email provider, social media channel ; sometimes Google needs to keep its best interests at heart.

  Google is an online advertising giant and a bonus of Google Search 360 is that you can integrate your Bing Ads, Baidu and Yahoo Japan Search campaigns but that’s about it when it comes to integrations from its direct competitors. 

  Benefit of using Matomo : No biased treatment. You can integrate your Google, Yahoo and Bing search consoles for accurate search engine reporting, and in early 2019, Matomo will be releasing a Google Ads, Bing Ads and Facebook Ads Manager integration feature.

  Roll-Up Reporting
  

  Roll-up reporting lets you combine multiple accounts and properties into one view. This is a great benefit when upgrading from GA freemium to GA360. For example, if you’re a digital agency with multiple clients or you manage multiple websites under the one account, the roll-up reporting feature is wonderful when you need to combine data and reporting, instantly.

  Benefit of using Matomo : Matomo’s got this covered ! Roll-up reporting is available in the Matomo Business package (starting at $29 per month) for cloud hosting or you can purchase as a Premium Feature for On-Premise starting at $99 per year.

  Staying in full control of your data

  Who would have thought that one of biggest reasons people choose Matomo isn’t because of anything that leads to a higher ROI, but for the fact that users want more control of their data.
  

  Matomo’s philosophy around data ownership is simple, you own your data, no one else. If you choose to host Matomo Analytics On-Premise then you are in complete control because your data is stored on your own servers where no one can gain access to it in whichever country you choose.

  So what about when you cloud host Matomo ? For users who don’t have the technical knowledge to host Matomo On-Premise, you can still have 100% data ownership and fully respect your user’s privacy when choosing to host Matomo Analytics through our cloud service.

  The difference between cloud hosting Matomo Analytics vs Google Analytics is that when you choose Matomo, we acknowledge you own the data and we have no right to access it. This means we can’t on-sell it to third-parties, we can’t claim ownership of it, you can export your data at anytime (how awesome is that !) and you can migrate between cloud hosting and hosting on-premise for ultimate flexibility whenever you want.

  Matomo also prides itself in allowing its users to be GDPR compliant with ease with a powerful GDPR Manager.

  Businesses can’t rely on Google Analytics 360 to give them all the insights they need

  Unlike Google Analytics 360, Matomo blends its Premium Web Analytics platform with Conversion Optimization features to allow its users to fully evaluate the user-experience on your website.

  Matomo is designed to be a complete analytics platform, meaning you have everything you need all in the one place which gives you greater insights and better business outcomes.

  
  These features include :

  Premium Web Analytics – You can still (accurately) measure all the basic metrics you love and are familiar with in Google Analytics like Location, Referrer traffic, Multi Attribution, Campaign Tracking and Ecommerce etc.

  Conversion Optimization – Eliminate the need for multiple analytics tools to get what Google Analytics doesn’t offer. These features include Heatmaps, Session Recordings, Form Analytics and more – giving you the best chance possible to convert more traffic by evaluating the user-experience.

  By having one tool for all your features you can integrate metrics, have one single view for all your data and it’s easy to use.

  Enhanced SEO – Get more insights into the performance of your search campaigns with unbiased search engine reporting, keyword ranking positions, integration with multiple search consoles and crawling stats. Google Analytics offers limited features to help with your SEO campaigns and only integrates with Google products.

  Visitor Profiles – Get a detailed life-time evaluation of every user who visits your website.

  Tag Manager – A powerful open-source Tag Manager tool to embed your third-party marketing tags. By being open-source and with our commitment to giving you 100% data ownership, you can always ensure you are in full control.

  Full Matomo vs Google Analytics 360 Comparison Table

  Just putting it out there ...

  Google leads the market with its freemium tool which offers great insights for businesses (fyi – Matomo has a forever free analytics tool too !), but when it comes to upgrading to get accurate reporting (kind of a big deal), owning your own data (a huge deal !) and having a complete range of features to excel ROI for your business, Matomo Analytics is often a preferred option to the Google Analytics 360 suite.

  Matomo is designed to be easy to use, is fully flexible and gives users full peace of mind by respecting user privacy. Want to learn more about the benefits of Matomo ?

• Anomalie #3750 : Itérateur JSON, erreur de compilation lorsque la source contient &

  8 octobre 2018, par placido .

  J’avais fait le ticket au départ, notamment parce que le message de debug de compilation partait complètement en vrille, ce qui semble trahir un cas de remontée d’erreur pas bien géré.
  Pour le reste on a qu’à dire que cette page fera office de précis de documentation pour les urls de ressources JSON avec ’&’.

  Donc oui, on peut sans doute fermer.