Recherche avancée

Sur d’autres sites (7653)

• Reverse Engineering Italian Literature

  1er juillet 2014, par Multimedia Mike — Reverse Engineering

  Some time ago, Diego “Flameeyes” Pettenò tried his hand at reverse engineering a set of really old CD-ROMs containing even older Italian literature. The goal of this RE endeavor would be to extract the useful literature along with any structural metadata (chapters, etc.) and convert it to a more open format suitable for publication at, e.g., Project Gutenberg or Archive.org.

  Unfortunately, the structure of the data thwarted the more simplistic analysis attempts (like inspecting for blocks of textual data). This will require deeper RE techniques. Further frustrating the effort, however, is the fact that the binaries that implement the reading program are written for the now-archaic Windows 3.1 operating system.

  In pursuit of this RE goal, I recently thought of a way to glean more intelligence using DOSBox.

  Prior Work
  There are 6 discs in the full set (distributed along with 6 sequential issues of a print magazine named L’Espresso). Analysis of the contents of the various discs reveals that many of the files are the same on each disc. It was straightforward to identify the set of files which are unique on each disc. This set of files all end with the extension “LZn”, where n = 1..6 depending on the disc number. Further, the root directory of each disc has a file indicating the sequence number (1..6) of the CD. Obviously, these are the interesting targets.

  The LZ file extensions stand out to an individual skilled in the art of compression– could it be a variation of the venerable LZ compression ? That’s actually unlikely because LZ — also seen as LIZ — stands for Letteratura Italiana Zanichelli (Zanichelli’s Italian Literature).

  The Unix ‘file’ command was of limited utility, unable to plausibly identify any of the files.

  Progress was stalled.

  Saying Hello To An Old Frenemy
  I have been showing this screenshot to younger coworkers to see if any of them recognize it :

  
  
  

  Not a single one has seen it before. Senior computer citizen status : Confirmed.

  I recently watched an Ancient DOS Games video about Windows 3.1 games. This episode showed Windows 3.1 running under DOSBox. I had heard this was possible but that it took a little work to get running. I had a hunch that someone else had probably already done the hard stuff so I took to the BitTorrent networks and quickly found a download that had the goods ready to go– a directory of Windows 3.1 files that just had to be dropped into a DOSBox directory and they would be ready to run.

  Aside : Running OS software procured from a BitTorrent network ? Isn’t that an insane security nightmare ? I’m not too worried since it effectively runs under a sandboxed virtual machine, courtesy of DOSBox. I suppose there’s the risk of trojan’d OS software infecting binaries that eventually leave the sandbox.

  Using DOSBox Like ‘strace’
  strace is a tool available on some Unix systems, including Linux, which is able to monitor the system calls that a program makes. In reverse engineering contexts, it can be useful to monitor an opaque, binary program to see the names of the files it opens and how many bytes it reads, and from which locations. I have written examples of this before (wow, almost 10 years ago to the day ; now I feel old for the second time in this post).

  Here’s the pitch : Make DOSBox perform as strace in order to serve as a platform for reverse engineering Windows 3.1 applications. I formed a mental model about how DOSBox operates — abstracted file system classes with methods for opening and reading files — and then jumped into the source code. Sure enough, the code was exactly as I suspected and a few strategic print statements gave me the data I was looking for.

  Eventually, I even took to running DOSBox under the GNU Debugger (GDB). This hasn’t proven especially useful yet, but it has led to an absurd level of nesting :

  
  
  

  The target application runs under Windows 3.1, which is running under DOSBox, which is running under GDB. This led to a crazy situation in which DOSBox had the mouse focus when a GDB breakpoint was triggered. At this point, DOSBox had all desktop input focus and couldn’t surrender it because it wasn’t running. I had no way to interact with the Linux desktop and had to reboot the computer. The next time, I took care to only use the keyboard to navigate the application and trigger the breakpoint and not allow DOSBox to consume the mouse focus.

  New Intelligence
  
  By instrumenting the local file class (virtual HD files) and the ISO file class (CD-ROM files), I was able to watch which programs and dynamic libraries are loaded and which data files the code cares about. I was able to narrow down the fact that the most interesting programs are called LEGGENDO.EXE (‘reading’) and LEGGENDA.EXE (‘legend’ ; this has been a great Italian lesson as well as RE puzzle). The first calls the latter, which displays this view of the data we are trying to get at :

  
  
  

  When first run, the program takes an interest in a file called DBBIBLIO (‘database library’, I suspect) :

  === Read(’LIZ98\DBBIBLIO.LZ1’) : req 337 bytes ; read 337 bytes from pos 0x0
  === Read(’LIZ98\DBBIBLIO.LZ1’) : req 337 bytes ; read 337 bytes from pos 0x151
  === Read(’LIZ98\DBBIBLIO.LZ1’) : req 337 bytes ; read 337 bytes from pos 0x2A2
  [...]
  

  While we were unable to sort out all of the data files in our cursory investigation, a few things were obvious. The structure of this file looked to contain 336-byte records. Turns out I was off by 1– the records are actually 337 bytes each. The count of records read from disc is equal to the number of items shown in the UI.

  Next, the program is interested in a few more files :

  *** isoFile() : ’DEPOSITO\BLOKCTC.LZ1’, offset 0x27D6000, 2911488 bytes large
  === Read(’DEPOSITO\BLOKCTC.LZ1’) : req 96 bytes ; read 96 bytes from pos 0x0
  *** isoFile() : ’DEPOSITO\BLOKCTX0.LZ1’, offset 0x2A9D000, 17152 bytes large
  === Read(’DEPOSITO\BLOKCTX0.LZ1’) : req 128 bytes ; read 128 bytes from pos 0x0
  === Seek(’DEPOSITO\BLOKCTX0.LZ1’) : seek 384 (0x180) bytes, type 0
  === Read(’DEPOSITO\BLOKCTX0.LZ1’) : req 256 bytes ; read 256 bytes from pos 0x180
  === Seek(’DEPOSITO\BLOKCTC.LZ1’) : seek 1152 (0x480) bytes, type 0
  === Read(’DEPOSITO\BLOKCTC.LZ1’) : req 32 bytes ; read 32 bytes from pos 0x480
  === Read(’DEPOSITO\BLOKCTC.LZ1’) : req 1504 bytes ; read 1504 bytes from pos 0x4A0
  [...]

  Eventually, it becomes obvious that BLOKCTC has the juicy meat. There are 32-byte records followed by variable-length encoded text sections. Since there is no text to be found in these files, the text is either compressed, encrypted, or both. Some rough counting (the program seems to disable copy/paste, which thwarts more precise counting), indicates that the text size is larger than the data chunks being read from disc, so compression seems likely. Encryption isn’t out of the question (especially since the program deems it necessary to disable copy and pasting of this public domain literary data), and if it’s in use, that means the key is being read from one of these files.

  Blocked On Disassembly
  So I’m a bit blocked right now. I know exactly where the data lives, but it’s clear that I need to reverse engineer some binary code. The big problem is that I have no idea how to disassemble Windows 3.1 binaries. These are NE-type executable files. Disassemblers abound for MZ files (MS-DOS executables) and PE files (executables for Windows 95 and beyond). NE files get no respect. It’s difficult (but not impossible) to even find data about the format anymore, and details are incomplete. It should be noted, however, the DOSBox-as-strace method described here lends insight into how Windows 3.1 processes NE-type EXEs. You can’t get any more authoritative than that.

  So far, I have tried the freeware version of IDA Pro. Unfortunately, I haven’t been able to get the program to work on my Windows machine for a long time. Even if I could, I can’t find any evidence that it actually supports NE files (the free version specifically mentions MZ and PE, but does not mention NE or LE).

  I found an old copy of Borland’s beloved Turbo Assembler and Debugger package. It has Turbo Debugger for Windows, both regular and 32-bit versions. Unfortunately, the normal version just hangs Windows 3.1 in DOSBox. The 32-bit Turbo Debugger loads just fine but can’t load the NE file.

  I’ve also wondered if DOSBox contains any advanced features for trapping program execution and disassembling. I haven’t looked too deeply into this yet.

  Future Work
  NE files seem to be the executable format that time forgot. I have a crazy brainstorm about repacking NE files as MZ executables so that they could be taken apart with an MZ disassembler. But this will take some experimenting.

  If anyone else has any ideas about ripping open these binaries, I would appreciate hearing them.

  And I guess I shouldn’t be too surprised to learn that all the literature in this corpus is already freely available and easily downloadable anyway. But you shouldn’t be too surprised if that doesn’t discourage me from trying to crack the format that’s keeping this particular copy of the data locked up.

• A Guide to GDPR Sensitive Personal Data

  13 mai 2024, par Erin

  The General Data Protection Regulation (GDPR) is one of the world’s most stringent data protection laws. It provides a legal framework for collection and processing of the personal data of EU individuals.

  The GDPR distinguishes between “special categories of personal data” (also referred to as “sensitive”) and other personal data and imposes stricter requirements on collection and processing of sensitive data. Understanding these differences will help your company comply with the requirements and avoid heavy penalties.

  In this article, we’ll explain what personal data is considered “sensitive” according to the GDPR. We’ll also examine how a web analytics solution like Matomo can help you maintain compliance.

  What is sensitive personal data ?

  The following categories of data are treated as sensitive :

  1. 1. Personal data revealing :
        • Racial or ethnic origin ;
        • Political opinions ;
        • Religious or philosophical beliefs ;
        • Trade union membership ;
     2. Genetic and biometric data ;
     3. Data concerning a person’s :
        • Health ; or
        • Sex life or sexual orientation.

  

  Sensitive vs. non-sensitive personal data : What’s the difference ?

  While both categories include information about an individual, sensitive data is seen as more private, or requiring a greater protection. 

  Sensitive data often carries a higher degree of risk and harm to the data subject, if the data is exposed. For example, a data breach exposing health records could lead to discrimination for the individuals involved. An insurance company could use the information to increase premiums or deny coverage. 

  In contrast, personal data like name or gender is considered less sensitive because it doesn’t carry the same degree of harm as sensitive data. 

  Unauthorised access to someone’s name alone is less likely to harm them or infringe on their fundamental rights and freedoms than an unauthorised access to their health records or biometric data. Note that financial information (e.g. credit card details) does not fall into the special categories of data.

  

  Legality of processing

  Under the GDPR, both sensitive and nonsensitive personal data are protected. However, the rules and conditions for processing sensitive data are more stringent.

  Article 6 deals with processing of non-sensitive data and it states that processing is lawful if one of the six lawful bases for processing applies. 

  In contrast, Art. 9 of the GDPR states that processing of sensitive data is prohibited as a rule, but provides ten exceptions. 

  It is important to note that the lawful bases in Art. 6 are not the same as exceptions in Art. 9. For example, while performance of a contract or legitimate interest of the controller are a lawful basis for processing non-sensitive personal data, they are not included as an exception in Art. 9. What follows is that controllers are not permitted to process sensitive data on the basis of contract or legitimate interest. 

  The exceptions where processing of sensitive personal data is permitted (subject to additional requirements) are : 

  • Explicit consent : The individual has given explicit consent to processing their sensitive personal data for specified purpose(s), except where an EU member state prohibits such consent. See below for more information about explicit consent. 
  • Employment, social security or social protection : Processing sensitive data is necessary to perform tasks under employment, social security or social protection law.
  • Vital interests : Processing sensitive data is necessary to protect the interests of a data subject or if the individual is physically or legally incapable of consenting. 
  • Non-for-profit bodies : Foundations, associations or nonprofits with a political, philosophical, religious or trade union aim may process the sensitive data of their members or those they are in regular contact with, in connection with their purposes (and no disclosure of the data is permitted outside the organisation, without the data subject’s consent).
  • Made public : In some cases, it may be permissible to process the sensitive data of a data subject if the individual has already made it public and accessible. 
  • Legal claims : Processing sensitive data is necessary to establish, exercise or defend legal claims, including legal or in court proceedings.
  • Public interest : Processing is necessary for reasons of substantial public interest, like preventing unlawful acts or protecting the public.
  • Health or social care : Processing special category data is necessary for : preventative or occupational medicine, providing health and social care, medical diagnosis or managing healthcare systems.
  • Public health : It is permissible to process sensitive data for public health reasons, like protecting against cross-border threats to health or ensuring the safety of medicinal products or medical devices. 
  • Archiving, research and statistics : You may process sensitive data if it’s done for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes.

  In addition, you must adhere to all data handling requirements set by the GDPR.

  Important : Note that for any data sent that you are processing, you always need to identify a lawful basis under Art. 6. In addition, if the data sent contains sensitive data, you must comply with Art. 9.

  Explicit consent

  While consent is a valid lawful basis for processing non-sensitive personal data, controllers are permitted to process sensitive data only with an “explicit consent” of the data subject.

  The GDPR does not define “explicit” consent, but it is accepted that it must meet all Art. 7 conditions for consent, at a higher threshold. To be “explicit” a consent requires a clear statement (oral or written) of the data subject. Consent inferred from the data subject’s actions does not meet the threshold. 

  The controller must retain records of the explicit consent and provide appropriate consent withdrawal method to allow the data subject to exercise their rights.

  Examples of compliant and non-compliant sensitive data processing

  Here are examples of when you can and can’t process sensitive data :

  • When you can process sensitive data : A doctor logs sensitive data about a patient, including their name, symptoms and medicine prescribed. The hospital can process this data to provide appropriate medical care to their patients. An IoT device and software manufacturer processes their customers’ health data based on explicit consent of each customer. 
  • When you can’t process sensitive data : One example is when you don’t have explicit consent from a data subject. Another is when there’s no lawful basis for processing it or you are collecting personal data you simply do not need. For example, you don’t need your customer’s ethnic origin to fulfil an online order.

  Other implications of processing sensitive data

  If you process sensitive data, especially on a large scale, GDPR imposes additional requirements, such as having Data Privacy Impact Assessments, appointing Data Protection Officers and EU Representatives, if you are a controller based outside the EU.

  Penalties for GDPR non-compliance

  Mishandling sensitive data (or processing it when you’re not allowed to) can result in huge penalties. There are two tiers of GDPR fines :

  • €10 million or 2% of a company’s annual revenue for less severe infringements
  • €20 million or 4% of a company’s annual revenue for more severe infringements

  In the first half of 2023 alone, fines imposed in the EU due to GDPR violations exceeded €1.6 billion, up from €73 million in 2019.

  Examples of high-profile violations in the last few years include :

  • Amazon : The Luxembourg National Commission fined the retail giant with a massive $887 million fine in 2021 for not processing personal data per the GDPR. 
  • Google : The National Data Protection Commission (CNIL) fined Google €50 million for not getting proper consent to display personalised ads.
  • H&M : The Hamburg Commissioner for Data Protection and Freedom of Information hit the multinational clothing company with a €35.3 million fine in 2020 for unlawfully gathering and storing employees’ data in its service centre.

  One of the criteria that affects the severity of a fine is “data category” — the type of personal data being processed. Companies need to take extra precautions with sensitive data, or they risk receiving more severe penalties.

  What’s more, GDPR violations can negatively affect your brand’s reputation and cause you to lose business opportunities from consumers concerned about your data practices. 76% of consumers indicated they wouldn’t buy from companies they don’t trust with their personal data.

  Organisations should lay out their data practices in simple terms and make this information easily accessible so customers know how their data is being handled.

  Get started with GDPR-compliant web analytics

  The GDPR offers a framework for securing and protecting personal data. But it also distinguishes between sensitive and non-sensitive data. Understanding these differences and applying the lawful basis for processing this data type will help ensure compliance.

  Looking for a GDPR-compliant web analytics solution ?

  At Matomo, we take data privacy seriously. 

  Our platform ensures 100% data ownership, putting you in complete control of your data. Unlike other web analytics solutions, your data remains solely yours and isn’t sold or auctioned off to advertisers. 

  Additionally, with Matomo, you can be confident in the accuracy of the insights you receive, as we provide reliable, unsampled data.

  Matomo also fully complies with GDPR and other data privacy laws like CCPA, LGPD and more.

  Start your 21-day free trial today ; no credit card required. 

  Disclaimer

  We are not lawyers and don’t claim to be. The information provided here is to help give an introduction to GDPR. We encourage every business and website to take data privacy seriously and discuss these issues with your lawyer if you have any concerns.

  Try Matomo for Free

  21 day free trial. No credit card required.

  
  

  Your email address
  

  Your website address
  

  Your Analytics subdomain will be .matomo.cloud (change)

  Choose your analytics subdomain
  

  .matomo.cloud

  I accept the terms and conditions and data processing agreement (DPA)

  
  Your information will be used to create an account on our cloud service. For more information please consult our privacy policy.
  
  
  

  Start improving your websites now

  

• IJG swings again, and misses

  1er février 2010, par Mans — Multimedia

  Earlier this month the IJG unleashed version 8 of its ubiquitous libjpeg library on the world. Eager to try out the “major breakthrough in image coding technology” promised in the README file accompanying v7, I downloaded the release. A glance at the README file suggests something major indeed is afoot :

  Version 8.0 is the first release of a new generation JPEG standard to overcome the limitations of the original JPEG specification.

  The text also hints at the existence of a document detailing these marvellous new features, and a Google search later a copy has found its way onto my monitor. As I read, however, my state of mind shifts from an initial excited curiosity, through bewilderment and disbelief, finally arriving at pure merriment.

  Already on the first page it becomes clear no new JPEG standard in fact exists. All we have is an unsolicited proposal sent to the ITU-T by members of the IJG. Realising that even the most brilliant of inventions must start off as mere proposals, I carry on reading. The summary informs me that I am about to witness the introduction of three extensions to the T.81 JPEG format :

  1. An alternative coefficient scan sequence for DCT coefficient serialization
  2. A SmartScale extension in the Start-Of-Scan (SOS) marker segment
  3. A Frame Offset definition in or in addition to the Start-Of-Frame (SOF) marker segment

  Together these three extensions will, it is promised, “bring DCT based JPEG back to the forefront of state-of-the-art image coding technologies.”

  Alternative scan

  The first of the proposed extensions introduces an alternative DCT coefficient scan sequence to be used in place of the zigzag scan employed in most block transform based codecs.

  

  Alternative scan sequence

  The advantage of this scan would be that combined with the existing progressive mode, it simplifies decoding of an initial low-resolution image which is enhanced through subsequent passes. The author of the document calls this scheme “image-pyramid/hierarchical multi-resolution coding.” It is not immediately obvious to me how this constitutes even a small advance in image coding technology.

  At this point I am beginning to suspect that our friend from the IJG has been trapped in a half-world between interlaced GIF images transmitted down noisy phone lines and today’s inferno of SVC, MVC, and other buzzwords.

  (Not so) SmartScale

  Disguised behind this camel-cased moniker we encounter a method which, we are told, will provide better image quality at high compression ratios. The author has combined two well-known (to us) properties in a (to him) clever way.

  The first property concerns the perceived impact of different types of distortion in an image. When encoding with JPEG, as the quantiser is increased, the decoded image becomes ever more blocky. At a certain point, a better subjective visual quality can be achieved by down-sampling the image before encoding it, thus allowing a lower quantiser to be used. If the decoded image is scaled back up to the original size, the unpleasant, blocky appearance is replaced with a smooth blur.

  The second property belongs to the DCT where, as we all know, the top-left (DC) coefficient is the average of the entire block, its neighbours represent the lowest frequency components etc. A top-left-aligned subset of the coefficient block thus represents a low-resolution version of the full block in the spatial domain.

  In his flash of genius, our hero came up with the idea of using the DCT for down-scaling the image. Unfortunately, he appears to possess precious little knowledge of sampling theory and human visual perception. Any block-based resampling will inevitably produce sharp artefacts along the block edges. The human visual system is particularly sensitive to sharp edges, so this is one of the most unwanted types of distortion in an encoded image.

  Despite the obvious flaws in this approach, I decided to give it a try. After all, the software is already written, allowing downscaling by factors of 8/8..16.

  Using a 1280×720 test image, I encoded it with each of the nine scaling options, from unity to half size, each time adjusting the quality parameter for a final encoded file size of no more than 200000 bytes. The following table presents the encoded file size, the libjpeg quality parameter used, and the SSIM metric for each of the images.

  Scale	Size	Quality	SSIM
  8/8	198462	59	0.940
  8/9	196337	70	0.936
  8/10	196133	79	0.934
  8/11	197179	84	0.927
  8/12	193872	89	0.915
  8/13	197153	92	0.914
  8/14	188334	94	0.899
  8/15	198911	96	0.886
  8/16	197190	97	0.869

  Although the smaller images allowed a higher quality setting to be used, the SSIM value drops significantly. Numbers may of course be misleading, but the images below speak for themselves. These are cut-outs from the full image, the original on the left, unscaled JPEG-compressed in the middle, and JPEG with 8/16 scaling to the right.

  

  

  Looking at these images, I do not need to hesitate before picking the JPEG variant I prefer.

  Frame offset

  The third and final extension proposed is quite simple and also quite pointless : a top-left cropping to be applied to the decoded image. The alleged utility of this feature would be to enable lossless cropping of a JPEG image. In a typical image workflow, however, JPEG is only used for the final published version, so the need for this feature appears quite far-fetched.

  The grand finale

  Throughout the text, the author makes references to “the fundamental DCT property for image representation.” In his own words :

  This property was found by the author during implementation of the new DCT scaling features and is after his belief one of the most important discoveries in digital image coding after releasing the JPEG standard in 1992.

  The secret is to be revealed in an annex to the main text. This annex quotes in full a post by the author to the comp.dsp Usenet group in a thread with the subject why DCT. Reading the entire thread proves quite amusing. A few excerpts follow.

  The actual reason is much simpler, and therefore apparently very difficult to recognize by complicated-thinking people.

  Here is the explanation :

  What are people doing when they have a bunch of images and want a quick preview ? They use thumbnails ! What are thumbnails ? Thumbnails are small downscaled versions of the original image ! If you want more details of the image, you can zoom in stepwise by enlarging (upscaling) the image.

  So with proper understanding of the fundamental DCT property, the MPEG folks could make their videos more scalable, but, as in the case of JPEG, they are unable to recognize this simple but basic property, unfortunately, and pursue rather inferior approaches in actual developments.

  These are just phrases, and they don’t explain anything. But this is typical for the current state in this field : The relevant people ignore and deny the true reasons, and thus they turn in a circle and no progress is being made.

  However, there are dark forces in action today which ignore and deny any fruitful advances in this field. That is the reason that we didn’t see any progress in JPEG for more than a decade, and as long as those forces dominate, we will see more confusion and less enlightenment. The truth is always simple, and the DCT *is* simple, but this fact is suppressed by established people who don’t want to lose their dubious position.

  I believe a trip to the Total Perspective Vortex may be in order. Perhaps his tin-foil hat will save him.