Recherche avancée

Sur d’autres sites (14961)

• How to complete your privacy policy with Matomo analytics under GDPR

  25 avril 2018, par InnoCraft

  Important note : this blog post has been written by digital analysts, not lawyers. The purpose of this article is to show you how to complete your existing privacy policy by adding the parts related to Matomo in order to comply with GDPR. This work comes from our interpretation of the UK privacy commission : ICO. It cannot be considered as professional legal advice. So as GDPR, this information is subject to change. We strongly advise you to have a look at the different privacy authorities in order to have up to date information. This blog post contains public sector information licensed under the Open Government Licence v3.0.

  Neither the GDPR official text or ICO are mentioning the words ‘privacy policy’. They use the words ‘privacy notice’ instead. As explained within our previous blog post about “How to write a privacy notice for Matomo”, the key concepts of privacy information are transparency and accessibility which are making the privacy notice very long.

  As a result, we prefer splitting the privacy notice into two parts :

  • Privacy notice : straight to the point information about how personal data is processed at the time of the data collection. This is the subject of the our previous blog post.
  • Privacy policy : a web page explaining in detail all the personal data you are processing and how visitors/users can exercise their rights. This is the blog post you are reading.

  Writing/updating your privacy policy page can be one of the most challenging task under GDPR.

  In order to make this mission less complicated, we have designed a template which you can use to complete the privacy policy part that concerns Matomo.

  Which information should your privacy policy include ?

  ICO is giving a clear checklist about what a privacy policy has to contain when the data is obtained from the data subject :

  1. Identity and contact details of the controller and where applicable, the controller’s representative and the data protection officer.
  2. Purpose of the processing and the legal basis for the processing.
  3. The legitimate interests of the controller or third party, where applicable.
  4. Any recipient or categories of recipients of the personal data.
  5. Details of transfers to third country and safeguards.
  6. Retention period or criteria used to determine the retention period.
  7. The existence of each of data subject’s rights.
  8. The right to withdraw consent at any time, where relevant.
  9. The right to lodge a complaint with a supervisory authority.
  10. Whether the provision of personal data part of a statutory or contractual requirement or obligation and possible consequences of failing to provide the personal data.
  11. The existence of automated decision-making, including profiling and information about how decisions are made, the significance and the consequences.

  So in order to use Matomo with due respect to GDPR you need to answer each of those points within your privacy policy.

  Matomo’s privacy policy template

  You will find below some examples to each point requested by GDPR. Those answers are just guidelines, they are not perfect, feel free to copy/paste them according to your needs.

  Note that this template needs to be tweaked according to the lawful basis you choose.

  1 – About Matomo

  Note : this part should describe the data controller instead, which is your company. But as you may already have included this part within your existing privacy policy, we prefer here to introduce what is Matomo.

  Matomo is an open source web analytics platform. A web analytics platform is used by a website owner in order to measure, collect, analyse and report visitors data for purposes of understanding and optimizing their website. If you would like to see what Matomo looks like, you can access a demo version at : https://demo.matomo.org.

  2 – Purpose of the processing

  Matomo is used to analyse the behaviour of the website visitors to identify potential pitfalls ; not found pages, search engine indexing issues, which contents are the most appreciated… Once the data is processed (number of visitors reaching a not found pages, viewing only one page…), Matomo is generating reports for website owners to take action, for example changing the layout of the pages, publishing some fresh content… etc.

  Matomo is processing the following personal data :

  Pick up the one you are using :

  • Cookies
  • IP address
  • User ID
  • Custom Dimensions
  • Custom Variables
  • Order ID
  • Location of the user

  And also :

  • Date and time
  • Title of the page being viewed
  • URL of the page being viewed
  • URL of the page that was viewed prior to the current page
  • Screen resolution
  • Time in local timezone
  • Files that were clicked and downloaded
  • Link clicks to an outside domain
  • Pages generation time
  • Country, region, city
  • Main Language of the browser
  • User Agent of the browser

  This list can be completed with additional features such as :

  • Session recording, mouse events (movements, content forms and clicks)
  • Form interactions
  • Media interactions
  • A/B Tests

  Pick up one of the two :

  1. The processing of personal data with Matomo is based on legitimate interests, or :
  2. The processing of personal data with Matomo is based on explicit consent. Your privacy is our highest concern. That’s why we will not process any personal data with Matomo unless you give us clear explicit consent.

  3 – The legitimate interests

  This content applies only if you are processing personal data based on legitimate interests. You need here to justify your legitimate interests to process personal data. It is a set of questions described here.

  Processing your personal data such as cookies is helping us identify what is working and what is not on our website. For example, it helps us identify if the way we are communicating is engaging or not and how we can organize the structure of the website better. Our team is benefiting from the processing of your personal data, and they are directly acting on the website. By processing your personal data, you can profit from a website which is getting better and better.

  Without the data, we would not be able to provide you the service we are currently offering to you. Your data will be used only to improve the user experience on our website and help you find the information you are looking for.

  4 – Recipient of the personal data

  The personal data received through Matomo are sent to :

  • Our company.
  • Our web hosting provider : name and contact details of the web hosting provider.

  Note : If you are using the Matomo Analytics Cloud by InnoCraft the web hosting provider is “InnoCraft, 150 Willis St, 6011 Wellington, New Zealand“.

  5 – Details of transfers to third country and safeguards

  Matomo data is hosted in Name of the country.

  If the country mentioned is not within the EU, you need to mention here the appropriate safeguards, for example : our data is hosted in the United States within company XYZ, registered to the Privacy Shield program.

  Note : The Matomo Analytics Cloud by InnoCraft is currently hosted in France. If you are using the cloud-hosted solution of Matomo, use “France” as name of the country.

  6 – Retention period or criteria used to determine the retention period

  We are keeping the personal data captured within Matomo for a period of indicate here the period.

  Justify your choice, for example : as our data is hosted in France, we are applying the French law which defines a retention period of no more than 13 months. You can set the retention period in Matomo by using the following feature.

  7 – The existence of each of the data subject’s rights

  If you are processing personal data with Matomo based on legitimate interest :

  As Matomo is processing personal data on legitimate interests, you can exercise the following rights :

  • Right of access : you can ask us at any time to access your personal data.
  • Right to erasure : you can ask us at any time to delete all the personal data we are processing about you.
  • Right to object : you can object to the tracking of your personal data by using the following opt-out feature :

  Insert here the opt-out feature.

  If you are processing personal data with Matomo based on explicit consent :

  As Matomo is processing personal data on explicit consent, you can exercise the following rights :

  • Right of access : you can ask us at any time to access your personal data.
  • Right to erasure : you can ask us at any time to delete all the personal data we are processing about you.
  • Right to portability : you can ask us at any time for a copy of all the personal data we are processing about you in Matomo.
  • Right to withdraw consent : you can withdraw your consent at any time by clicking on the following button.

  8 – The right to withdraw consent at any time

  If you are processing personal data under the consent lawful basis, you need to include the following section :

  You can withdraw at any time your consent by clicking here (insert here the Matomo tracking code to remove consent).

  9 – The right to lodge a complaint with a supervisory authority

  If you think that the way we process your personal data with Matomo analytics is infringing the law, you have the right to lodge a complaint with a supervisory authority.

  10 – Whether the provision of personal data is part of a statutory or contractual requirement ; or obligation and possible consequences of failing to provide the personal data

  If you wish us to not process any personal data with Matomo, you can opt-out from it at any time. There will be no consequences at all regarding the use of our website.

  11 – The existence of automated decision-making, including profiling and information about how decisions are made, the significance and the consequences

  Matomo is not doing any profiling.

   

  That’s the end of our blog post. We hope you enjoyed reading it and that it will help you get through the GDPR compliance process. If you have any questions dealing with this privacy policy in particular, do not hesitate to contact us.

  The post How to complete your privacy policy with Matomo analytics under GDPR appeared first on Analytics Platform - Matomo.

• Muxing Android MediaCodec encoded H264 packets into RTMP

  31 décembre 2015, par Vadym

  I am coming from a thread Encoding H.264 from camera with Android MediaCodec. My setup is very similar. However, I attempt to write mux the encoded frames and with javacv and broadcast them via rtmp.

  RtmpClient.java

  ...
  
  private volatile BlockingQueue mFrameQueue = new LinkedBlockingQueue(MAXIMUM_VIDEO_FRAME_BACKLOG);
  
  ...
  
  private void startStream() throws FrameRecorder.Exception, IOException {
  
      if (TextUtils.isEmpty(mDestination)) {
  
          throw new IllegalStateException("Cannot start RtmpClient without destination");
  
      }
  
  
  
      if (mCamera == null) {
  
          throw new IllegalStateException("Cannot start RtmpClient without camera.");
  
      }
  
  
  
      Camera.Parameters cameraParams = mCamera.getParameters();
  
  
  
      mRecorder = new FFmpegFrameRecorder(
  
              mDestination,
  
              mVideoQuality.resX,
  
              mVideoQuality.resY,
  
              (mAudioQuality.channelType.equals(AudioQuality.CHANNEL_TYPE_STEREO) ? 2 : 1));
  
  
  
      mRecorder.setFormat("flv");
  
  
  
      mRecorder.setFrameRate(mVideoQuality.frameRate);
  
      mRecorder.setVideoBitrate(mVideoQuality.bitRate);
  
      mRecorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);
  
  
  
      mRecorder.setSampleRate(mAudioQuality.samplingRate);
  
      mRecorder.setAudioBitrate(mAudioQuality.bitRate);
  
      mRecorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);
  
  
  
      mVideoStream = new VideoStream(mRecorder, mVideoQuality, mFrameQueue, mCamera);
  
      mAudioStream = new AudioStream(mRecorder, mAudioQuality);
  
  
  
      mRecorder.start();
  
  
  
      // Setup a bufferred preview callback
  
      setupCameraCallback(mCamera, mRtmpClient, DEFAULT_PREVIEW_CALLBACK_BUFFERS,
  
              mVideoQuality.resX * mVideoQuality.resY * ImageFormat.getBitsPerPixel(
  
                      cameraParams.getPreviewFormat())/8);
  
  
  
      try {
  
          mVideoStream.start();
  
          mAudioStream.start();
  
      }
  
      catch(Exception e) {
  
          e.printStackTrace();
  
          stopStream();
  
      }
  
  }
  
  ...
  
  @Override
  
  public void onPreviewFrame(byte[] data, Camera camera) {
  
      boolean frameQueued = false;
  
  
  
      if (mRecorder == null || data == null) {
  
          return;
  
      }
  
  
  
      frameQueued = mFrameQueue.offer(data);
  
  
  
      // return the buffer to be reused - done in videostream
  
      //camera.addCallbackBuffer(data);
  
  }
  
  ...

  VideoStream.java

  ...
  
  @Override
  
  public void run() {
  
      try {
  
          mMediaCodec = MediaCodec.createEncoderByType("video/avc");
  
          MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/avc", mVideoQuality.resX, mVideoQuality.resY);
  
          mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, mVideoQuality.bitRate);
  
          mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, mVideoQuality.frameRate);
  
          mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar);
  
          mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
  
          mMediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
  
          mMediaCodec.start();
  
      }
  
      catch(IOException e) {
  
          e.printStackTrace();
  
      }
  
  
  
      long startTimestamp = System.currentTimeMillis();
  
      long frameTimestamp = 0;
  
      byte[] rawFrame = null;
  
  
  
      try {
  
          while (!Thread.interrupted()) {
  
              rawFrame = mFrameQueue.take();
  
  
  
              frameTimestamp = 1000 * (System.currentTimeMillis() - startTimestamp);
  
  
  
              encodeFrame(rawFrame, frameTimestamp);
  
  
  
              // return the buffer to be reused
  
              mCamera.addCallbackBuffer(rawFrame);
  
          }
  
      }
  
      catch (InterruptedException ignore) {
  
          // ignore interrup while waiting
  
      }
  
  
  
      // Clean up video stream allocations
  
      try {
  
          mMediaCodec.stop();
  
          mMediaCodec.release();
  
          mOutputStream.flush();
  
          mOutputStream.close();
  
      } catch (Exception e){
  
          e.printStackTrace();
  
      }
  
  }
  
  ...
  
  private void encodeFrame(byte[] input, long timestamp) {
  
      try {
  
          ByteBuffer[] inputBuffers = mMediaCodec.getInputBuffers();
  
          ByteBuffer[] outputBuffers = mMediaCodec.getOutputBuffers();
  
  
  
          int inputBufferIndex = mMediaCodec.dequeueInputBuffer(0);
  
  
  
          if (inputBufferIndex >= 0) {
  
              ByteBuffer inputBuffer = inputBuffers[inputBufferIndex];
  
              inputBuffer.clear();
  
              inputBuffer.put(input);
  
              mMediaCodec.queueInputBuffer(inputBufferIndex, 0, input.length, timestamp, 0);
  
          }
  
  
  
          MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
  
  
  
          int outputBufferIndex = mMediaCodec.dequeueOutputBuffer(bufferInfo, 0);
  
  
  
          if (outputBufferIndex >= 0) {
  
              while (outputBufferIndex >= 0) {
  
                  ByteBuffer outputBuffer = outputBuffers[outputBufferIndex];
  
  
  
                  // Should this be a direct byte buffer?
  
                  byte[] outData = new byte[bufferInfo.size - bufferInfo.offset];
  
                  outputBuffer.get(outData);
  
  
  
                  mFrameRecorder.record(outData, bufferInfo.offset, outData.length, timestamp);
  
  
  
                  mMediaCodec.releaseOutputBuffer(outputBufferIndex, false);
  
                  outputBufferIndex = mMediaCodec.dequeueOutputBuffer(bufferInfo, 0);
  
              }
  
          }
  
          else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
  
              outputBuffers = mMediaCodec.getOutputBuffers();
  
          } else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
  
              // ignore for now
  
          }
  
      } catch (Throwable t) {
  
          t.printStackTrace();
  
      }
  
  
  
  }
  
  ...

  FFmpegFrameRecorder.java

  ...
  
  // Hackish codec copy frame recording function
  
  public boolean record(byte[] encodedData, int offset, int length, long frameCount) throws Exception {
  
      int ret;
  
  
  
      if (encodedData == null) {
  
          return false;
  
      }
  
  
  
      av_init_packet(video_pkt);
  
  
  
      // this is why i wondered whether I should get outputbuffer data into direct byte buffer
  
      video_outbuf.put(encodedData, 0, encodedData.length);
  
  
  
      video_pkt.data(video_outbuf);
  
      video_pkt.size(video_outbuf_size);
  
  
  
      video_pkt.pts(frameCount);
  
      video_pkt.dts(frameCount);
  
  
  
      video_pkt.stream_index(video_st.index());
  
  
  
      synchronized (oc) {
  
          /* write the compressed frame in the media file */
  
          if (interleaved && audio_st != null) {
  
              if ((ret = av_interleaved_write_frame(oc, video_pkt)) < 0) {
  
                  throw new Exception("av_interleaved_write_frame() error " + ret + " while writing interleaved video frame.");
  
              }
  
          } else {
  
              if ((ret = av_write_frame(oc, video_pkt)) < 0) {
  
                  throw new Exception("av_write_frame() error " + ret + " while writing video frame.");
  
              }
  
          }
  
      }
  
      return (video_pkt.flags() & AV_PKT_FLAG_KEY) == 1;
  
  }
  
  ...

  When I try to stream the video and run ffprobe on it, I get the following output :

  ffprobe version 2.5.3 Copyright (c) 2007-2015 the FFmpeg developers
  
    built on Jan 19 2015 12:56:57 with gcc 4.1.2 (GCC) 20080704 (Red Hat 4.1.2-55)
  
    configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --enable-bzlib --disable-crystalhd --enable-libass --enable-libdc1394 --enable-libfaac --enable-nonfree --disable-indev=jack --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-openal --enable-libopencv --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid --enable-x11grab --enable-avfilter --enable-avresample --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --enable-libcaca --shlibdir=/usr/lib64 --enable-runtime-cpudetect
  
    libavutil      54. 15.100 / 54. 15.100
  
    libavcodec     56. 13.100 / 56. 13.100
  
    libavformat    56. 15.102 / 56. 15.102
  
    libavdevice    56.  3.100 / 56.  3.100
  
    libavfilter     5.  2.103 /  5.  2.103
  
    libavresample   2.  1.  0 /  2.  1.  0
  
    libswscale      3.  1.101 /  3.  1.101
  
    libswresample   1.  1.100 /  1.  1.100
  
    libpostproc    53.  3.100 / 53.  3.100
  
  Metadata:
  
    Server                NGINX RTMP (github.com/arut/nginx-rtmp-module)
  
    width                 320.00
  
    height                240.00
  
    displayWidth          320.00
  
    displayHeight         240.00
  
    duration              0.00
  
    framerate             0.00
  
    fps                   0.00
  
    videodatarate         261.00
  
    videocodecid          7.00
  
    audiodatarate         62.00
  
    audiocodecid          10.00
  
    profile
  
    level
  
  [live_flv @ 0x1edb0820] Could not find codec parameters for stream 0 (Video: none, none, 267 kb/s): unknown codec
  
  Consider increasing the value for the 'analyzeduration' and 'probesize' options
  
  Input #0, live_flv, from 'rtmp://<server>/input/<stream>':
  
    Metadata:
  
      Server          : NGINX RTMP (github.com/arut/nginx-rtmp-module)
  
      displayWidth    : 320
  
      displayHeight   : 240
  
      fps             : 0
  
      profile         :
  
      level           :
  
    Duration: 00:00:00.00, start: 16.768000, bitrate: N/A
  
      Stream #0:0: Video: none, none, 267 kb/s, 1k tbr, 1k tbn, 1k tbc
  
      Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp, 63 kb/s
  
  Unsupported codec with id 0 for input stream 0
  
  </stream></server>

  I am not, by any means, an expert in H264 or video encoding. I know that the encoded frames that come out from MediaCodec contain SPS NAL, PPS NAL, and frame NAL units. I’ve also written the MediaCodec output into a file and was able to play it back (I did have to specify the format and framerate as otherwise it would play too fast).

  My assumption is that things should work (see how little I know :)). Knowing that SPS and PPS are written out, decoder should know enough. Yet, ffprobe fails to recognize codec, fps, and other video information. Do I need to pass packet flag information to FFmpegFrameRecorder.java:record() function ? Or should I use direct buffer ? Any suggestion will be appreciated ! I should figure things out with a hint.

  PS : I know that some codecs use Planar and other SemiPlanar color formats. That distinction will come later if I get past this. Also, I didn’t go the Surface to MediaCodec way because I need to support API 17 and it requires more changes than this route, which I think helps me understand the more basic flow. Agan, I appreciate any suggestions. Please let me know if something needs to be clarified.

  Update #1

  So having done more testing, I see that my encoder outputs the following frames :

  000000016742800DDA0507E806D0A1350000000168CE06E2
  
  0000000165B840A6F1E7F0EA24000AE73BEB5F51CC7000233A84240...
  
  0000000141E2031364E387FD4F9BB3D67F51CC7000279B9F9CFE811...
  
  0000000141E40304423FFFFF0B7867F89FAFFFFFFFFFFCBE8EF25E6...
  
  0000000141E602899A3512EF8AEAD1379F0650CC3F905131504F839...
  
  ...

  The very first frame contains SPS and PPS. From what I was able to see, these are transmitted only once. The rest are NAL types 1 and 5. So, my assumption is that, for ffprobe to see stream info not only when the stream starts, I should capture SPS and PPS frames and re-transmit them myself periodically, after a certain number of frames, or perhaps before every I-frame. What do you think ?

  Update #2

  Unable to validate that I’m writing frames successfully. After having tried to read back the written packet, I cannot validate written bytes. As strange, on successful write of IPL image and streaming, I also cannot print out bytes of encoded packet after avcodec_encode_video2. Hit the official dead end.

• Xbox Sphinx Protocol

  21 octobre 2013, par Multimedia Mike — DRM, xbox

  I’ve gone down the rabbit hole of trying to read the Xbox DVD drive from Linux. Honestly, I’m trying to remember why I even care at this point. Perhaps it’s just my metagame of trying to understand how games and related technologies operate. In my last post of the matter, I determined that it is possible to hook an Xbox drive up to a PC using a standard 40-pin IDE interface and read data sectors. However, I learned that just because the Xbox optical drive is reading an Xbox disc, that doesn’t mean it’s just going to read the sectors in response to a host request.

  Oh goodness, no. The drive is going to make the host work for those sectors.

  To help understand the concept of locked/unlocked sectors on an Xbox disc, I offer this simplistic diagram :

  
  
  

  Any DVD drive (including the Xbox drive) is free to read those first 6992 sectors (about 14 MB of data) which just contain a short DVD video asking the user to insert the disc into a proper Xbox console. Reading the remaining sectors involves performing a sequence of SCSI commands that I have taken to calling the “Sphinx Protocol” for reasons I will explain later in this post.

  References
  Doing a little Googling after my last post on the matter produced this site hosting deep, technical Xbox information. It even has a page about exactly what I am trying to achieve : Use an Xbox DVD Drive in Your PC. The page provides a tool named dvdunlocker written by “The Specialist” to perform the necessary unlocking. The archive includes a compiled Windows binary as well as its source code. The source code is written in Delphi Pascal and leverages Windows SCSI APIs. Still, it is well commented and provides a roadmap, which I will try to describe in this post.

  Sphinx Protocol
  Here is a rough flowchart of the steps that are (probably) involved in the unlocking of those remaining sectors. I reverse engineered this based on the Pascal tool described in the previous section. Disclaimer : at the time of this writing, I haven’t tested all of the steps due to some Linux kernel problems, described later.

  
  
  

  Concerning the challenge/response table that the drive sends back, it’s large (0×664 / 1636 bytes), and not all of the bytes’ meanings are known. However, these are the bytes that seem to be necessary (all multi-byte numbers are big endian) :

   bytes 0-1        Size of mode page payload data (should be 0x0662)
   bytes 2-771      Unknown
   byte  772        Should be 1
   byte  773        Number of entries in challenge/response table
   bytes 774-1026   Encrypted challenge/response table
   bytes 1027-1186  Unknown
   bytes 1187-1230  Key basis (44 bytes)
   bytes 1231-1635  Unknown
  

  The challenge/response table is the interesting part, but it’s encrypted with RC4 a.k.a. ARCFOUR. The key is derived from the 44 bytes I have labeled “key basis”– cryptographic literature probably has a better term for it ; chime in if you know what that might be. An SHA-1 hash is computed over the 44 bytes.

  The resulting SHA-1 hash — the first part of it, to be exact — is fed as the key into the RC4 decryption. The output of SHA-1 contains 160 bits of information. 160 / 8 = 20 bytes of information. To express this as a printable hex digest requires 40 characters. The SHA-1 hash is converted to a hex digest and then the first 7 of the characters are fed into the RC4 initialization function as the key. Then, the RC4 decrypter does its work on the 253 bytes of the challenge/response table.

  So that’s why I took to calling this the “Sphinx Protocol” — I felt like I was being challenged with a bizarre riddle. Perhaps that describes a lot of cryptosystems, though You have to admit it sounds kind of cool.

  The challenge/response table contains 23 11-byte records. The format of this table is (again, multi-byte numbers are big-endian) :

   byte  0     This is 1 if this challenge/response pair is valid
   byte  1     Challenge ID
   bytes 2-5   Challenge
   byte  6     Response ID
   bytes 7-10  Response
  

  Example
  It’s useful to note that the challenge/response table and associated key is different for every disc (at least all the ones I have looked at). So this might be data that comes from the disc, since the values will always be the same for a given disc.

  Let’s examine Official Xbox Magazine disc #16 (Indiana Jones and The Emperor’s Tomb) :

  
  
  

  Before I decrypt the challenge/response table, it looks like this :

     0 : 180, 172 : 0xEB100059 ;  66 : 0xD56AFB56
     1 :  34,  71 : 0x8F9BF03A ; 192 : 0xC32CBDF8
     2 : 226, 216 : 0xA29B77F2 ;  12 : 0x4474A6F1
     3 :  72, 122 : 0x9F5ABF33 ; 255 : 0xC5E3C304
     4 :   1, 103 : 0x76142ADA ; 233 : 0xDE145D42 ****
     5 :  49, 193 : 0xA1CD6192 ; 189 : 0x2169DBA5
     6 : 182, 250 : 0x9977894F ;  96 : 0x5A929E2B
     7 : 148,  71 : 0x6DD10A54 ; 115 : 0xF0BDAC4F
     8 :  12,  45 : 0x5D5EB6FD ; 148 : 0x84E60A00
     9 :  99, 121 : 0xFEAED372 ; 201 : 0xDA9986F9
    10 : 172, 230 : 0xE6C0D0B4 ; 214 : 0x9050C250
    11 :  84,  65 : 0x95CB8775 ; 104 : 0x550886C6
    12 : 210,  65 : 0x1ED23619 ; 171 : 0x6DF4A35B
    13 :   2, 155 : 0xD0AAE1E0 ; 130 : 0x00D1FFCF
    14 :  40,   2 : 0x172EFEB8 ; 159 : 0x37E03E50
    15 :  49,  15 : 0x43E5E378 ; 223 : 0x267F9C9A
    16 : 240, 173 : 0x357D5D1C ; 250 : 0x24965D67
    17 :  80, 184 : 0x5E7AF1A3 ;  81 : 0x3A8F69A7
    18 : 154, 186 : 0x6626BEAC ; 245 : 0xE639540A
    19 : 231, 249 : 0xFABAAFB7 ; 227 : 0x4C686A07
    20 : 150, 186 : 0x9A6D7AA3 ; 133 : 0x25971CF0
    21 : 236, 192 : 0x5CD97DD4 ; 247 : 0x26655EFB
    22 :  68, 173 : 0xE2D372E4 ; 207 : 0x103FBF94
  there are 1 valid pairs in the list : 4
  

  My best clue that it’s not right is that there is only 1 valid entry (denoted by my tool using ****). The source I reverse engineered for this data indicates that there needs to be at least 2 valid pairs. After running the RC4 decryption on the table, it looks like this and I get far more valid pairs :

     0 :   1, 174 : 0xBD628255 ;   0 : 0x9F0A31AF ****
     1 :   2, 176 : 0x3151B341 ;   2 : 0x9C87C180
     2 :   3, 105 : 0x018879E5 ;   1 : 0xFF068B5C
     3 :   2,   7 : 0x1F316AAF ;   3 : 0xF420D3ED
     4 :   3,  73 : 0xC2EBFBE9 ;   0 : 0x17062B5B
     5 : 252, 163 : 0xFF14B5CB ; 236 : 0xAF813FBC
     6 :   2, 233 : 0x5EE95C49 ;   1 : 0x37AA5511
     7 :   1, 126 : 0xBD628255 ;   0 : 0x5BA3FBD4 ****
     8 :   3,   4 : 0xB68BFEE6 ;   3 : 0xA8F3B918
     9 :   3,  32 : 0xEA614943 ;   2 : 0xA678D715
    10 :   2, 248 : 0x1BDD374E ;   0 : 0x8D2AC2C7
    11 :   3,  17 : 0x0EABCE81 ;   2 : 0xC90A7242
    12 :   1, 186 : 0xBD628255 ;   0 : 0xC4820242 ****
    13 :   3, 145 : 0xB178F942 ;   3 : 0x4D78AD62
    14 :   3,  37 : 0x4A6CE5E2 ;   2 : 0xBF94E1C6
    15 :   1, 102 : 0xBD628255 ;   0 : 0xFFB83D8D ****
    16 :   3, 122 : 0xF97B0905 ;   1 : 0x38533125
    17 :   3, 197 : 0x57A6865D ;   2 : 0xA61D31EF
    18 :   3,  27 : 0xC7227D7C ;   2 : 0xA3F9BA1E
    19 :   1,  16 : 0xBD628255 ;   0 : 0x8557CCC8 ****
    20 :   2,  53 : 0x1DA9D156 ;   3 : 0xC9051754
    21 :   2,  90 : 0x3CD66BEE ;   3 : 0xFD851D3E
    22 :   1, 252 : 0xBD628255 ;   0 : 0xB3F22701 ****
  there are 6 valid pairs in the list : 0 7 12 15 19 22
  

  So, hopefully, I have the decryption correct.

  Also of note is that you only get one chance to get this unlocking correct– fail, and the drive won’t return a valid DVD structure block again. You will either need to reboot the Xbox or eject & close the tray before you get to try again.

  Problems Making It Work In Linux
  There are a couple of ways to play with SCSI protocols under Linux. In more recent kernels, block devices are named /dev/sda, /dev/sdb, etc. Each of these block devices has a corresponding character device named /dev/sg0, /dev/sg1, etc. ‘sg’ stands for SCSI generic. This character devices can be opened as readable and/or writable and SCSI commands can be freely written with write() and data retrieved with read(). Pretty powerful.

  Except that the one machine I still possess which supports 40-pin IDE/ATAPI devices is running Linux kernel 2.6.24 which dates back to early 2008 and it still enumerates the IDE block devices as /dev/hda, /dev/hdb, etc. There are no corresponding /dev/sgX character devices. What to do ? It seems that a program can still issue SCSI commands using an ioctl() facility named SG_IO.

  I was able to make the SG_IO ioctl() work for the most part (except for the discovery that the Xbox drive doesn’t respond to a basic SCSI Inquiry command). However, I ran into a serious limitation– a program can only open a /dev/hdX block device in read-only mode if the device corresponds to a read-only drive like, for example, a DVD-ROM drive. This means that a program can’t issue SCSI mode select commands to the drive, which counts as writing. This means that my tool can’t unlock the drive.

  Current Status
  So this is where my experiment is blocked right now. I have been trying to compile various Linux kernels to remedy the situation. But I always seem to find myself stuck in one of 2 situations, depending on the configuration options I choose : Either the drives are enumerated with the /dev/hdX convention and I am stuck in read-only mode (with no mode select) ; or the drives are enumerated with /dev/sdX along with corresponding /dev/sgN character devices, in which case the kernel does not recognize the Xbox DVD-ROM drive.

  This makes me wonder if there’s a discrepancy between the legacy ATA/ATAPI drivers (which sees the drive) and the newer SATA/PATA subsystem (which doesn’t see the drive). I also wonder about hacking the kernel logic to allow SCSI mode select logic to proceed to the device for a read-only file handle.