Recherche avancée

Sur d’autres sites (6588)

• What is data anonymization in web analytics ?

  11 février 2020, par Joselyn Khor — Analytics Tips, Privacy
  Collecting information via web analytics platforms is needed to help a website grow and improve. When doing so, it’s best to strike a balance between getting valuable insights, and keeping the trust of your users by protecting their privacy.

  This means not collecting or processing any personally identifiable information (PII). But what if your organisation requires you to collect PII ?

  That’s where data anonymization comes in.

  

  What is data anonymization ?

  Data anonymization makes identifiable information unidentifiable. This is done through data processing techniques which remove or modify PII data. So data becomes anonymous and can’t be linked to any individual.

  In the context of web analytics, data anonymization is handy because you can collect useful data while protecting the privacy of website visitors.

  Why is data anonymization important ?

  Through modern threats of identity theft, credit card fraud and the like, data anonymization is a way to protect the identity and privacy of individuals. As well as protect private and sensitive information of organisations. 

  Data anonymization lets you follow the many laws around the world which protect user privacy. These laws provide safeguards around collecting personal data or personally identifiable information (PII), so data anonymization is a good solution to ensure you’re not processing such sensitive information.

  In some cases, implementing data anonymization techniques means you can avoid having to show your users a consent screen. Which means you may not need to ask for consent in order to track data. This is a bonus as consent screens can annoy and stop people from engaging with your site.

  GDPR and data anonymization

  

  The GDPR is a law in the EU that limits the collection and processing of personal data. The aim is to give people more control over their online personal information. Which is why website owners need to follow certain rules to become GDPR compliant and protect user privacy. According to the GDPR, you can be fined up to 4% of your yearly revenue for data breaches or non-compliance. 

  In the case of web analytics, tools can be easily made compliant by following a number of steps. 

  This is why anonymizing data is a big deal.

  Anonymized data isn’t personal data according to the GDPR : 

  “The principles of data protection should therefore not apply to anonymous information, namely information which does not relate to an identified or identifiable natural person or to personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable.”

  This means, you still get the best of both worlds. By anonymizing data, you’re still able to collect useful information like visitor behavioural data.

  US privacy laws and data anonymization

  In the US, there isn’t one single law that governs the protection of personal data, called personally identifiable information (PII). There are hundreds of federal and state laws that protect the personal data of US residents. As well as, industry-specific statutes related to data privacy, like the California Consumer Privacy Act (CCPA) and the Health Insurance Portability and Accountability Act (HIPAA).

  Website owners in the US need to know exactly what laws govern their area of business in order to follow them.

  A general guideline is to protect user privacy regardless of whether you are or aren’t allowed to collect PII. This means anonymizing identifiable information so your website users aren’t put at risk.

  Data anonymization techniques in Matomo Analytics

  If you carry these out, you won’t need to ask your website visitors for tracking consent since anonymized data is no longer considered personal data under the GDPR.

  • Anonymize previously tracked raw data – like visitors/visits
  • Anonymize Visitor IP addresses
  • Anonymize geo-location information (country, region, city)
  • Pseudonymize User ID ( if you don’t want this data to land on your Matomo server, the solution is simply to not set the User ID. Then Matomo will use the normal anonymized IDs)
  • Data can also be anonymized before it’s even sent to Matomo
  • Learn how to further configure privacy settings in a web analytics tool like Matomo
  • 11 ways Matomo Analytics helps you to protect your visitors privacy

  The techniques listed above make it easy for you when using a tool like Matomo, as they are automatically anonymized.

  Tools like Google Analytics on the other hand don’t provide some of the privacy options and leave it up to you to take on the burden of implementation without providing steps.

  Data anonymization tools

  If you’re a website owner who wants to grow your business or learn more about your website visitors, privacy-friendly tools like Matomo Analytics are a great option. By following the easy steps to be GDPR compliant, you can anonymize all data that could put your visitors at risk.

• 2011 In Open Source Multimedia

  5 janvier 2012, par Multimedia Mike — Open Source Multimedia

  Sometimes I think that the pace of multimedia technology is slowing down. Obviously, I’m not paying close enough attention. I thought I would do a little 2011 year-end review of what happened in the world of open source multimedia, mainly for my own benefit. Let me know in the comments what I missed.

  The Split
  The biggest deal in open source multimedia was the matter of the project split. Where once stood one project (FFmpeg) there now stands two (also Libav). Where do things stand with the projects now ? Still very separate but similar. Both projects obsessively monitor each other’s git commits and prodigiously poach each other’s work, both projects being LGPL and all. Most features that land in one code base end up in the other. Thus, I refer to FFmpeg and Libav collectively as “the projects”.

  Some philosophical reasons for the split included project stagnation and development process friction. Curiously, these problems are fond memories now and the spirit of competition has pushed development forward at a blinding pace.

  People inside the project have strong opinions about the split ; that’s understandable. People outside the project have strong opinions about the split ; that’s somewhat less understandable, but whatever. After 5 years of working for Adobe on the Flash Player (a.k.a. the most hated software in all existence if internet nerds are to be believed on the matter), I’m so over internet nerd drama.

  For my part, I just try to maintain some appearance of neutrality since I manage some shared resources for the open source multimedia community (like the wiki and samples repo) and am trying to keep them from fracturing as well.

  Apple and Open Source
  It was big news that Apple magnanimously open sourced their lossless audio codec. That sets a great example and precedent.

  New Features
  I mined the 'git log' of the projects in order to pick out some features that were added during 2011.

  First off, Apple’s ProRes video codec was reverse engineered and incorporated into the multimedia libraries. And for some weird reason, this is an item that made the rounds in the geek press. I’m not entirely sure why, but it may have something to do with inter-project conflict. Anyway, here is the decoder in action, playing a video of some wild swine, one of the few samples we have :

  
  
  

  Other new video codecs included a reverse engineered Indeo 4 decoder. Gotta catch ‘em all ! That completes our collection of Indeo codecs. But that wasn’t enough– this year, we got a completely revised Indeo 3 decoder (the previous one, while functional, exhibited a lot of code artifacts betraying a direct ASM ->C translation). Oh, and many thanks to Kostya for this gem :

  
  
  

  That’s the new Origin Xan decoder (best known for Wing Commander IV cinematics) in action, something I first started reverse engineering back in 2002. Thanks to Kostya for picking up my slack yet again.

  Continuing with the codec section, there is a decoder for Adobe Flash Screen Video 2 — big congrats on this ! One of my jobs at Adobe was documenting this format to the outside world and I was afraid I could never quite make it clear enough to build a complete re-implementation. But the team came through.

  Let’s see, there are decoders for VBLE video, Ut Video, Windows Media Image (WMVP/WMP2), Bink audio version ‘b’, H.264 4:2:2 intra frames, and MxPEG video. There is a DPX image encoder, a Cirrus Logic AccuPak video encoder, and a v410 codec.

  How about some more game stuff ? The projects saw — at long last — an SMJPEG demuxer. This will finally allow usage and testing of the SMJPEG IMA ADPCM audio decoder I added about a decade ago. Funny story behind that– I was porting all of my decoders from xine which included the SMJPEG ADPCM. I just never quite got around to writing a corresponding demuxer. Thanks to Paul Mahol for taking care of that.

  Here’s a DFA playback system for a 1995 DOS CD-ROM title called Chronomaster. No format is too obscure, nor its encoded contents too cheesy :

  
  
  

  There’s now a demuxer for a format called XMV that was (is ?) prevalent on Xbox titles. Now the projects can handle FMV files from many Xbox games, such as Thrillville.

  
  
  

  The projects also gained the ability to play BMV files. I think this surfing wizard comes from Discworld II. It’s non-computer-generated animation at a strange resolution.

  
  
  

  More demuxers : xWMA, PlayStation Portable PMP format, and CRI ADX format ; muxer for OpenMG audio and LATM muxer/demuxer.

  One more thing : an AVX-optimized fast Fourier transform (FFT). If you have a machine that supports AVX, there’s no way you’ll even notice the speed increase of a few measly FFT calls for audio coding/decoding, but that’s hardly the point. The projects always use everything on offer for any CPU.

  Please make me aware of features that I missed in the list !

  Continuous Testing
  As a result of the split, each project has its own FATE server, one for FFmpeg and one for Libav. As of the new year, FFmpeg has just over 1000 tests while Libav had 965. This is one area where I’m obviously ecstatic to see competition. Some ad-hoc measurements on my part indicate that the total code coverage via the FATEs has not appreciably increased. But that’s a total percentage. Both the test count and the code count have been steadily rising.

  Google Summer of Code and Google Code-In
  Once again, the projects were allowed to participate in the Google Summer of Code as well as Google Code-In. I confess that I didn’t keep up with these too carefully (and Code-In is still in progress as of this writing). I do know that the project split occurred after FFmpeg had already been accepted for GSoC season 2011 and the admins were gracious enough to allow FFmpeg and Libav to allow both projects to participate in the same slot as long as they could both be mature about it.

  Happy New Year
  Let’s see what we can accomplish in 2012.

• rapid TS fragment ffmpeg decoding - memory leak

  9 août 2017, par NadavRub

  Environment :

  • Ubuntu 16.04 (x64)
  • C++
  • ffmpeg

  Use-case

  • Multiple MPEG-TS fragments are rapidly decoded ( numerous every sec )
  • The format of the TS fragments is dynamic and can’t be known ahead of time
  • The first A/V frames of each fragment are needed to be extracted

  Problem statement

  • The code bellow successfully decodes A/V, BUT, has a huge memory leak ( MBytes/sec )
  • According to the docs seems all memory is freed as it should ( does it... ? )

  Why do I get this huge mem leak, what am I missing in the following code snap ?

      struct MEDIA_TYPE {
  
          ffmpeg::AVMediaType eType;
  
          union {
  
              struct {
  
                  ffmpeg::AVPixelFormat   colorspace;
  
                  int                     width, height;
  
                  float                   fFPS;
  
              } video;
  
              struct : WAVEFORMATEX {
  
                  short sSampleFormat;
  
              } audio;
  
          } format;
  
      };
  
  
  
      struct FRAME {
  
          enum { MAX_PALNES = 3 + 1 };
  
          int         iStrmId;
  
          int64_t     pts;    // Duration in 90Khz clock resolution
  
  
  
          uint8_t**   ppData; // Null terminated
  
          int32_t*    pStride;// Zero terminated
  
      };
  
  
  
      HRESULT ProcessTS(IN Operation op, IN uint8_t* pTS, IN uint32_t uiBytes, bool(*cb)(IN const MEDIA_TYPE& mt, IN FRAME& frame, IN PVOID pCtx), IN PVOID pCbCtx) 
  
      {
  
          uiBytes -= uiBytes % 188;// align to 188 packet size
  
  
  
          struct CONTEXT {
  
              uint8_t* pTS;
  
              uint32_t uiBytes;
  
              int32_t  iPos;
  
          } ctx = { pTS, uiBytes, 0 };
  
  
  
          LOGTRACE(TSDecoder, "ProcessTS(%d, 0x%.8x, %d, 0x%.8x, 0x%.8x), this=0x%.8x\r\n", (int)op, pTS, uiBytes, cb, pCbCtx, this);
  
  
  
          ffmpeg::AVFormatContext* pFmtCtx = 0;
  
          if (0 == (pFmtCtx = ffmpeg::avformat_alloc_context()))
  
              return E_OUTOFMEMORY;
  
  
  
          ffmpeg::AVIOContext* pIoCtx = ffmpeg::avio_alloc_context(pTS, uiBytes, 0, &ctx
  
                                                              , [](void *opaque, uint8_t *buf, int buf_size)->int {
  
                                                                      auto pCtx = (CONTEXT*)opaque;
  
                                                                      int size = pCtx->uiBytes;
  
                                                                      if (pCtx->uiBytes - pCtx->iPos < buf_size)
  
                                                                          size = pCtx->uiBytes - pCtx->iPos;
  
                                                                      if (size > 0) {
  
                                                                          memcpy(buf, pCtx->pTS + pCtx->iPos, size);
  
                                                                          pCtx->iPos += size;
  
                                                                      }
  
                                                                      return size;
  
                                                                  }
  
                                                              , 0 
  
                                                              , [](void* opaque, int64_t offset, int whence)->int64_t { 
  
                                                                  auto pCtx = (CONTEXT*)opaque;
  
                                                                  switch (whence)
  
                                                                      {
  
                                                                      case SEEK_SET:
  
                                                                      pCtx->iPos = offset;
  
                                                                      break;
  
                                                                  case SEEK_CUR:
  
                                                                      pCtx->iPos += offset;
  
                                                                      break;
  
                                                                  case SEEK_END:
  
                                                                      pCtx->iPos = pCtx->uiBytes - offset;
  
                                                                      break;
  
                                                                  case AVSEEK_SIZE:
  
                                                                      return pCtx->uiBytes;
  
                                                                  }
  
                                                                  return pCtx->iPos;
  
                                                              });
  
  
  
      pFmtCtx->pb = pIoCtx;
  
  
  
      int iRet = ffmpeg::avformat_open_input(&pFmtCtx, "fakevideo.ts", m_pInputFmt, 0);
  
      if (ERROR_SUCCESS != iRet) {
  
          assert(false);
  
          pFmtCtx = 0;// a user-supplied AVFormatContext will be freed on failure.
  
          return E_FAIL;
  
      }
  
  
  
      struct DecodeContext {
  
          ffmpeg::AVStream*   pStream;
  
          ffmpeg::AVCodec*    pDecoder;
  
          int                 iFramesProcessed;
  
      };
  
  
  
      HRESULT hr                  = S_OK;
  
      int     iStreamsProcessed   = 0;
  
  
  
      bool    bVideoFound         = false;
  
      int64_t ptsLast             = 0;
  
      int64_t dtsLast             = 0;
  
  
  
      auto pContext = (DecodeContext*)alloca(sizeof(DecodeContext) * pFmtCtx->nb_streams);
  
      for (unsigned int i = 0; i < pFmtCtx->nb_streams; i++) {
  
          assert(pFmtCtx->streams[i]->index == i);
  
          pContext[i].pStream         = pFmtCtx->streams[i];
  
          pContext[i].pDecoder        = ffmpeg::avcodec_find_decoder(pFmtCtx->streams[i]->codec->codec_id);
  
          pContext[i].iFramesProcessed= 0;
  
          if (0 == pContext[i].pDecoder)
  
              continue;
  
          if ((iRet = ffmpeg::avcodec_open2(pFmtCtx->streams[i]->codec, pContext[i].pDecoder, NULL)) < 0) {
  
              _ASSERT(FALSE);
  
              hr = E_FAIL;
  
              goto ErrExit;
  
          }
  
      }
  
  
  
      while (S_OK == hr) {
  
          ffmpeg::AVFrame* pFrame = 0;
  
          ffmpeg::AVPacket pkt;
  
          ffmpeg::av_init_packet(&pkt);
  
          if (ERROR_SUCCESS != (iRet = ffmpeg::av_read_frame(pFmtCtx, &pkt))) {
  
              hr = E_FAIL;
  
              break;
  
          }
  
          if ((0 == dtsLast) && (0 != pkt.dts))
  
              dtsLast = pkt.dts;
  
          if ((0 == ptsLast) && (0 != pkt.pts))
  
              ptsLast = pkt.pts;
  
          DecodeContext& ctx = pContext[pkt.stream_index];
  
          if (Operation::DECODE_FIRST_FRAME_OF_EACH_STREAM == op) {
  
              if (iStreamsProcessed == pFmtCtx->nb_streams) {
  
                  hr = S_FALSE;
  
                  goto Next;
  
              }
  
              if (ctx.iFramesProcessed > 0)
  
                  goto Next;
  
              iStreamsProcessed++;
  
          }
  
          if (0 == ctx.pDecoder)
  
              goto Next;
  
  
  
          if (0 == (pFrame = ffmpeg::av_frame_alloc())) {
  
              hr = E_OUTOFMEMORY;
  
              goto Next;
  
          }
  
  
  
          LOGTRACE(TSDecoder, "ProcessTS(%d, 0x%.8x, %d, 0x%.8x, 0x%.8x), this=0x%.8x, decode, S:%d, T:%d\r\n", (int)op, pTS, uiBytes, cb, pCbCtx, this, pkt.stream_index, ctx.pStream->codec->codec_type);
  
          int         bGotFrame   = false;
  
          int         iBytesUsed  =   0;
  
          MEDIA_TYPE  mt;
  
          memset(&mt, 0, sizeof(mt));
  
          mt.eType = ctx.pStream->codec->codec_type;
  
          switch (mt.eType) {
  
          case ffmpeg::AVMediaType::AVMEDIA_TYPE_AUDIO:
  
              ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
  
              if((iRet = ffmpeg::avcodec_decode_audio4(ctx.pStream->codec, pFrame, &bGotFrame, &pkt)) < 0) {
  
                  hr = E_FAIL;
  
                  goto Next;
  
              }
  
              _ASSERT(pkt.size == iRet);
  
              // FFMPEG AAC decoder oddity, first call to 'avcodec_decode_audio4' results mute audio where the second result the expected audio
  
              bGotFrame = false;
  
              if ((iRet = ffmpeg::avcodec_decode_audio4(ctx.pStream->codec, pFrame, &bGotFrame, &pkt)) < 0) {
  
                  hr = E_FAIL;
  
                  goto Next;
  
              }
  
              _ASSERT(pkt.size == iRet);
  
              ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
  
              if (false == bGotFrame)
  
                  goto Next;
  
  
  
              iBytesUsed                      = ctx.pStream->codec->frame_size;
  
              mt.format.audio.nChannels       = ctx.pStream->codec->channels;
  
              mt.format.audio.nSamplesPerSec  = ctx.pStream->codec->sample_rate;
  
              mt.format.audio.wBitsPerSample  = ffmpeg::av_get_bytes_per_sample(ctx.pStream->codec->sample_fmt) * 8;
  
              mt.format.audio.nBlockAlign     = mt.format.audio.nChannels * mt.format.audio.wBitsPerSample / 8;
  
              mt.format.audio.sSampleFormat   = (short)pFrame->format;
  
              break;
  
          case ffmpeg::AVMediaType::AVMEDIA_TYPE_VIDEO:
  
              if ((iRet = ffmpeg::avcodec_decode_video2(ctx.pStream->codec, pFrame, &bGotFrame, &pkt)) < 0) {
  
                  hr = E_FAIL;
  
                  break;
  
              }
  
              if (false == bGotFrame)
  
                  goto Next;
  
  
  
              assert(ffmpeg::AVPixelFormat::AV_PIX_FMT_YUV420P == ctx.pStream->codec->pix_fmt);// Thats is the only color space currently supported
  
              iBytesUsed                      = (ctx.pStream->codec->width * ctx.pStream->codec->height * 3) / 2;
  
              mt.format.video.width           = ctx.pStream->codec->width;
  
              mt.format.video.height          = ctx.pStream->codec->height;
  
              mt.format.video.colorspace      = ctx.pStream->codec->pix_fmt;
  
              mt.format.video.fFPS            = (float)ctx.pStream->codec->framerate.num / ctx.pStream->codec->framerate.den;
  
              bVideoFound                     = true;
  
              break;
  
          default:
  
              goto Next;
  
          }
  
  
  
          ctx.iFramesProcessed++;
  
  
  
          {
  
              FRAME f = { ctx.pStream->index, ((0 == ptsLast) ? dtsLast : ptsLast), (uint8_t**)pFrame->data, (int32_t*)pFrame->linesize };
  
              if ((iRet > 0) && (false == cb(mt, f, pCbCtx)))
  
                  hr = S_FALSE;// Breaks the loop
  
          }
  
      Next:
  
          ffmpeg::av_free_packet(&pkt);
  
          if (0 != pFrame) {
  
              //ffmpeg::av_frame_unref(pFrame);
  
              ffmpeg::av_frame_free(&pFrame);
  
              pFrame = 0;
  
          }
  
      }
  
  
  
  ErrExit:
  
      for (unsigned int i = 0; i < pFmtCtx->nb_streams; i++)
  
          ffmpeg::avcodec_close(pFmtCtx->streams[i]->codec);
  
  
  
      pIoCtx->buffer = 0;// We have allocated the buffer, no need for ffmpeg to free it 4 us
  
      pFmtCtx->pb = 0;
  
      ffmpeg::av_free(pIoCtx);
  
      ffmpeg::avformat_close_input(&pFmtCtx);
  
      ffmpeg::avformat_free_context(pFmtCtx);
  
      return hr;
  
  }