Sur d’autres sites (3952)

• ffmpeg/libx264 C API : frames dropped from end of short MP4

  19 juillet 2017, par Blake McConnell

  In my C++ application, I am taking a series of JPEG images, manipulating their data using FreeImage, and then encoding the bitmaps as H264 using the ffmpeg/libx264 C API. The output is an MP4 which shows the series of 22 images at 12fps. My code is adapted from the "muxing" example that comes with ffmpeg C source code.

  My problem : no matter how I tune the codec parameters, a certain number of frames at the end of the sequence which are passed to the encoder do not appear in the final output. I’ve set the AVCodecContext parameters like this :

  //set context params
  
  ctx->codec_id = AV_CODEC_ID_H264;
  
  ctx->bit_rate = 4000 * 1000;
  
  ctx->width = _width;
  
  ctx->height = _height;
  
  ost->st->time_base = AVRational{ 1, 12 };
  
  ctx->time_base = ost->st->time_base;
  
  ctx->gop_size = 1;
  
  ctx->pix_fmt = AV_PIX_FMT_YUV420P;

  I have found that the higher the gop_size the more frames are dropped from the end of the video. I can also see from the output that, with this gop size (where I’m essentially directing that all output frames be I frames) that only 9 frames are written.

  I’m not sure why this is occurring. I experimented with encoding duplicate frames and making a much longer video. This resulted in no frames being dropped. I know with the ffmpeg command line tool there is a concatenation command that accomplishes what I am trying to do, but I’m not sure how to accomplish the same goal using the C API.

  Here’s the output I’m getting from the console :

  [libx264 @ 026d81c0] using cpu capabilities : MMX2 SSE2Fast SSSE3
  SSE4.2 AVX FMA3 BMI2 AVX2 [libx264 @ 026d81c0] profile High, level
  3.1 [libx264 @ 026d81c0] 264 - core 152 r2851 ba24899 - H.264/MPEG-4 AVC codec - Cop yleft 2003-2017 - http://www.videolan.org/x264.html -
  options : cabac=1 ref=1 deb lock=1:0:0 analyse=0x3:0x113 me=hex subme=7
  psy=1 psy_rd=1.00:0.00 mixed_ref=0 m e_range=16 chroma_me=1 trellis=1
  8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chro ma_qp_offset=-2
  threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1
  interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0
  keyint=1 ke yint_min=1 scenecut=40 intra_refresh=0 rc=abr mbtree=0
  bitrate=4000 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4
  ip_ratio=1.40 aq=1:1.00 Output #0, mp4, to
  ’....\images\c411a991-46f6-400c-8bb0-77af3738559a.mp4’ :
  Stream #0:0 : Video : h264, yuv420p, 700x700, q=2-31, 4000 kb/s, 12 tbn

  [libx264 @ 026d81c0] frame I:9 Avg QP:17.83 size:111058 [libx264
  @ 026d81c0] mb I I16..4 : 1.9% 47.7% 50.5% [libx264 @ 026d81c0] final
  ratefactor : 19.14 [libx264 @ 026d81c0] 8x8 transform intra:47.7%
  [libx264 @ 026d81c0] coded y,uvDC,uvAC intra : 98.4% 96.9% 89.5%
  [libx264 @ 026d81c0] i16 v,h,dc,p : 64% 6% 2% 28% [libx264 @
  026d81c0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu : 32% 15% 9% 5% 5% 6% 8%
  10% 10% [libx264 @ 026d81c0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu : 28% 18%
  7% 6% 8% 8% 8% 9% 8% [libx264 @ 026d81c0] i8c dc,h,v,p : 43% 22%
  25% 10% [libx264 @ 026d81c0] kb/s:10661.53

  Code included below :

  MP4Writer.h

  #ifndef MPEG_WRITER
  
  #define MPEG_WRITER
  
  
  
  #include <iostream>
  
  #include <string>
  
  #include <vector>
  
  #include 
  
  extern "C" {
  
      #include <libavformat></libavformat>avformat.h>
  
      #include <libswscale></libswscale>swscale.h>
  
      #include <libswresample></libswresample>swresample.h>
  
      #include <libswscale></libswscale>swscale.h>
  
  }
  
  
  
  typedef struct OutputStream 
  
  {
  
      AVStream *st;
  
      AVCodecContext *enc;
  
  
  
      //pts of the next frame that will be generated
  
      int64_t next_pts;
  
      int samples_count;
  
  
  
      AVFrame *frame;
  
      AVFrame *tmp_frame;
  
  
  
      float t, tincr, tincr2;
  
  
  
      struct SwsContext *sws_ctx;
  
      struct SwrContext *swr_ctx;
  
  };
  
  
  
  class MP4Writer {
  
      public:
  
          MP4Writer();
  
          void Init();
  
          int16_t SetOutput( const std::string &amp; path );
  
          int16_t AddFrame( uint8_t * imgData );
  
          int16_t Write( std::vector<imgdata> &amp; imgData );
  
          int16_t Finalize();
  
          void SetHeight( const int height ) { _height = _width = height; } //assuming 1:1 aspect ratio
  
  
  
      private:
  
          int16_t AddStream( OutputStream * ost, AVFormatContext * formatCtx, AVCodec ** codec, enum AVCodecID codecId );
  
          int16_t OpenVideo( AVFormatContext * formatCtx, AVCodec *codec, OutputStream * ost, AVDictionary * optArg );
  
          static AVFrame * AllocPicture( enum AVPixelFormat pixFmt, int width, int height );
  
          static AVFrame * GetVideoFrame( uint8_t * imgData, OutputStream * ost, const int width, const int height );
  
          static int WriteFrame( AVFormatContext * formatCtx, const AVRational * timeBase, AVStream * stream, AVPacket * packet );
  
  
  
          int _width;
  
          int _height;
  
          OutputStream _ost;
  
          AVFormatContext * _formatCtx;
  
          AVDictionary * _dict;
  
  };
  
  
  
  #endif //MPEG_WRITER
  
  </imgdata></vector></string></iostream>

  MP4Writer.cpp

  #include 
  
  #include <algorithm>
  
  
  
  MP4Writer::MP4Writer()
  
  {
  
      _width = 0;
  
      _height = 0;
  
  }
  
  
  
  void MP4Writer::Init()
  
  {
  
      av_register_all();
  
  }
  
  
  
  /**
  
   sets up output stream for the specified path.
  
   note that the output format is deduced automatically from the file extension passed
  
   @param path: output file path
  
   @returns: -1 = output could not be deduced, -2 = invalid codec, -3 = error opening output file,
  
             -4 = error writing header
  
  */
  
  int16_t MP4Writer::SetOutput( const std::string &amp; path )
  
  {
  
      int error;
  
      AVCodec * codec;
  
      AVOutputFormat * format;
  
  
  
      _ost = OutputStream{}; //TODO reset state in a more focused way?
  
  
  
      //allocate output media context
  
      avformat_alloc_output_context2( &amp;_formatCtx, NULL, NULL, path.c_str() );
  
      if ( !_formatCtx ) {
  
          std::cout &lt;&lt; "could not deduce output format from file extension.  aborting" &lt;&lt; std::endl;
  
          return -1;
  
      }
  
      //set format
  
      format = _formatCtx->oformat;
  
      if ( format->video_codec != AV_CODEC_ID_NONE ) {
  
          AddStream( &amp;_ost, _formatCtx, &amp;codec, format->video_codec );
  
      }
  
      else {
  
          std::cout &lt;&lt; "there is no video codec set.  aborting" &lt;&lt; std::endl;
  
          return -2;
  
      }
  
  
  
      OpenVideo( _formatCtx, codec, &amp;_ost, _dict );
  
  
  
      av_dump_format( _formatCtx, 0, path.c_str(), 1 );
  
  
  
      //open output file
  
      if ( !( format->flags &amp; AVFMT_NOFILE )) {
  
          error = avio_open( &amp;_formatCtx->pb, path.c_str(), AVIO_FLAG_WRITE );
  
          if ( error &lt; 0 ) {
  
              std::cout &lt;&lt; "there was an error opening output file " &lt;&lt; path &lt;&lt; ".  aborting" &lt;&lt; std::endl;
  
              return -3;
  
          }
  
      }
  
  
  
      //write header
  
      error = avformat_write_header( _formatCtx, &amp;_dict );
  
      if ( error &lt; 0 ) {
  
          std::cout &lt;&lt; "an error occurred writing header. aborting" &lt;&lt; std::endl;
  
          return -4;
  
      }
  
  
  
      return 0;
  
  }
  
  
  
  /**
  
   initialize the output stream
  
   @param ost: the output stream
  
   @param formatCtx: the context format
  
   @param codec: the output codec
  
   @param codec: the ffmpeg enumerated id of the codec
  
   @returns: -1 = encoder not found, -2 = stream could not be allocated, -3 = encoding context could not be allocated
  
  */
  
  int16_t MP4Writer::AddStream( OutputStream * ost, AVFormatContext * formatCtx, AVCodec ** codec, enum AVCodecID codecId )
  
  {
  
      AVCodecContext * ctx; //TODO not sure why this is here, could just set ost->enc directly
  
      int i;
  
  
  
      //detect the encoder
  
      *codec = avcodec_find_encoder( codecId );
  
      if ( (*codec) == NULL ) {
  
          std::cout &lt;&lt; "could not find encoder.  aborting" &lt;&lt; std::endl;
  
          return -1;
  
      }
  
  
  
      //allocate stream
  
      ost->st = avformat_new_stream( formatCtx, NULL );
  
      if ( ost->st == NULL ) {
  
          std::cout &lt;&lt; "could not allocate stream.  aborting" &lt;&lt; std::endl;
  
          return -2;
  
      }
  
  
  
      //allocate encoding context
  
      ost->st->id = formatCtx->nb_streams - 1;
  
      ctx = avcodec_alloc_context3( *codec );
  
      if ( ctx == NULL ) {
  
          std::cout &lt;&lt; "could not allocate encoding context.  aborting" &lt;&lt; std::endl;
  
          return -3;
  
      }
  
  
  
      ost->enc = ctx;
  
  
  
      //set context params
  
      ctx->codec_id = AV_CODEC_ID_H264;
  
      ctx->bit_rate = 4000 * 1000;
  
      ctx->width = _width;
  
      ctx->height = _height;
  
      ost->st->time_base = AVRational{ 1, 12 };
  
      ctx->time_base = ost->st->time_base;
  
      ctx->gop_size = 1;
  
      ctx->pix_fmt = AV_PIX_FMT_YUV420P;
  
  
  
      //if neccesary, set stream headers and formats separately
  
      if ( formatCtx->oformat->flags &amp; AVFMT_GLOBALHEADER ) {
  
          std::cout &lt;&lt; "setting stream and headers to be separate" &lt;&lt; std::endl;
  
          ctx->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
  
      }
  
  
  
      return 0;
  
  }
  
  
  
  /**
  
   open the video for writing
  
   @param formatCtx: the format context
  
   @param codec: output codec
  
   @param ost: output stream
  
   @param optArg: dictionary
  
   @return: -1 = error opening codec, -2 = allocate new frame, -3 = copy stream params
  
  */
  
  int16_t MP4Writer::OpenVideo( AVFormatContext * formatCtx, AVCodec *codec, OutputStream * ost, AVDictionary * optArg )
  
  {
  
      int error;
  
      AVCodecContext * ctx = ost->enc;
  
      AVDictionary * dict = NULL;
  
      av_dict_copy( &amp;dict, optArg, 0 );
  
  
  
      //open codec
  
      error = avcodec_open2( ctx, codec, &amp;dict );
  
      av_dict_free( &amp;dict );
  
      if ( error &lt; 0 ) {
  
          std::cout &lt;&lt; "there was an error opening the codec.  aborting" &lt;&lt; std::endl;
  
          return -1;
  
      }
  
  
  
      //allocate new frame
  
      ost->frame = AllocPicture( ctx->pix_fmt, ctx->width, ctx->height );
  
      if ( ost->frame == NULL ) {
  
          std::cout &lt;&lt; "there was an error allocating a new frame.  aborting" &lt;&lt; std::endl;
  
          return -2;
  
      }
  
  
  
      //copy steam params
  
      error = avcodec_parameters_from_context( ost->st->codecpar, ctx );
  
      if ( error &lt; 0 ) {
  
          std::cout &lt;&lt; "could not copy stream parameters.  aborting" &lt;&lt; std::endl;
  
          return -3;
  
      }
  
  
  
      return 0;
  
  }
  
  
  
  /**
  
   allocate a new frame
  
   @param pixFmt: ffmpeg enumerated pixel format
  
   @param width: output width
  
   @param height: output height
  
   @returns: an inititalized frame
  
  */
  
  AVFrame * MP4Writer::AllocPicture( enum AVPixelFormat pixFmt, int width, int height )
  
  {
  
      AVFrame * picture;
  
      int error;
  
  
  
      //allocate the frame
  
      picture = av_frame_alloc();
  
      if ( picture == NULL ) {
  
          std::cout &lt;&lt; "there was an error allocating the picture" &lt;&lt; std::endl;
  
          return NULL;
  
      }
  
  
  
      picture->format = pixFmt;
  
      picture->width = width;
  
      picture->height = height;
  
  
  
      //allocate the frame's data buffer
  
      error = av_frame_get_buffer( picture, 32 );
  
      if ( error &lt; 0 ) {
  
          std::cout &lt;&lt; "could not allocate frame data" &lt;&lt; std::endl;
  
          return NULL;
  
      }
  
      picture->pts = 0;
  
      return picture;
  
  }
  
  
  
  /**
  
   convert raw RGB buffer to YUV frame
  
   @return: frame that contains image data
  
  */
  
  AVFrame * MP4Writer::GetVideoFrame( uint8_t * imgData, OutputStream * ost, const int width, const int height )
  
  {
  
      int error;
  
      AVCodecContext * ctx = ost->enc;
  
  
  
      //prepare the frame
  
      error = av_frame_make_writable( ost->frame );
  
      if ( error &lt; 0 ) {
  
          std::cout &lt;&lt; "could not make frame writeable" &lt;&lt; std::endl;
  
          return NULL;
  
      }
  
  
  
      //TODO set this context one time per run, or even better, one time at init
  
      //convert RGB to YUV
  
      struct SwsContext* fooContext = sws_getContext( width, height, AV_PIX_FMT_BGR24, 
  
          width, height, AV_PIX_FMT_YUV420P, SWS_BICUBIC, NULL, NULL, NULL );
  
      int inLinesize[1] = { 3 * width }; // RGB stride
  
      uint8_t * inData[1] = { imgData };
  
      int sliceHeight = sws_scale( fooContext, inData, inLinesize, 0, height, ost->frame->data, ost->frame->linesize );
  
      sws_freeContext( fooContext );
  
  
  
      ost->frame->pts = ost->next_pts++;
  
      //TODO does the frame need to be returned here as it is available at the class level?
  
      return ost->frame;
  
  }
  
  
  
  /**
  
   write frame to file
  
   @param formatCtx: the output format context
  
   @param timeBase: the framerate
  
   @param stream: output stream
  
   @param packet: data packet
  
   @returns: see return values for av_interleaved_write_frame
  
  */
  
  int MP4Writer::WriteFrame( AVFormatContext * formatCtx, const AVRational * timeBase, AVStream * stream, AVPacket * packet )
  
  {
  
      av_packet_rescale_ts( packet, *timeBase, stream->time_base );
  
      packet->stream_index = stream->index;
  
  
  
      //write compressed file to media file
  
      return av_interleaved_write_frame( formatCtx, packet );
  
  }
  
  
  
  int16_t MP4Writer::Write( std::vector<imgdata> &amp; imgData )
  
  {
  
      int16_t errorCount = 0;
  
      int16_t retVal = 0;
  
      bool countingUp = true;
  
      size_t i = 0;
  
      while ( true ) {
  
          //don't show first frame again when counting back down
  
          if ( !countingUp &amp;&amp; i == 0 ) {
  
              break;
  
          }
  
          uint8_t * pixels = imgData[i].GetBits( imgData[i].mp4Input );
  
          AddFrame( pixels );
  
  
  
          //handle inc/dec without repeating last frame
  
          if ( countingUp ) {
  
              if ( i == imgData.size() -1 ) {
  
                  countingUp = false;
  
                  i--;
  
              }
  
              else {
  
                  i++;
  
              }
  
          }
  
          else {
  
              i--;
  
          }
  
      }
  
      Finalize();
  
      return 0; //TODO return error code
  
  }
  
  
  
  /**
  
   add another frame to output video
  
   @param imgData: the raw image data
  
   @returns -1 = error encoding video frame, -2 = error writing frame
  
  */
  
  int16_t MP4Writer::AddFrame( uint8_t * imgData )
  
  {
  
      int error;
  
      AVCodecContext * ctx;
  
      AVFrame * frame;
  
      int gotPacket = 0;
  
      AVPacket pkt = { 0 };
  
  
  
      ctx = _ost.enc;
  
      av_init_packet( &amp;pkt );
  
  
  
      frame = GetVideoFrame( imgData, &amp;_ost, _width, _height );
  
  
  
      //encode the image
  
      error = avcodec_encode_video2( ctx, &amp;pkt, frame, &amp;gotPacket );
  
      if ( error &lt; 0 ) {
  
          std::cout &lt;&lt; "there was an error encoding the video frame" &lt;&lt; std::endl;
  
          return -1;
  
      }
  
  
  
      //write the frame.  NOTE: this doesn't kick in until the encoder has received a certain number of frames
  
      if ( gotPacket ) {
  
          error = WriteFrame( _formatCtx, &amp;ctx->time_base, _ost.st, &amp;pkt );
  
          if ( error &lt; 0 ) {
  
              std::cout &lt;&lt; "the video frame could not be written" &lt;&lt; std::endl;
  
              return -2;
  
          }
  
      }
  
      return 0;
  
  }
  
  
  
  /**
  
   finalize output video and cleanup
  
  */
  
  int16_t MP4Writer::Finalize()
  
  {
  
      av_write_trailer( _formatCtx );
  
      avcodec_free_context( &amp;_ost.enc );
  
      av_frame_free( &amp;_ost.frame);
  
      av_frame_free( &amp;_ost.tmp_frame );
  
      avio_closep( &amp;_formatCtx->pb );
  
      avformat_free_context( _formatCtx );
  
      sws_freeContext( _ost.sws_ctx );
  
      swr_free( &amp;_ost.swr_ctx);
  
      return 0;
  
  }
  
  </imgdata></algorithm>

  usage

  #include 
  
  #include 
  
  #include <vector>
  
  
  
  struct ImgData
  
  {
  
      unsigned int width;
  
      unsigned int height;
  
      std::string path;
  
      FIBITMAP * mp4Input;
  
  
  
      uint8_t * GetBits( FIBITMAP * bmp ) { return FreeImage_GetBits( bmp ); }
  
  };
  
  
  
  int main()
  
  {
  
       std::vector<imgdata> imgDataVec;
  
       //load images and push to imgDataVec
  
       MP4Writer mp4Writer;
  
       mp4Writer.SetHeight( 1200 ); //assumes 1:1 aspect ratio
  
       mp4Writer.Init();
  
       mp4Writer.SetOutput( "test.mp4" );
  
       mp4Writer.Write( imgDataVec );
  
  }
  
  </imgdata></vector>

• Exceeded GA’s 10M hits data limit, now what ?

  1er décembre 2021, par Joselyn Khor
  Exceeded Google Analytics’ 10M hits data limit, now what ?

  “Your data volume (1XXM hits) exceeds the limit of 10M hits per month as outlined in our Terms of Service. If you continue to exceed the limit, we will stop processing new data on XXX 21, 2019. Learn more about possible solutions.”

  Yikes. Alarm bells were ringing when a Google Analytics free user came to us faced with this notice. Let’s call him ‘Mark’. Mark had reached the limits on the data he could collect through Google Analytics and was shocked by the limited options available to fix the problem, without blowing the budget. The thoughts racing through his head were :

  • “What happens to all my data ?”
  • “What if Google starts charging USD150K now ?”

  Then he came across Matomo and decided to get in touch with our support team …

  “Can you fix this issue ?” he asked us.

  “Absolutely !” we said.

  We’ll get back to helping Mark in a minute. For now let’s go over why this was such a dilemma for him.

  In order to resolve this data limits issue, one of the solutions was for him to upgrade to Google Analytics 360, which meant shelling out USD150,000 per year for their 1 billion hits per month option. Going from free to USD150,000 was too much of a stretch for a growing company.

  “Your data volume (1XXM hits) exceeds the limit of 10M hits per month …”, what did this message mean ?

  With the free version, Mark could collect up to 10 million “hits” per month, per account. Going over meant Google Analytics could stop collecting any more data for free as outlined in their Terms.

  Google Analytics’ Terms of Service (2018, sec. 2) states, “Subject to Section 15, the Service is provided without charge to You for up to 10 million Hits per month per account.”[1]

  What is a "hit" in Google Analytics ?

  Data being sent to Google Analytics. It can be a transaction, event, social interaction or pageview - these all produce what Google calls a “hit”.

  

  And their Analytics Help Data Limits (n.d.) support page makes clear that : “If a property sends more hits per month to Analytics than allowed by the Analytics Terms of Service, there is no assurance that the excess hits will be processed. If the property’s hit volume exceeds this limit, a warning may be displayed in the user interface and you may be prevented from accessing reports.”[2]

  

  Possible solutions

  So the possible solutions given by Google Analytics’ Data Limits support page were (also shown in image below) :

  • To pay USD150K to upgrade to Google Analytics 360
  • To send fewer hits by setting up sampling
  • Or choose the slightly less relevant option to upgrade mobile app tracking to Google Analytics for Firebase.

  Without the means to pay, the free version was fast becoming inaccessible for Mark as he was facing a future where he risked no longer having access to up-to-date data used in his business’ reporting.

  Mark was facing a problem that potentially didn’t have a cost-effective solution.

  

  So what can you really do about it ?

  This is where we can help provide some assistance. If you’re reading this article, we’ll assume you can relate to Mark and share with you the advice on options we gave him.

  Options :

  One option posed by Google is for you to send fewer hits by auditing your data collection processes

  If you really don’t have the budget, you’ll need to reassess your data collection priorities and go over your strategies to see what is necessary to track, and what isn’t.

  • Make sure you know what you’re tracking and why. Look at what websites are being tracked by Google and into what properties.
  • Go through what data you’re tracking and decide what is or isn’t of value.
  • Set up data sampling, this however, will lead to inaccurate data.

  From here you can start to course correct. If you’ve found data you’re not using for analysis, get rid of these events/pageviews in your Google Analytics.

  But the limitations here are that eventually, you’re going to run out of irrelevant metrics and everything you’re tracking will be essential. So you’ll hit another brick wall and return to the same situation.

  Option 2 – Ignore and continue using the free version of Google Analytics

  With this option, you’ll have to bear the business risks involved by basing decisions off of analytics reports that may or may not be updated. In this case, you may still get contacted about exceeding the limits. As the free service is provided for only up to 10 million hits, once you’ve gone over them, you’re violating what’s stipulated in the Terms of Service. 

  There’s also the warning that “… you may be prevented from accessing reports” (Data limits, n.d.). So while we may not know for certain what Google Analytics will do, in this case it may be better to be safe rather than sorry by acting quickly to resolve it. 

  Option 3 -The Matomo solution – a privacy-friendly Google Analytics alternative

  Upgrade to a web analytics platform that can handle your demanding data requirements. Save money while continuing to gain valuable insights by moving over to Matomo Analytics (recommended)

  This is where you can save up to USD130,000 a year. As well as that, the transition from Google Analytics to the Matomo Cloud is a seamless experience as setup and maintenance is taken care of by our experts.

  For example, you can get up to 25M hits for USD3,241/month (or USD38,900/year) on the Essentials plan.

  Or even 25M hits for USD4,991/month (or USD59,900/year) on the Business plan – which offers additional web analytics and conversion optimization resources.

  Matomo Cloud is a great option if you’re looking for a secure, cost-effective and powerful analytics solution. You also get what Google Analytics could never offer you : full control and ownership of your own data and privacy. 

  Try Matomo free for 21 days – no credit card required.

  No need to worry about losing your Google Analytics data because …

  Now you can import your historic Google Analytics data directly into your Matomo with the Google Analytics Importer tool. Simply follow the step-by-step guide to get started for free.

  Along with savings you can get :

  • A solution for the data limits issue forever. You choose the right plan to suit your data needs and adapt as you continue growing
  • 100% accurate data (no data sampling)
  • 100% data ownership of all your information without signing away your data to a third party
  • Powerful web analytics and conversion optimization features
  • Matomo Tag Manager
  • Easy setup
  • Support from Matomo’s specialists

  Learn more about Matomo Cloud pricing.

  

  Or go for Matomo On-Premise

  If you have the in-house infrastructure to support self-hosting Matomo on your own servers then there’s also the option of Matomo On-Premise. Here you’ll get full security knowing the data is on your own servers. 

  Setup will also require technical knowledge. There will also be costs associated with acquiring your own servers, and keeping up with regular maintenance and updates. With On-Premise you get maximum flexibility, with no data limits whatsoever. But if you’re coming over from Google Analytics and don’t have the infrastructure and team to host On-Premise, the Matomo Cloud could be right for you.

  Learn more about Matomo On-Premise.

  Where do you go from here ?

  Getting 10 millions hits per month is no small feat, it’s actually pretty fantastic. But if it means having to shell out USD150,000 just to be able to continue with Google Analytics, we feel your problem could be fixed with Matomo Cloud. You could then put the rest of the money you save to better use.

  If you choose Matomo, you now have the option to : 

  • Raise your data limits for a fraction of Google Analytics 360’s price
  • Get a comprehensive range of analytics features for the most impactful insights to ensure your website continues excelling
  • Get data that’s not sampled – meaning 100% accuracy in your reports
  • Migrate your data easily with the help of Matomo’s support team

  We’ll have you covered. 

  By sharing with you the options and advice we gave to Mark, we hope you’ll be able to find a solution that makes your life easier and solves the issue of data restrictions forever.

  The team at Matomo is here to help you every step of the way to ensure a stress-free transition from Google Analytics if that is what works best for you.

  For next steps, check out our live online Matomo demo and start your free 21-day trial.

  References :

  [1] Terms of Service. (2018, July 24). In Google Analytics Terms of Service. Retrieved June 12, 2019, from https://www.google.com/analytics/terms/us.html

  [2] Data limits. (n.d.). In Analytics Help Data limits. Retrieved June 12, 2019, from https://support.google.com/analytics/answer/1070983?hl=en

• Exceeded GA’s 10M hits data limit, now what ?

  21 juin 2019, par Joselyn Khor
  Exceeded GA’s 10M hits data limit, now what ? Matomo has the answers

  “Your data volume (1XXM hits) exceeds the limit of 10M hits per month as outlined in our Terms of Service. If you continue to exceed the limit, we will stop processing new data on XXX 21, 2019. Learn more about possible solutions.”

  Yikes. Alarm bells were ringing when a Google Analytics free user came to us faced with this notice. Let’s call him ‘Mark’. Mark had reached the limits on the data he could collect through Google Analytics and was shocked by the limited options available to fix the problem, without blowing the budget. The thoughts racing through his head were :

  • “What happens to all my data ?”
  • “What if Google starts charging USD150K now ?”

  Then he came across Matomo and decided to get in touch with our support team …

  “Can you fix this issue ?” he asked us.

  “Absolutely !” we said.

  We’ll get back to helping Mark in a minute. For now let’s go over why this was such a dilemma for him.

  In order to resolve this data limits issue, one of the solutions was for him to upgrade to Google Analytics 360, which meant shelling out USD150,000 per year for their 1 billion hits per month option. Going from free to USD150,000 was too much of a stretch for a growing company.

  “Your data volume (1XXM hits) exceeds the limit of 10M hits per month …”, what did this message mean ?

  With the free version, Mark could collect up to 10 million “hits” per month, per account. Going over meant Google Analytics could stop collecting any more data for free as outlined in their Terms.

  Google Analytics’ Terms of Service (2018, sec. 2) states, “Subject to Section 15, the Service is provided without charge to You for up to 10 million Hits per month per account.”[1]

  In general, what’s a "hit" ?

  Data being sent to Google Analytics. It can be a transaction, event, social interaction or pageview - these all produce what Google calls a “hit”.

  

  And their Analytics Help Data Limits (n.d.) support page makes clear that : “If a property sends more hits per month to Analytics than allowed by the Analytics Terms of Service, there is no assurance that the excess hits will be processed. If the property’s hit volume exceeds this limit, a warning may be displayed in the user interface and you may be prevented from accessing reports.”[2]

  

  Possible solutions

  So the possible solutions given by Google Analytics’ Data Limits support page were (also shown in image below) :

  • To pay USD150K to upgrade to Google Analytics 360
  • To send fewer hits by setting up sampling
  • Or choose the slightly less relevant option to upgrade mobile app tracking to Google Analytics for Firebase.

  Without the means to pay, the free version was fast becoming inaccessible for Mark as he was facing a future where he risked no longer having access to up-to-date data used in his business’ reporting.

  Mark was facing a problem that potentially didn’t have a cost-effective solution.

  

  So what can you really do about it ?

  This is where we can help provide some assistance. If you’re reading this article, we’ll assume you can relate to Mark and share with you the advice on options we gave him.

  Options :

  One option posed by Google is for you to send fewer hits by auditing your data collection processes

  If you really don’t have the budget, you’ll need to reassess your data collection priorities and go over your strategies to see what is necessary to track, and what isn’t.

  • Make sure you know what you’re tracking and why. Look at what websites are being tracked by Google and into what properties.
  • Go through what data you’re tracking and decide what is or isn’t of value.
  • Set up data sampling, this however, will lead to inaccurate data.

  From here you can start to course correct. If you’ve found data you’re not using for analysis, get rid of these events/pageviews in your Google Analytics.

  But the limitations here are that eventually, you’re going to run out of irrelevant metrics and everything you’re tracking will be essential. So you’ll hit another brick wall and return to the same situation.

  Option 2 – Ignore and continue using the free version of Google Analytics

  With this option, you’ll have to bear the business risks involved by basing decisions off of analytics reports that may or may not be updated. In this case, you may still get contacted about exceeding the limits. As the free service is provided for only up to 10 million hits, once you’ve gone over them, you’re violating what’s stipulated in the Terms of Service. 

  There’s also the warning that “… you may be prevented from accessing reports” (Data limits, n.d.). So while we may not know for certain what Google Analytics will do, in this case it may be better to be safe rather than sorry by acting quickly to resolve it. 

  Option 3 –The Matomo solution. Upgrade to a web analytics platform that can handle your demanding data requirements

  Save money while continuing to gain valuable insights by moving over to Matomo Analytics (recommended)

  This is where you can save up to USD130,000 a year. As well as that, the transition from Google Analytics to the Matomo Cloud is a seamless experience as setup and maintenance is taken care of by our experts.

  For example, you can get up to 15M pageviews for USD1,612.50/month (or USD19,350/year) on the Essentials plan.

  Or even 25M pageviews for USD2400/month (or USD24000/year) on the Business plan – which offers additional web analytics and conversion optimization resources.

  Matomo Cloud is a great option if you’re looking for a secure, cost-effective and powerful analytics solution. You also get what Google Analytics could never offer you : full control and ownership of your own data and privacy. 

  No need to worry about losing your Google Analytics data because …

  Now you can import your historic Google Analytics data directly into your Matomo with the Google Analytics Importer tool. Simply follow the step-by-step guide to get started for free.

  Along with savings you can get :

  • A solution for the data limits issue forever. You choose the right plan to suit your data needs and adapt as you continue growing
  • 100% accurate data (no data sampling)
  • 100% data ownership of all your information without signing away your data to a third party
  • Powerful web analytics and conversion optimization features
  • Matomo Tag Manager
  • Easy setup
  • Support from Matomo’s specialists

  Learn more about Matomo Cloud pricing.

  

  Or go for Matomo On-Premise

  If you have the in-house infrastructure to support self-hosting Matomo on your own servers then there’s also the option of Matomo On-Premise. Here you’ll get full security knowing the data is on your own servers. 

  Setup will also require technical knowledge. There will also be costs associated with acquiring your own servers, and keeping up with regular maintenance and updates. With On-Premise you get maximum flexibility, with no data limits whatsoever. But if you’re coming over from Google Analytics and don’t have the infrastructure and team to host On-Premise, the Matomo Cloud could be right for you.

  Learn more about Matomo On-Premise.

  Where do you go from here ?

  Getting 10 millions hits per month is no small feat, it’s actually pretty fantastic. But if it means having to shell out USD150,000 just to be able to continue with Google Analytics, we feel your problem could be fixed with Matomo Cloud. You could then put the rest of the money you save to better use.

  If you choose Matomo, you now have the option to : 

  • Raise your data limits for a fraction of Google Analytics 360’s price
  • Get a comprehensive range of analytics features for the most impactful insights to ensure your website continues excelling
  • Get data that’s not sampled – meaning 100% accuracy in your reports
  • Migrate your data easily with the help of Matomo’s support team

  We’ll have you covered. 

  By sharing with you the options and advice we gave to Mark, we hope you’ll be able to find a solution that makes your life easier and solves the issue of data restrictions forever.

  The team at Matomo is here to help you every step of the way to ensure a stress-free transition from Google Analytics if that is what works best for you.

  For next steps, why not check out our pricing page to see what could suit your needs !

  References :

  [1] Terms of Service. (2018, July 24). In Google Analytics Terms of Service. Retrieved June 12, 2019, from https://www.google.com/analytics/terms/us.html

  [2] Data limits. (n.d.). In Analytics Help Data limits. Retrieved June 12, 2019, from https://support.google.com/analytics/answer/1070983?hl=en