Recherche avancée

Sur d’autres sites (10449)

• Stopping Referrer Spam

  13 mai 2015, par Piwik Core Team — Community

  In this blog post we explain what is Referrer spam, this new kind of spam that has recently appeared on the Internet. We also provide solutions to stop it and preserve the quality of your analytics data.

  What is Referrer Spam ?

  Referrer spam (also known as log spam or referrer bombing) is a kind of spamming aimed at web analytics tools. A spammer bot makes repeated web site requests using a fake referrer URL to the site the spammer wishes to advertise.

  Here is an example of referrer spam in action :

  

  Half of those referrers are spams, here are some well know spammers that you may have seen in your logs : buttons-for-you-website.com, best-seo-offer.com, semalt.com…

  The benefit for spammers is that their website will appear in analytics tools like Piwik or Google Analytics :

  • public analytics reports (or logs) will be indexed by search engines : links to the spammer’s website will improve its ranking
  • curious webmasters are likely to visit their referrers, thus bringing traffic to the spammer’s website

  How to deal with Referrer Spam ?

  Referrer spam is still new and analytics tools are all handling it differently.

  Referrer Spam in Piwik

  At Piwik we started working on mitigating Referrer spam more than a year ago. If you use Piwik and keep it up to date, you do not need to do anything.

  Referrer spammers are automatically excluded from your reports to keep your data clean and useful.

  New spammers are continuously detected and added to Piwik’s blacklist on each update. If you find a new spammer in your analytics data, you can even report it so that it is added to the Piwik’s open referrer blacklist and blocked for everyone.

  Referrer Spam in Google Analytics

  Google Analytics doesn’t offer any spam protection by default. It can however be configured manually using a custom Filter.

  To create a filter in Google Analytics go to the Admin section and click on All Filters. Create a new custom filter that excludes based on the Campaign Source field. In the Filter pattern enter the spammers domains you want to exclude (this is a regular expression) :

  

  If new spammers arise you will need to update this list. You can also use Piwik’s referrer blacklist to exclude all the spammers currently detected.

  Other Analytics Tools

  Many web analytics tools do not yet handle Referrer spam and when using these tools, you will often find a lot of spam data in your Referrer Websites analytics reports.

  If you use an analytics tool that does not exclude Referrer spam, we recommend to contact the vendor and ask them to implement a mechanism to remove these referrer spammers. As of today many analytics vendors still have not mitigated this issue.

  Public List of Referrer Spammers

  At Piwik with the help of our large community we have decided to tackle this growing spam issue. We have created a list of up to date referrer spammers that anyone can edit.

  The list is available in a simple text file on Github : github.com/piwik/referrer-spam-blacklist.

  The list is released under the Public Domain and anyone can use it within their applications to exclude referrer spammers.

  Many people have already contributed new spammers to the list. We invite you to use the list in your apps and websites and help us keep the list up to date !

  Let’s unite and fight the spammers together.

  Happy Analytics !

• Stopping Referrer Spam

  13 mai 2015, par Piwik Core Team — Community

  In this blog post we explain what is Referrer spam, this new kind of spam that has recently appeared on the Internet. We also provide solutions to stop it and preserve the quality of your analytics data.

  What is Referrer Spam ?

  Referrer spam (also known as log spam or referrer bombing) is a kind of spamming aimed at web analytics tools. A spammer bot makes repeated web site requests using a fake referrer URL to the site the spammer wishes to advertise.

  Here is an example of referrer spam in action :

  

  Half of those referrers are spams, here are some well know spammers that you may have seen in your logs : buttons-for-you-website.com, best-seo-offer.com, semalt.com…

  The benefit for spammers is that their website will appear in analytics tools like Piwik or Google Analytics :

  • public analytics reports (or logs) will be indexed by search engines : links to the spammer’s website will improve its ranking
  • curious webmasters are likely to visit their referrers, thus bringing traffic to the spammer’s website

  How to deal with Referrer Spam ?

  Referrer spam is still new and analytics tools are all handling it differently.

  Referrer Spam in Piwik

  At Piwik we started working on mitigating Referrer spam more than a year ago. If you use Piwik and keep it up to date, you do not need to do anything.

  Referrer spammers are automatically excluded from your reports to keep your data clean and useful.

  New spammers are continuously detected and added to Piwik’s blacklist on each update. If you find a new spammer in your analytics data, you can even report it so that it is added to the Piwik’s open referrer blacklist and blocked for everyone.

  Referrer Spam in Google Analytics

  Google Analytics doesn’t offer any spam protection by default. It can however be configured manually using a custom Filter.

  To create a filter in Google Analytics go to the Admin section and click on All Filters. Create a new custom filter that excludes based on the Campaign Source field. In the Filter pattern enter the spammers domains you want to exclude (this is a regular expression) :

  

  If new spammers arise you will need to update this list. You can also use Piwik’s referrer blacklist to exclude all the spammers currently detected.

  Other Analytics Tools

  Many web analytics tools do not yet handle Referrer spam and when using these tools, you will often find a lot of spam data in your Referrer Websites analytics reports.

  If you use an analytics tool that does not exclude Referrer spam, we recommend to contact the vendor and ask them to implement a mechanism to remove these referrer spammers. As of today many analytics vendors still have not mitigated this issue.

  Public List of Referrer Spammers

  At Piwik with the help of our large community we have decided to tackle this growing spam issue. We have created a list of up to date referrer spammers that anyone can edit.

  The list is available in a simple text file on Github : github.com/piwik/referrer-spam-blacklist.

  The list is released under the Public Domain and anyone can use it within their applications to exclude referrer spammers.

  Many people have already contributed new spammers to the list. We invite you to use the list in your apps and websites and help us keep the list up to date !

  Let’s unite and fight the spammers together.

  Happy Analytics !

• Xuggler decode H264 packets

  18 novembre 2015, par Roy Bean

  I’m making a test with xuggler, and I’m observing a behavior that I don’t understand.

  So I have a H264 frame, stored in a file and I want to decode it.
  Bassically I need of the IVideoPicture to then get a buffred image.

  for the same same packet I need of 10 tries until it retrieved that the IVideoPicture is complete.

  this is what ffmpeg does when reads the frame :

  Input #0, h264, from 'sample1.jpg':
  
  Duration: N/A, bitrate: N/A
  
  Stream #0:0: Video: h264 (Baseline), yuv420p, 352x288, 25 tbr, 1200k tbn, 50 tbc

  Can someone of xuggler or ffmpeg knowledge explain what is happen ?

  My code bellow :

  IContainer container = IContainer.make();
  
  
  
  RandomAccessFile aFile = new RandomAccessFile("c://sample1Frame/sample1.jpg","r");
  
  IContainerFormat format = IContainerFormat.make();
  
  format.setInputFormat("h264");
  
  container.open(aFile, IContainer.Type.READ, null);
  
  int numStreams = container.getNumStreams();
  
  
  
  
  
  IStreamCoder videoCoder = null;
  
  for(int i = 0; i < numStreams; i++)
  
  {
  
      IStream stream = container.getStream(i);
  
      // Get the pre-configured decoder that can decode this stream;
  
      IStreamCoder coder = stream.getStreamCoder();
  
      if(coder.getCodecType() == ICodec.Type.CODEC_TYPE_VIDEO)
  
      {
  
          System.out.println("coder"); 
  
      }
  
      videoCoder = coder;
  
  }
  
  
  
  // The timebase here is used as the camera frame rate    
  
  videoCoder.setTimeBase(IRational.make(25,1));
  
  
  
  // we need to tell the driver what video with and height to use
  
  videoCoder.setWidth(352);
  
  videoCoder.setHeight(288);
  
  
  
  if(videoCoder.open(null , null) < 0){
  
      System.err.println("Could not open video decoder for input container");
  
      return;
  
  }
  
  //      videoCoder.setFlag(IStreamCoder.Flags.FLAG_QSCALE,true);
  
  //      videoCoder.setFlag(IStreamCoder.Flags.FLAG2_FAST,true);
  
  
  
  IPacket packet = IPacket.make();
  
  //      long firstTimestampInStream = Global.NO_PTS;
  
  //      long systemClockStartTime = 0;
  
  int complete = 1;
  
  
  
  
  
  while(container.readNextPacket(packet) >= 0){
  
      int trys= 1;
  
      System.out.println("PCK = ");
  
      while(true)
  
      {
  
          trys++;
  
          System.out.println("trys = " + trys);
  
          System.out.println("packet " + packet.getSize());
  
          IVideoPicture picture = IVideoPicture.make(videoCoder.getPixelType(), videoCoder.getWidth(), videoCoder.getHeight());
  
  
  
  
  
          //TODO
  
          int offset = 0;
  
          while(offset < packet.getSize())
  
          {
  
              /*
  
               * Now, we decode the video, checking for any errors.
  
               * 
  
               */
  
              System.out.println("packet getTimeBase " + packet.getTimeBase());
  
              System.out.println("vc getTimeBase " + videoCoder.getTimeBase());
  
              System.out.println("vc getCodecID " + videoCoder.getCodecID().toString());
  
              int bytOffset = videoCoder.decodeVideo(picture, packet, offset);
  
              Logger.getLogger(VideoSamsung.class.getName()).log(Level.SEVERE,"bytOffset =  " + bytOffset);
  
  
  
              if(bytOffset < 0){
  
                  //                  videoCoder.close();
  
                  //                  //                  picture.setComplete(true, com.xuggle.xuggler.IPixelFormat.Type.YUV420P, 352, 288, packet.getPts());
  
                  //                  System.err.println("picture.isComplete()"+picture.isComplete());
  
                  //                                      BufferedImage sdf = Utils.videoPictureToImage(picture);
  
                  //                                      File outputfile = new File("c://sdf/image.jpg");
  
                  //                                      ImageIO.write(sdf, "jpg", outputfile);
  
                  Logger.getLogger(VideoSamsung.class.getName()).log(Level.SEVERE,"BUMMMMMM!!!!!!!!!!!!!!!!!!!");
  
                  break;
  
              }else{
  
                  //System.out.println(DatatypeConverter.printBase64Binary(videoCoder.getExtraData().getByteArray(0, videoCoder.getExtraData().getSize())));
  
              }
  
              offset += bytOffset;
  
              System.err.println("picture.isComplete()"+picture.isComplete());
  
  
  
              /*
  
               * Some decoders will consume data in a packet, but will not be able to construct
  
               * a full video picture yet.  Therefore you should always check if you
  
               * got a complete picture from the decoder
  
               */
  
  
  
          }//end while(offset < packet.getSize())
  
  
  
          if (picture.isComplete())
  
          {
  
              System.out.println("isComplete");
  
              IVideoPicture newPic = picture;
  
              BufferedImage sdf = Utils.videoPictureToImage(picture);
  
              File outputfile = new File("c://sdf/image_"+complete+".jpg");
  
              ImageIO.write(sdf, "jpg", outputfile);
  
              complete++;
  
              System.out.println("trys = " + trys);
  
              break;
  
              //              System.exit(0);
  
          }else{
  
              System.out.println("Picture is not Complete");
  
          }
  
  
  
          if(trys > 20){
  
              System.out.println("FAIL!!!!!!!!!!  trys = " + trys);
  
              break;
  
          }
  
      }
  
  }
  
  /*
  
   * Technically since we're exiting anyway, these will be cleaned up by 
  
   * the garbage collector... but because we're nice people and want
  
   * to be invited places for Christmas, we're going to show how to clean up.
  
   */
  
  if (videoCoder != null)
  
  {
  
      videoCoder.close();
  
      videoCoder = null;
  
  }
  
  if (container !=null)
  
  {
  
      container.close();
  
      container = null;
  
  }

  the console output :

      PCK = 
  
  trys = 2
  
  packet 1748
  
  packet getTimeBase 1/1200000
  
  vc getTimeBase 25/1
  
  vc getCodecID CODEC_ID_H264
  
  Nov 18, 2015 5:39:11 PM filepath main
  
  SEVERE: bytOffset =  1748
  
  Picture is not Complete
  
  trys = 3
  
  packet 1748
  
  packet getTimeBase 1/1200000
  
  vc getTimeBase 25/1
  
  vc getCodecID CODEC_ID_H264
  
  picture.isComplete()false
  
  Nov 18, 2015 5:39:11 PM filepath main
  
  SEVERE: bytOffset =  1748
  
  Picture is not Complete
  
  picture.isComplete()false
  
  trys = 4
  
  packet 1748
  
  packet getTimeBase 1/1200000
  
  vc getTimeBase 25/1
  
  vc getCodecID CODEC_ID_H264
  
  Nov 18, 2015 5:39:11 PM filepath main
  
  SEVERE: bytOffset =  1748
  
  picture.isComplete()false
  
  Picture is not Complete
  
  trys = 5
  
  packet 1748
  
  packet getTimeBase 1/1200000
  
  vc getTimeBase 25/1
  
  vc getCodecID CODEC_ID_H264
  
  Nov 18, 2015 5:39:11 PM filepath main
  
  SEVERE: bytOffset =  1748
  
  picture.isComplete()false
  
  Picture is not Complete
  
  trys = 6
  
  packet 1748
  
  packet getTimeBase 1/1200000
  
  vc getTimeBase 25/1
  
  vc getCodecID CODEC_ID_H264
  
  Nov 18, 2015 5:39:11 PM filepath main
  
  SEVERE: bytOffset =  1748
  
  picture.isComplete()false
  
  Picture is not Complete
  
  trys = 7
  
  packet 1748
  
  packet getTimeBase 1/1200000
  
  vc getTimeBase 25/1
  
  vc getCodecID CODEC_ID_H264
  
  Nov 18, 2015 5:39:11 PM filepath main
  
  SEVERE: bytOffset =  1748
  
  picture.isComplete()false
  
  Picture is not Complete
  
  trys = 8
  
  packet 1748
  
  packet getTimeBase 1/1200000
  
  vc getTimeBase 25/1
  
  vc getCodecID CODEC_ID_H264
  
  Nov 18, 2015 5:39:11 PM filepath main
  
  SEVERE: bytOffset =  1748
  
  picture.isComplete()false
  
  Picture is not Complete
  
  trys = 9
  
  packet 1748
  
  packet getTimeBase 1/1200000
  
  vc getTimeBase 25/1
  
  vc getCodecID CODEC_ID_H264
  
  Nov 18, 2015 5:39:11 PM filepath main
  
  SEVERE: bytOffset =  1748
  
  picture.isComplete()false
  
  Picture is not Complete
  
  trys = 10
  
  packet 1748
  
  packet getTimeBase 1/1200000
  
  vc getTimeBase 25/1
  
  vc getCodecID CODEC_ID_H264
  
  Nov 18, 2015 5:39:11 PM filepath main
  
  SEVERE: bytOffset =  1748
  
  picture.isComplete()true
  
  isComplete
  
  trys = 10