Piwik

Les articles publiés sur le site

• Track API calls in Node.js with Piwik

  25 juin 2014, par Frederic Hemberger — Community, API, Node.js

  When using Piwik for analytics, sometimes you don’t want to track only your website’s visitors. Especially as modern web services usually offer RESTful APIs, why not use Piwik to track those requests as well? It really gives you a more accurate view on how users interact with your services: In which ways do your clients use your APIs compared to your website? Which of your services are used the most? And what kind of tools are consuming your API?

  If you’re using Node.js as your application platform, you can use piwik-tracker. It’s a lightweight wrapper for Piwik’s own Tracking HTTP API, which helps you tracking your requests.

  First, start with installing piwik-tracker as a dependency for your project:

  npm install piwik-tracker --save
  

  Then create a new tracking instance with your Piwik URL and the site ID of the project you want to track. As Piwik requires a fully qualified URL for analytics, add it in front of the actual request URL.

  var PiwikTracker = require('piwik-tracker');
  
  // Initialize with your site ID and Piwik URL
  var piwik = new PiwikTracker(1, 'http://mywebsite.com/piwik.php');
  
  // Piwik works with absolute URLs, so you have to provide protocol and hostname
  var baseUrl = 'http://example.com';
  
  // Track a request URL: 
  piwik.track(baseUrl + req.url);
  

  Of cause you can do more than only tracking simple URLs: All parameters offered by Piwik’s Tracking HTTP API Reference are supported, this also includes custom variables. During Piwik API calls, those are referenced as JSON string, so for better readability, you should use JSON.stringify({}) instead of manual encoding.

  piwik.track({
      // The full request URL
      url: baseUrl + req.url,
  
      // This will be shown as title in your Piwik backend
      action_name: 'API call',
  
      // User agent and language settings of the client
      ua: req.header('User-Agent'),
      lang: req.header('Accept-Language'),
  
      // Custom request variables
      cvar: JSON.stringify({
        '1': ['API version', 'v1'],
        '2': ['HTTP method', req.method]
      })
  });
  

  As you can see, you can pass along arbitrary fields of a Node.js request object like HTTP header fields, status code or request method (GET, POST, PUT, etc.) as well. That should already cover most of your needs.

  But so far, all requests have been tracked with the IP/hostname of your Node.js application. If you also want the API user’s IP to show up in your analytics data, you have to override Piwik’s default setting, which requires your secret Piwik token:

  function getRemoteAddr(req) {
      if (req.ip) return req.ip;
      if (req._remoteAddress) return req._remoteAddress;
      var sock = req.socket;
      if (sock.socket) return sock.socket.remoteAddress;
      return sock.remoteAddress;
  }
  
  piwik.track({
      // …
      token_auth: '<YOUR SECRET API TOKEN>',
      cip: getRemoteAddr(req)
  });
  

  As we have now collected all the values that we wanted to track, we’re basically done. But if you’re using Express or restify for your backend, we can still go one step further and put all of this together into a custom middleware, which makes tracking requests even easier.

  First we start off with the basic code of our new middleware and save it as lib/express-piwik-tracker.js:

  // ./lib/express-piwik-tracker.js
  var PiwikTracker = require('piwik-tracker');
  
  function getRemoteAddr(req) {
      if (req.ip) return req.ip;
      if (req._remoteAddress) return req._remoteAddress;
      var sock = req.socket;
      if (sock.socket) return sock.socket.remoteAddress;
      return sock.remoteAddress;
  }
  
  exports = module.exports = function analytics(options) {
      var piwik = new PiwikTracker(options.siteId, options.piwikUrl);
  
      return function track(req, res, next) {
          piwik.track({
              url: options.baseUrl + req.url,
              action_name: 'API call',
              ua: req.header('User-Agent'),
              lang: req.header('Accept-Language'),
              cvar: JSON.stringify({
                '1': ['API version', 'v1'],
                '2': ['HTTP method', req.method]
              }),
              token_auth: options.piwikToken,
              cip: getRemoteAddr(req)
  
          });
          next();
      }
  }
  

  Now to use it in our application, we initialize it in our main app.js file:

  // app.js
  var express      = require('express'),
      piwikTracker = require('./lib/express-piwik-tracker.js'),
      app          = express();
  
  // This tracks ALL requests to your Express application
  app.use(piwikTracker({
      siteId    : 1,
      piwikUrl  : 'http://mywebsite.com/piwik.php',
      baseUrl   : 'http://example.com',
      piwikToken: '<YOUR SECRET API TOKEN>'
  }));
  

  This will now track each request going to every URL of your API. If you want to limit tracking to a certain path, you can also attach it to a single route instead:

  var tracker = piwikTracker({
      siteId    : 1,
      piwikUrl  : 'http://mywebsite.com/piwik.php',
      baseUrl   : 'http://example.com',
      piwikToken: '<YOUR SECRET API TOKEN>'
  });
  
  router.get('/only/track/me', tracker, function(req, res) {
      // Your code that handles the route and responds to the request
  });
  

  And that’s everything you need to track your API users alongside your regular website users.

• Track API calls in Node.js with Piwik

  25 juin 2014, par Frederic Hemberger — Community, API, Node.js

  When using Piwik for analytics, sometimes you don’t want to track only your website’s visitors. Especially as modern web services usually offer RESTful APIs, why not use Piwik to track those requests as well? It really gives you a more accurate view on how users interact with your services: In which ways do your clients use your APIs compared to your website? Which of your services are used the most? And what kind of tools are consuming your API?

  If you’re using Node.js as your application platform, you can use piwik-tracker. It’s a lightweight wrapper for Piwik’s own Tracking HTTP API, which helps you tracking your requests.

  First, start with installing piwik-tracker as a dependency for your project:

  npm install piwik-tracker --save
  

  Then create a new tracking instance with your Piwik URL and the site ID of the project you want to track. As Piwik requires a fully qualified URL for analytics, add it in front of the actual request URL.

  var PiwikTracker = require('piwik-tracker');
  
  // Initialize with your site ID and Piwik URL
  var piwik = new PiwikTracker(1, 'http://mywebsite.com/piwik.php');
  
  // Piwik works with absolute URLs, so you have to provide protocol and hostname
  var baseUrl = 'http://example.com';
  
  // Track a request URL: 
  piwik.track(baseUrl + req.url);
  

  Of cause you can do more than only tracking simple URLs: All parameters offered by Piwik’s Tracking HTTP API Reference are supported, this also includes custom variables. During Piwik API calls, those are referenced as JSON string, so for better readability, you should use JSON.stringify({}) instead of manual encoding.

  piwik.track({
      // The full request URL
      url: baseUrl + req.url,
  
      // This will be shown as title in your Piwik backend
      action_name: 'API call',
  
      // User agent and language settings of the client
      ua: req.header('User-Agent'),
      lang: req.header('Accept-Language'),
  
      // Custom request variables
      cvar: JSON.stringify({
        '1': ['API version', 'v1'],
        '2': ['HTTP method', req.method]
      })
  });
  

  As you can see, you can pass along arbitrary fields of a Node.js request object like HTTP header fields, status code or request method (GET, POST, PUT, etc.) as well. That should already cover most of your needs.

  But so far, all requests have been tracked with the IP/hostname of your Node.js application. If you also want the API user’s IP to show up in your analytics data, you have to override Piwik’s default setting, which requires your secret Piwik token:

  function getRemoteAddr(req) {
      if (req.ip) return req.ip;
      if (req._remoteAddress) return req._remoteAddress;
      var sock = req.socket;
      if (sock.socket) return sock.socket.remoteAddress;
      return sock.remoteAddress;
  }
  
  piwik.track({
      // …
      token_auth: '<YOUR SECRET API TOKEN>',
      cip: getRemoteAddr(req)
  });
  

  As we have now collected all the values that we wanted to track, we’re basically done. But if you’re using Express or restify for your backend, we can still go one step further and put all of this together into a custom middleware, which makes tracking requests even easier.

  First we start off with the basic code of our new middleware and save it as lib/express-piwik-tracker.js:

  // ./lib/express-piwik-tracker.js
  var PiwikTracker = require('piwik-tracker');
  
  function getRemoteAddr(req) {
      if (req.ip) return req.ip;
      if (req._remoteAddress) return req._remoteAddress;
      var sock = req.socket;
      if (sock.socket) return sock.socket.remoteAddress;
      return sock.remoteAddress;
  }
  
  exports = module.exports = function analytics(options) {
      var piwik = new PiwikTracker(options.siteId, options.piwikUrl);
  
      return function track(req, res, next) {
          piwik.track({
              url: options.baseUrl + req.url,
              action_name: 'API call',
              ua: req.header('User-Agent'),
              lang: req.header('Accept-Language'),
              cvar: JSON.stringify({
                '1': ['API version', 'v1'],
                '2': ['HTTP method', req.method]
              }),
              token_auth: options.piwikToken,
              cip: getRemoteAddr(req)
  
          });
          next();
      }
  }
  

  Now to use it in our application, we initialize it in our main app.js file:

  // app.js
  var express      = require('express'),
      piwikTracker = require('./lib/express-piwik-tracker.js'),
      app          = express();
  
  // This tracks ALL requests to your Express application
  app.use(piwikTracker({
      siteId    : 1,
      piwikUrl  : 'http://mywebsite.com/piwik.php',
      baseUrl   : 'http://example.com',
      piwikToken: '<YOUR SECRET API TOKEN>'
  }));
  

  This will now track each request going to every URL of your API. If you want to limit tracking to a certain path, you can also attach it to a single route instead:

  var tracker = piwikTracker({
      siteId    : 1,
      piwikUrl  : 'http://mywebsite.com/piwik.php',
      baseUrl   : 'http://example.com',
      piwikToken: '<YOUR SECRET API TOKEN>'
  });
  
  router.get('/only/track/me', tracker, function(req, res) {
      // Your code that handles the route and responds to the request
  });
  

  And that’s everything you need to track your API users alongside your regular website users.

• Help us Reset The Net today on June 5th

  5 juin 2014, par Piwik Core Team — Community, Meta

  This blog post explains why the Piwik project is joining ResetTheNet online protest and how you can help make a difference against mass surveillance. It also includes an infographic and links to useful resources which may be of interest to you.

  Snowden revelations, a year ago today

  On June 5, 2013 the Guardian newspaper published the first of Edward Snowden’s astounding revelations. It was the first of a continuous stream of stories that pointed out what we’ve suspected for a long time: that the world’s digital communications are being continuously spied upon by nation states with precious little oversight.

  Unfortunately, mass surveillance is affecting the internet heavily. The Internet is a powerful force that can promote democracy, innovation, and creativity, but it’s being subverted as a tool for government spying. That is why Piwik has decided to join Reset The Net.

  June 5, 2014 marks a new year: a year that will not just be about listening to the inside story of mass surveillance, but a new year of fighting back!

  How do I protect myself and others?

  Reset the Net is asking everyone to help by installing free software tools that are designed to protect your privacy on a computer or a mobile device.

  Reset the Net is also calling on websites and developers to add surveillance resistant features such as HTTPS and forward secrecy.

  Participate in ResetTheNet online protest

  Have you got your own website, blog or tumblr? Maybe you can show the Internet Defense League’s “Cat Signal!” on your website.Get the code now to run the Reset the Net splash screen or banner to help make privacy viral on June 5th.

  Message from Edward Snowden

  Evan from FFTF sent us this message from Edward Snowden and we thought we would share it with you:

  One year ago, we learned that the internet is under surveillance, and our activities are being monitored to create permanent records of our private lives — no matter how innocent or ordinary those lives might be.

  Today, we can begin the work of effectively shutting down the collection of our online communications, even if the US Congress fails to do the same. That’s why I’m asking you to join me on June 5th for Reset the Net, when people and companies all over the world will come together to implement the technological solutions that can put an end to the mass surveillance programs of any government. This is the beginning of a moment where we the people begin to protect our universal human rights with the laws of nature rather than the laws of nations.

  We have the technology, and adopting encryption is the first effective step that everyone can take to end mass surveillance. That’s why I am excited for Reset the Net — it will mark the moment when we turn political expression into practical action, and protect ourselves on a large scale.

  Join us on June 5th, and don’t ask for your privacy. Take it back.

  – Message by Edward Snowden

  ResetTheNet privacy pack infographic

  

  Additional Resources

  Configure Piwik for Security and Privacy

  • Turn on automatic SSL redirection in your Piwik.
  • Configure Piwik for advanced Privacy.
  • Best security practises for Piwik.

  More info

  • Learn why this matters (EFF)
  • Data Privacy Day – Why is privacy important? (Piwik)
  • Web Analytics Privacy (Piwik)

• Help us Reset The Net today on June 5th

  5 juin 2014, par Piwik Core Team — Community, Meta

  This blog post explains why the Piwik project is joining ResetTheNet online protest and how you can help make a difference against mass surveillance. It also includes an infographic and links to useful resources which may be of interest to you.

  Snowden revelations, a year ago today

  On June 5, 2013 the Guardian newspaper published the first of Edward Snowden’s astounding revelations. It was the first of a continuous stream of stories that pointed out what we’ve suspected for a long time: that the world’s digital communications are being continuously spied upon by nation states with precious little oversight.

  Unfortunately, mass surveillance is affecting the internet heavily. The Internet is a powerful force that can promote democracy, innovation, and creativity, but it’s being subverted as a tool for government spying. That is why Piwik has decided to join Reset The Net.

  June 5, 2014 marks a new year: a year that will not just be about listening to the inside story of mass surveillance, but a new year of fighting back!

  How do I protect myself and others?

  Reset the Net is asking everyone to help by installing free software tools that are designed to protect your privacy on a computer or a mobile device.

  Reset the Net is also calling on websites and developers to add surveillance resistant features such as HTTPS and forward secrecy.

  Participate in ResetTheNet online protest

  Have you got your own website, blog or tumblr? Maybe you can show the Internet Defense League’s “Cat Signal!” on your website.Get the code now to run the Reset the Net splash screen or banner to help make privacy viral on June 5th.

  Message from Edward Snowden

  Evan from FFTF sent us this message from Edward Snowden and we thought we would share it with you:

  One year ago, we learned that the internet is under surveillance, and our activities are being monitored to create permanent records of our private lives — no matter how innocent or ordinary those lives might be.

  Today, we can begin the work of effectively shutting down the collection of our online communications, even if the US Congress fails to do the same. That’s why I’m asking you to join me on June 5th for Reset the Net, when people and companies all over the world will come together to implement the technological solutions that can put an end to the mass surveillance programs of any government. This is the beginning of a moment where we the people begin to protect our universal human rights with the laws of nature rather than the laws of nations.

  We have the technology, and adopting encryption is the first effective step that everyone can take to end mass surveillance. That’s why I am excited for Reset the Net — it will mark the moment when we turn political expression into practical action, and protect ourselves on a large scale.

  Join us on June 5th, and don’t ask for your privacy. Take it back.

  – Message by Edward Snowden

  ResetTheNet privacy pack infographic

  

  Additional Resources

  Configure Piwik for Security and Privacy

  • Turn on automatic SSL redirection in your Piwik.
  • Configure Piwik for advanced Privacy.
  • Best security practises for Piwik.

  More info

  • Learn why this matters (EFF)
  • Data Privacy Day – Why is privacy important? (Piwik)
  • Web Analytics Privacy (Piwik)

• How Piwik uses Travis CI to deliver a reliable analytics platform to the community

  26 mai 2014, par Matthieu Aubry — Development, Meta

  In this post, we will explain how the Piwik project uses continuous integration to deliver a quality software platform to dozens of thousands of users worldwide. Read this post if you are interested in Piwik project, Quality Assurance or Automated testing.

  Why do we care about tests?

  Continuous Integration brings us agility and peace of mind. From the very beginning of the Piwik project, it was clear to us that writing and maintaining automated tests was a necessity, in order to create a successful open source software platform.

  Over the years we have invested a lot of time into writing and maintaining our tests suites. This work has paid off in so many ways! Piwik platform has fewer bugs, fewer regressions, and we are able to release new minor and major versions frequently.

  Which parts of Piwik software are automatically tested?

  • Piwik back-end in PHP5: we use PHPUnit to write and run our PHP tests: unit tests, integration tests, and plugin tests.
  • piwik.js Tracker: the JS tracker is included into all websites that use Piwik. For this reason, it is critical that piwik.js JavaScript tracker always works without any issue or regression. Our Javascript Tracker tests includes both unit and integration tests.
  • Piwik front-end: more recently we’ve started to write JavaScript tests for the user interface partially written in AngularJS.
  • Piwik front-end screenshots tests: after each change to Piwik, more than 150 different screenshots are automatically taken. For example, we take screenshots of each of the 8-step installation process, we take screenshots of the password reset workflow, etc. Each of these screenshot is then compared pixel by pixel, with the “expected” screenshot, and we can automatically detect whether the last code change has introduced an undesired visual change. Learn more about Piwik screenshot tests.

  How often do we run the tests?

  The tests are executed by Travis CI after each change to the Piwik source code. On average all our tests run 20 times per day. Whenever a Piwik developer pushes some code to Github, or when a community member issues a Pull request, Travis CI automatically runs the tests. In case some of the automated tests started failing after a change, the developer that has made the change is notified by email.

  Should I use Travis CI?

  Over the last six years, we have used various Continuous Integration servers such as Bamboo, Hudson, Jenkins… and have found that the Travis CI is the ideal continuous integration service for open source projects that are hosted on Github. Travis CI is free for open source projects and the Travis CI team is very friendly and reactive! If you work on commercial closed source software, you may also use Travis by signing up to Travis CI Pro.

  Summary

  Tests make the Piwik analytics platform better. Writing tests make Piwik contributors better developers. We save a lot of time and effort, and we are not afraid of change!

  Here is the current status of our builds:
  Main build:
  Screenshot tests build:

  PS: If you are a developer looking for a challenge, Piwik is hiring a software developer to join our engineering team in New Zealand or Poland.