On other websites (9203)

• How to create a command – Introducing the Piwik Platform

  2 October 2014, by Thomas Steur — Development

  This is the next post of our blog series where we introduce the capabilities of the Piwik platform (our previous post was How to publish your plugin or theme on the Piwik Marketplace). This time you’ll learn how to create a new command. For this tutorial you will need to have basic knowledge of PHP.

  What is a command?

  A command can execute any task on the command line. Piwik provides currently about 50 commands via the Piwik Console. These commands let you start the archiver, change the number of available custom variables, enable the developer mode, clear caches, run tests and more. You could write your own command to sync users or websites with another system for instance.

  Getting started

  In this series of posts, we assume that you have already set up your development environment. If not, visit the Piwik Developer Zone where you’ll find the tutorial Setting up Piwik.

  To summarize the things you have to do to get setup:

  • Install Piwik (for instance via git).
  • Activate the developer mode: ./console development:enable --full.
  • Generate a plugin: ./console generate:plugin --name="MyCommandPlugin". There should now be a folder plugins/MyCommandPlugin.
  • And activate the created plugin under Settings => Plugins.

  Let’s start creating a command

  We start by using the Piwik Console to create a new command. As you can see there is even a command that lets you easily create a new command:

  ./console generate:command

  The command will ask you to enter the name of the plugin the created command should belong to. I will simply use the above chosen plugin name “MyCommandPlugin”. It will ask you for a command name as well. I will use “SyncUsers” in this example. There should now be a file plugins/MyCommandPlugin/Commands/Syncusers.php which contains already an example to get you started easily:

  class Syncusers extends ConsoleCommand
  {
      protected function configure()
      {
          $this->setName('mycommandplugin:syncusers');
          $this->setDescription('MyCommandPlugin');
          $this->addOption('name', null, InputOption::VALUE_REQUIRED, 'Your name:');
      }
   
      /**
       * Execute command like: ./console mycommandplugin:syncusers --name="The Piwik Team"
       */
      protected function execute(InputInterface $input, OutputInterface $output)
      {
          $name    = $input->getOption('name');
   
          $message = sprintf('Syncusers: %s', $name);
   
          $output->writeln($message);
      }
  }
  Download

  Any command that is placed in the “Commands” folder of your plugin will be available on the command line automatically. Therefore, the newly created command can now be executed via ./console mycommandplugin:syncusers --name="The Piwik Team".

  The code template explained

  protected function configure()
  {
      $this->setName('mycommandplugin:checkdatabase');
      $this->setDescription('MyCommandPlugin');
      $this->addOption('name', null, InputOption::VALUE_REQUIRED, 'Your name:');
  }
  Download

  As the name says the method configure lets you configure your command. You can define the name and description of your command as well as all the options and arguments you expect when executing it.

  protected function execute(InputInterface $input, OutputInterface $output)
  {
      $name    = $input->getOption('name');
      $message = sprintf('Syncusers: %s', $name);
      $output->writeln($message);
  }
  Download

  The actual task is defined in the execute method. There you can access any option or argument that was defined on the command line via $input and write anything to the console via $output argument.

  In case anything went wrong during the execution you should throw an exception to make sure the user will get a useful error message. Throwing an exception when an error occurs will make sure the command does exit with a status code different than 0 which can sometimes be important.

  Advanced features

  The Piwik Console is based on the powerful Symfony Console component. For instance you can ask a user for any interactive input, you can use different output color schemes and much more. If you are interested in learning more all those features have a look at the Symfony console website.

  How to test a command

  After you have created a command you are surely wondering how to test it. Ideally, the actual command is quite short as it acts like a controller. It should only receive the input values, execute the task by calling a method of another class and output any useful information. This allows you to easily create a unit or integration test for the classes behind the command. We will cover this topic in one of our future blog posts. Just one hint: You can use another command ./console generate:test to create a test. If you want to know how to test a command have a look at the Testing Commands documentation.

  Publishing your Plugin on the Marketplace

  In case you want to share your commands with other Piwik users you can do this by pushing your plugin to a public GitHub repository and creating a tag. Easy as that. Read more about how to distribute a plugin and best practices when publishing a plugin.

  Isn’t it easy to create a command? We never even created a file! If you have any feedback regarding our APIs or our guides in the Developer Zone feel free to send it to us.

• ValueError: I/O operation on closed file when making animation

  3 July 2018, by user3851187

  I am using matplotlib and ffmpeg to do some animations. I usually code on a remote server because the code runs faster; we are having some issues making animations on the remote server. Here is an example of code that works perfectly on my local mac but does not work remotely.

  import matplotlib as mpl
  
  mpl.use('agg')
  
  import matplotlib as mpl 
  
  from matplotlib import animation
  
  import pylab
  
  
  
  def init():
  
      pylab.plot(pylab.arange(10), [0]*10)
  
  
  
  def redraw(frame):
  
      pylab.plot(pylab.arange(10), pylab.arange(10) * frame)
  
  
  
  fig = pylab.figure()
  
  ani = animation.FuncAnimation(fig, redraw, frames=10, interval=1000, init_func=init)
  
  ani.save('animation.mp4')

  I get the animation I want on my local machine (macOS Sierra). When I run it on the remote host (Debian GNU/Linux 8 (jessie)), I get the following error message after 5 frames

  Traceback (most recent call last):
  
    File "animation.py", line 14, in <module>
  
      ani.save('animation.mp4')
  
    File "/usr/local/lib/python2.7/dist-packages/matplotlib/animation.py", line 1200, in save
  
      writer.grab_frame(**savefig_kwargs)
  
    File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
  
      self.gen.throw(type, value, traceback)
  
    File "/usr/local/lib/python2.7/dist-packages/matplotlib/animation.py", line 241, in saving
  
      self.finish()
  
    File "/usr/local/lib/python2.7/dist-packages/matplotlib/animation.py", line 367, in finish
  
      self.cleanup()
  
    File "/usr/local/lib/python2.7/dist-packages/matplotlib/animation.py", line 405, in cleanup
  
      out, err = self._proc.communicate()
  
    File "/usr/local/lib/python2.7/dist-packages/subprocess32.py", line 724, in communicate
  
      stdout, stderr = self._communicate(input, endtime, timeout)
  
    File "/usr/local/lib/python2.7/dist-packages/subprocess32.py", line 1535, in _communicate
  
      orig_timeout)
  
    File "/usr/local/lib/python2.7/dist-packages/subprocess32.py", line 1591, in _communicate_with_poll
  
      register_and_append(self.stdout, select_POLLIN_POLLPRI)
  
    File "/usr/local/lib/python2.7/dist-packages/subprocess32.py", line 1570, in register_and_append
  
      poller.register(file_obj.fileno(), eventmask)
  
  ValueError: I/O operation on closed file
  
  </module>

  My local machine uses matplotlib version 2.0.0; the remote machine uses matplotlib version 2.2.2

  On my local machine I have ffmpeg version 3.2.4

  $ ffmpeg -version
  
  ffmpeg version 3.2.4 Copyright (c) 2000-2017 the FFmpeg developers
  
  built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
  
  configuration: --prefix=/usr/local/Cellar/ffmpeg/3.2.4 --enable-shared - 
  
  -enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables 
  
  --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable libmp3lame --enable-libx264 --enable-libxvid --enable-opencl --disable-lzma --enable-vda
  
  libavutil      55. 34.101 / 55. 34.101
  
  libavcodec     57. 64.101 / 57. 64.101
  
  libavformat    57. 56.101 / 57. 56.101
  
  libavdevice    57.  1.100 / 57.  1.100
  
  libavfilter     6. 65.100 /  6. 65.100
  
  libavresample   3.  1.  0 /  3.  1.  0
  
  libswscale      4.  2.100 /  4.  2.100
  
  libswresample   2.  3.100 /  2.  3.100
  
  libpostproc    54.  1.100 / 54.  1.100 

  On the remote host i have ffmpeg version 4.0.1

  ffmpeg -version
  
  ffmpeg version 4.0.1 Copyright (c) 2000-2018 the FFmpeg developers
  
  built with gcc 4.9.2 (Debian 4.9.2-10+deb8u1)
  
  configuration: --prefix=/usr/local
  
  libavutil      56. 14.100 / 56. 14.100
  
  libavcodec     58. 18.100 / 58. 18.100
  
  libavformat    58. 12.100 / 58. 12.100
  
  libavdevice    58.  3.100 / 58.  3.100
  
  libavfilter     7. 16.100 /  7. 16.100
  
  libswscale      5.  1.100 /  5.  1.100
  
  libswresample   3.  1.100 /  3.  1.100

  If I recall correctly I installed ffmpeg locally through homebrew; I have the anaconda distribution of python. On the remote machine we have the default version of python that comes with Jessie; I’m not sure how the sysadmin installed ffmpeg.

  I am by no means an expert on ffmpeg, but I have generally never had issues with making animations in matplotlib on my local machine and I would really like to be able to make videos more quickly on the remote machine. Any help would be appreciated!

  Edit
  On the remote machine, the animation works if I use avconv as the writer instead of ffmpeg. I installed avconv locally...which led me to get the same ffmpeg issues locally (probably due to updating shared dependencies). However, I uninstalled ffmpeg and reinstalled it with x264 codec enables Animations in ipython (jupyter) notebook - ValueError: I/O operation on closed file

• What is Google Analytics data sampling and what’s so bad about it?

  16 August 2019, by Joselyn Khor — Analytics Tips, Development
  What is Google Analytics data sampling, and what’s so bad about it?

  Google (2019) explains what data sampling is:

  “In data analysis, sampling is the practice of analysing a subset of all data in order to uncover the meaningful information in the larger data set.”[1]

  This is basically saying instead of analysing all of the data, there’s a threshold on how much data is analysed and any data after that will be an assumption based on patterns.

  Google’s (2019) data sampling thresholds:

  Ad-hoc queries of your data are subject to the following general thresholds for sampling:
  [Google] Analytics Standard: 500k sessions at the property level for the date range you are using
  [Google] Analytics 360: 100M sessions at the view level for the date range you are using (para. 3) [2]

  This threshold is limiting because your data in GA may become more inaccurate as the traffic to your website increases.

  Say you’re looking through all your traffic data from the last year and find you have 5 million page views. Only 500K of that 5 million is accurate! The data for the remaining 4.5 million (90%) is an assumption based on the 500K sample size.

  This is a key weapon Google uses to sell to large businesses. In order to increase that threshold for more accurate reporting, upgrading to premium Google Analytics 360 for approximately US$150,000 per year seems to be the only choice.

  What’s so bad about data sampling?

  It’s unfair to say sampled data is to be disregarded completely. There is a calculation ensuring it is representative and can allow you to get good enough insights. However, we don’t encourage it as we don’t just want “good enough” data. We want the actual facts.

  In a recent survey sent to Matomo customers, we found a large proportion of users switched from GA to Matomo due to the data sampling issue.

  The two reasons why data sampling isn’t preferable: 

  1. If the selected sample size is too small, you won’t get a good representative of all the data. 
  2. The bigger your website grows, the more inaccurate your reports will become.

  An example of why we don’t fully trust sampled data is, say you have an ecommerce store and see your GA revenue reports aren’t matching the actual sales data, due to data sampling. In GA you may be seeing revenue for the month as $1 million, instead of actual sales of $800K.

  The sampling here has caused an inaccuracy that could have negative financial implications. What you get in the GA report is an estimated dollar figure rather than the actual sales. Making decisions based on inaccurate data can be costly in this case. 

  Another disadvantage to sampled data is that you might be missing out on opportunities you would’ve noticed if you were given a view of the whole. E.g. not being able to see real patterns occurring due to the data already being predicted. 

  By not getting a chance to see things as they are and only being able to jump to the conclusions and assumptions made by GA is risky. The bigger your business grows, the less you can risk making business decisions based on assumptions that could be inaccurate. 

  If you feel you could be missing out on opportunities because your GA data is sampled data, get 100% accurately reported data. 

  The benefits of 100% accurate data

  Matomo doesn’t use data sampling on any of our products or plans. You get to see all of your data and not a sampled data set.

  Data quality is necessary for high impact decision-making. It’s hard to make strategic changes if you don’t have confidence that your data is reliable and accurate.

  Learn about how Matomo is a serious contender to Google Analytics 360. 

  Now you can import your Google Analytics data directly into your Matomo

  If you’re wanting to make the switch to Matomo but worried about losing all your historic Google Analytics data, you can now import this directly into your Matomo with the Google Analytics Importer tool.

  
  Take the challenge!

  Compare your Google Analytics data (sampled data) against your Matomo data, or if you don’t have Matomo data yet, sign up to our 30-day free trial and start tracking!

  References:

  [1 & 2] About data sampling. (2019). In Analytics Help About data sampling. Retrieved August 14, 2019, from https://support.google.com/analytics/answer/2637192