Recherche avancée

Sur d’autres sites (6121)

• Finding Optimal Code Coverage

  7 mars 2012, par Multimedia Mike — Programming

  A few months ago, I published a procedure for analyzing code coverage of the test suites exercised in FFmpeg and Libav. I used it to add some more tests and I have it on good authority that it has helped other developers fill in some gaps as well (beginning with students helping out with the projects as part of the Google Code-In program). Now I’m wondering about ways to do better.

  Current Process
  When adding a test that depends on a sample (like a demuxer or decoder test), it’s ideal to add a sample that’s A) small, and B) exercises as much of the codebase as possible. When I was studying code coverage statistics for the WC4-Xan video decoder, I noticed that the sample didn’t exercise one of the 2 possible frame types. So I scouted samples until I found one that covered both types, trimmed the sample down, and updated the coverage suite.

  I started wondering about a method for finding the optimal test sample for a given piece of code, one that exercises every code path in a module. Okay, so that’s foolhardy in the vast majority of cases (although I was able to add one test spec that pushed a module’s code coverage from 0% all the way to 100% — but the module in question only had 2 exercisable lines). Still, given a large enough corpus of samples, how can I find the smallest set of samples that exercise the complete codebase ?

  This almost sounds like an NP-complete problem. But why should that stop me from trying to find a solution ?

  Science Project
  Here’s the pitch :

  • Instrument FFmpeg with code coverage support
  • Download lots of media to exercise a particular module
  • Run FFmpeg against each sample and log code coverage statistics
  • Distill the resulting data in some meaningful way in order to obtain more optimal code coverage

  That first step sounds harsh– downloading lots and lots of media. Fortunately, there is at least one multimedia format in the projects that tends to be extremely small : ANSI. These are files that are designed to display elaborate scrolling graphics using text mode. Further, the FATE sample currently deployed for this test (TRE_IOM5.ANS) only exercises a little less than 50% of the code in libavcodec/ansi.c. I believe this makes the ANSI video decoder a good candidate for this experiment.

  Procedure
  First, find a site that hosts a lot ANSI files. Hi, sixteencolors.net. This site has lots (on the order of 4000) artpacks, which are ZIP archives that contain multiple ANSI files (and sometimes some other files). I scraped a list of all the artpack names.

  In an effort to be responsible, I randomized the list of artpacks and downloaded periodically and with limited bandwidth ('wget --limit-rate=20k').

  Run ‘gcov’ on ansi.c in order to gather the full set of line numbers to be covered.

  For each artpack, unpack the contents, run the instrumented FFmpeg on each file inside, run ‘gcov’ on ansi.c, and log statistics including the file’s size, the file’s location (artpack.zip:filename), and a comma-separated list of line numbers touched.

  Definition of ‘Optimal’
  The foregoing procedure worked and yielded useful, raw data. Now I have to figure out how to analyze it.

  I think it’s most desirable to have the smallest files (in terms of bytes) that exercise the most lines of code. To that end, I sorted the results by filesize, ascending. A Python script initializes a set of all exercisable line numbers in ansi.c, then iterates through each each file’s stats line, adding the file to the list of candidate samples if its set of exercised lines can remove any line numbers from the overall set of lines. Ideally, that set of lines should devolve to an empty set.

  I think a second possible approach is to find the single sample that exercises the most code and then proceed with the previously described method.

  Initial Results
  So far, I have analyzed 13324 samples from 357 different artpacks provided by sixteencolors.net.

  Using the first method, I can find a set of samples that covers nearly 80% of ansi.c :

  <br />
0 bytes: bad-0494.zip:5<br />
1 bytes: grip1293.zip:-ANSI---.---<br />
1 bytes: pur-0794.zip:.<br />
2 bytes: awe9706.zip:-ANSI───.───<br />
61 bytes: echo0197.zip:-(ART)-<br />
62 bytes: hx03.zip:HX005.DAT<br />
76 bytes: imp-0494.zip:IMPVIEW.CFG<br />
82 bytes: ice0010b.zip:_cont'd_.___<br />
101 bytes: bdp-0696.zip:BDP2.WAD<br />
112 bytes: plain12.zip:--------.---<br />
181 bytes: ins1295v.zip:-°VGA°-.  н<br />
219 bytes: purg-22.zip:NEM-SHIT.ASC<br />
289 bytes: srg1196.zip:HOWTOREQ.JNK<br />
315 bytes: karma-04.zip:FASHION.COM<br />
318 bytes: buzina9.zip:ox-rmzzy.ans<br />
411 bytes: solo1195.zip:FU-BLAH1.RIP<br />
621 bytes: ciapak14.zip:NA-APOC1.ASC<br />
951 bytes: lght9404.zip:AM-TDHO1.LIT<br />
1214 bytes: atb-1297.zip:TX-ROKL.ASC<br />
2332 bytes: imp-0494.zip:STATUS.ANS<br />
3218 bytes: acepak03.zip:TR-STAT5.ANS<br />
6068 bytes: lgc-0193.zip:LGC-0193.MEM<br />
16778 bytes: purg-20.zip:EZ-HIR~1.JPG<br />
20582 bytes: utd0495.zip:LT-CROW3.ANS<br />
26237 bytes: quad0597.zip:MR-QPWP.GIF<br />
29208 bytes: mx-pack17.zip:mx-mobile-source-logo.jpg<br />
----<br />
109440 bytes total<br />

  A few notes about that list : Some of those filenames are comprised primarily of control characters. 133t, and all that. The first file is 0 bytes. I wondered if I should discard 0-length files but decided to keep those in, especially if they exercise lines that wouldn’t normally be activated. Also, there are a few JPEG and GIF files in the set. I should point out that I forced the tty demuxer using -f tty and there isn’t much in the way of signatures for this format. So, again, whatever exercises more lines is better.

  Using this same corpus, I tried approach 2– which single sample exercises the most lines of the decoder ? Answer : blde9502.zip:REQUEST.EXE. Huh. I checked it out and ‘file’ ID’s it as a MS-DOS executable. So, that approach wasn’t fruitful, at least not for this corpus since I’m forcing everything through this narrow code path.

  Think About The Future
  Where can I take this next ? The cloud ! I have people inside the search engine industry who have furnished me with extensive lists of specific types of multimedia files from around the internet. I also see that Amazon Web Services Elastic Compute Cloud (AWS EC2) instances don’t charge for incoming bandwidth.

  I think you can see where I’m going with this.

  See Also :

  • Running this experiment on 4000 artpacks / 146,000 files

• Piwik 3 Development Update #1 – New UI design, API changes & release date

  1er septembre 2016, par Thomas Steur — Community, Development

  Over the last months, we have been actively working on the Piwik 3 release and we want to introduce you to the changes that we have done so far. They include a new and modern UI redesign, new widgets for your dashboard, and technical improvements under the hood.

  New Piwik 3 UI

  Based on Material Design we have revamped the user interface which is now more responsive, more usable and faster. While the UI is not yet finished, we invite you to check it out already and to let us know what you think.

  

  This new Piwik material design is a visual language that synthesizes classic principles of good design with the innovation and possibility of technology and science.

  More responsive

  Piwik 3 will look and feel much better on your mobile phone and tablet. Many elements have been improved : the menus, the segment editor and dashboard selector, the widgets, the settings pages and most other pages so you can fully experience and enjoy Piwik on any device !

  

  Improved usability

  We have updated the menu structure, the dashboard selector as well as the footer in reports to make your life easier when using Piwik. We love to keep these complicated things simple. There are also many other tiny improvements that you will appreciate.

  

  The Zen Mode lets you view in full screen your analytics reports and dashboards. Users love this feature and it can now be accessed by pressing “z” key.

  

  Faster

  To make the Piwik interface faster, we have refactored most of our CSS, HTML and JavaScript and moved more and more of our code into the client. As a result, Piwik now needs to reload the page much less often ! For example when you change the date or change the segment, Piwik will now load the reports instantly. To improve performance even further, Piwik will now load multiple reports on a single page in parallel.

  Native fonts

  Over the last months more and more web services have started using system fonts, and so will Piwik 3. System fonts look better, improve language support and give you a more native, familiar feeling.

  For more details and screenshots have a look at the pull request for the Piwik 3 UI update.

  New widgets

  With the “Latest Piwik Plugin Updates” widget you won’t miss any newly added or updated plugin anymore. This will help you learn about and discover the useful plugins and themes available on the Marketplace.

  

  Super users can now see at a glance the current state of the Piwik system, thanks to the new “System Check” and “System Summary” widgets.

  

  API Changes

  Piwik is the leading open analytics platform, highly customisable and extensible thanks to a flexible plugins architecture and a design based on APIs. In this new major Piwik 3 release, we significantly improve the foundation of our open platform and several of its core APIs.

  The new Widgets and Reporting API makes it possible to add reports and widgets to any existing reporting page. In the past, reporting pages had to be created manually which was time consuming and it was hard to maintain a consistent look across different reporting pages. Now reporting pages are generated automatically by the Piwik platform.

  The Plugin Settings API was changed to improve performance and to let plugin developers customize the Websites Manager. This is one step towards our goal to let users not only manage websites but also mobile apps, cars, coffee machines or any other thing.

  To see a full list of changes in the Piwik 3 analytics platform, have a look at the developer changelog.

  Developer docs

  The Piwik Developer Zone is full of guides and API references to help developers understand, integrate and extend Piwik. As some APIs have changed in the Piwik 3 release it is now possible to select the Piwik version in the top right corner of the developer zone.

  

  We are updating guides for Piwik 3 and added a Piwik 2 -> Piwik 3 Migration Guide for plugin developers. Many other guides were updated such as Menus, Custom Report and Widgets.

  Release date

  The first Piwik 3 beta release will be available in the next four weeks. Beta testers automatically receive the update if they are subscribed to the “Latest Beta” release channel. The final Piwik 3 release will be ready before the end of the year. If you want to give it a try, you can either use Piwik from Git and check out the “3.x-dev” branch, or download Piwik 3 from GitHub.

  Closing thoughts

  With faster & more beautiful reports, better APIs and more stability, Piwik 3 is a big step forward for all Piwik users. As our mission is to create, as a community, the leading international open source web analytics platform that gives every user full control of their data, we are very excited to introduce you to this upcoming release.

  We now offer Custom Development services if you like to sponsor a new feature or bug fix, or if you have any custom requirements. And if you use Piwik Analytics to grow your business and need quality help, contact the Piwik analytics experts to get started.

  Until our next Piwik 3 dev update, Happy analysis !

• Multiprocess FATE Revisited

  26 juin 2010, par Multimedia Mike — FATE Server, Python

  I thought I had brainstormed a simple, elegant, multithreaded, deadlock-free refactoring for FATE in a previous post. However, I sort of glossed over the test ordering logic which I had not yet prototyped. The grim, possibly deadlock-afflicted reality is that the main thread needs to be notified as tests are completed. So, the main thread sends test specs through a queue to be executed by n tester threads and those threads send results to a results aggregator thread. Additionally, the results aggregator will need to send completed test IDs back to the main thread.

  
  
  

  But when I step back and look at the graph, I can’t rationalize why there should be a separate results aggregator thread. That was added to cut down on deadlock possibilities since the main thread and the tester threads would not be waiting for data from each other. Now that I’ve come to terms with the fact that the main and the testers need to exchange data in realtime, I think I can safely eliminate the result thread. Adding more threads is not the best way to guard against race conditions and deadlocks. Ask xine.

  
  
  

  I’m still hung up on the deadlock issue. I have these queues through which the threads communicate. At issue is the fact that they can cause a thread to block when inserting an item if the queue is "full". How full is full ? Immaterial ; seeking to answer such a question is not how you guard against race conditions. Rather, it seems to me that one side should be doing non-blocking queue operations.

  This is how I’m planning to revise the logic in the main thread :

  test_set = set of all tests to execute
  tests_pending = test_set
  tests_blocked = empty set
  tests_queue = multi-consumer queue to send test specs to tester threads
  results_queue = multi-producer queue through which tester threads send results
  while there are tests in tests_pending :
    pop a test from test_set
    if test depends on any tests that appear in tests_pending :
      add test to tests_blocked
    else :
      add test to tests_queue in a non-blocking manner
      if tests_queue is full, add test to tests_blocked
    while there are results in the results_queue :
  
      get a result from result_queue in non-blocking manner
  
      remove the corresponding test from tests_pending
  if tests_blocked is non-empty :
  
    sleep for 1 second
  
    test_set = tests_blocked
  
    tests_blocked = empty set
  
  else :
  
    insert n shutdown signals, one from each thread
  go to the top of the loop and repeat until there are no more tests
  while there are results in the results_queue :
  
    get a result from result_queue in a blocking manner
  

  Not mentioned in the pseudocode (so it doesn’t get too verbose) is logic to check whether the retrieved test result is actually an end-of-thread signal. These are accounted and the whole test process is done when one is received for each thread.

  On the tester thread side, it’s safe for them to do blocking test queue retrievals and blocking result queue insertions. The reason for the 1-second delay before resetting tests_blocked and looping again is because I want to guard against the situation where tests A and B are to be run, A depends of B running first, and while B is running (and happens to be a long encoding test), the main thread is spinning about, obsessively testing whether it’s time to insert A into the tests queue.

  It all sounds just crazy enough to work. In fact, I coded it up and it does work, sort of. The queue gets blocked pretty quickly. Instead of sleeping, I decided it’s better to perform the put operation using a 1-second timeout.

  Still, I’m paranoid about the precise operation of the IPC queue mechanism at work here. What happens if I try to stuff in a test spec that’s a bit too large ? Will the module take whatever I give it and serialize it through the queue as soon as it can ? I think an impromptu science project is in order.

  big-queue.py :

  PLAIN TEXT

  PYTHON :

  1. # !/usr/bin/python
  2.  
  3. import multiprocessing
  4. import Queue
  5.  
  6. def f(q) :
  7.   str = q.get()
  8.   print "reader function got a string of %d characters" % (len(str))
  9.  
  10. q = multiprocessing.Queue()
  11. p = multiprocessing.Process(target=f, args=(q,))
  12. p.start()
  13. try :
  14.   q.put_nowait(’a’ * 100000000)
  15. except Queue.Full :
  16.   print "queue full"

  $ ./big-queue.py
  reader function got a string of 100000000 characters
  

  Since 100 MB doesn’t even make it choke, FATE’s little test specs shouldn’t pose any difficulty.