Recherche avancée

Sur d’autres sites (6899)

• FFMPEG - adding a nullsrc causes my script to report "1000 duplicate frames"

  14 juin 2020, par rossmcm

  I'm trying to add coloured rectangle highlights to a video, appearing at different locations and times. The highlights are on a 6x6 grid of 320x180 rectangles.

  



  Originally I didn't have the nullsrc=size=1920x1080 thinking that it would start with an empty image, but it seems that that causes it to make assumptions about where the input is coming from. So I added the nullsrc=size=1920x1080 to start with a transparent 1920x1080 image but this command returns a warning that 1000 duplicate frames have been produced and it keeps going past the end of the input video with no signs of stopping.

  



  ffmpeg -y \       
  -i "Input.mp4" \           
  -filter_complex \ 
 "nullsrc=size=1920x1080, drawbox=x=(3-1)*320:y=(3-1)*180:w=320:h=180:t=7:c=cyan,   fade=in:st=10:d=1:alpha=1, fade=out:st=40:d=1:alpha=1[tmp1]; \
  nullsrc=size=1920x1080, drawbox=x=(4-1)*320:y=(4-1)*180:w=320:h=180:t=7:c=blue,   fade=in:st=20:d=1:alpha=1, fade=out:st=50:d=1:alpha=1[tmp2]; \
  nullsrc=size=1920x1080, drawbox=x=(5-1)*320:y=(5-1)*180:w=320:h=180:t=7:c=green,  fade=in:st=30:d=1:alpha=1, fade=out:st=60:d=1:alpha=1[tmp3]; \
  nullsrc=size=1920x1080, drawbox=x=(6-1)*320:y=(6-1)*180:w=320:h=180:t=7:c=yellow, fade=in:st=40:d=1:alpha=1, fade=out:st=70:d=1:alpha=1[tmp4]; \
  [tmp1][tmp2] overlay=0:0[ovr1]; \ 
  [tmp3][tmp4] overlay=0:0[ovr2]; \ 
  [ovr1][ovr2] overlay=0:0[boxes]; \ 
  [0:v][boxes] overlay=0:0" \        
  "Output.mp4"


  



  The input video is around 01:45 long. Log of run :

  



  ffmpeg -y        -loglevel verbose       -i "Input.mp4"          -filter_complex " nullsrc=size=1920x1080, drawbox=x=(3-1)*320:y=(3-1)*180:w=320:h=180:t=7:c=cyan,   fade=in:st=10:d=1:alpha=1, fade=out:st=40:d=1:alpha=1[tmp1]; nullsrc=size=1920x1080, drawbox=x  =(4-1)*320:y=(4-1)*180:w=320:h=180:t=7:c=blue,   fade=in:st=20:d=1:alpha=1, fade=out:st=50:d=1:alpha=1[tmp2]; nullsrc=size=1920x1080, drawbox=x=(5-1)*320:y=(5-1)*180:w=320:h=180:t=7:c=green,  fa
de=in:st=30:d=1:alpha=1, fade=out:st=60:d=1:alpha=1[tmp3]; nullsrc=size=1920x1080, drawbox=x=(6-1)*320:y=(6-1)*180:w=320:h=180:t=7:c=yellow, fade=in:st=40:d=1:alpha=1, fade=out:st=70:d=1:alpha=1[tmp4]; [tmp1][tmp2] overlay=0:0[ovr1]; [tmp3][tmp4] overlay=0:0[ovr2]; [ovr1][ovr2] overlay=0:0[boxes]; [0:v][boxes] overlay=0:0"      "Output.mp4"
ffmpeg version 3.4 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 7.2.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvid
stab --enable-libvorbis --enable-cuda --enable-cuvid --enable-d3d11va --enable-nvenc --enable-dxva2 --enable-avisynth --enable-libmfx
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
[h264 @ 000001df2b623ea0] Reinit context to 1920x1088, pix_fmt: yuv420p
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Input.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.29.100
  Duration: 00:01:48.67, start: 0.000000, bitrate: 1825 kb/s
    Stream #0:0(und): Video: h264 (High), 1 reference frame (avc1 / 0x31637661), yuv420p(left), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 1693 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: mp3 (mp4a / 0x6134706D), 44100 Hz, stereo, s16p, 127 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
[Parsed_nullsrc_0 @ 000001df2b61af00] size:1920x1080 rate:25/1 duration:-1.000000 sar:1/1
[Parsed_fade_2 @ 000001df2b6a6860] type:in start_time:10.000000 duration:1.000000 alpha:1
[Parsed_fade_3 @ 000001df2b9ddec0] type:out start_time:40.000000 duration:1.000000 alpha:1
[Parsed_nullsrc_4 @ 000001df2bc00560] size:1920x1080 rate:25/1 duration:-1.000000 sar:1/1
[Parsed_fade_6 @ 000001df29fe6780] type:in start_time:20.000000 duration:1.000000 alpha:1
[Parsed_fade_7 @ 000001df29fe6840] type:out start_time:50.000000 duration:1.000000 alpha:1
[Parsed_nullsrc_8 @ 000001df2b642dc0] size:1920x1080 rate:25/1 duration:-1.000000 sar:1/1
[Parsed_fade_10 @ 000001df2b6442a0] type:in start_time:30.000000 duration:1.000000 alpha:1
[Parsed_fade_11 @ 000001df2b6444e0] type:out start_time:60.000000 duration:1.000000 alpha:1
[Parsed_nullsrc_12 @ 000001df2b62d000] size:1920x1080 rate:25/1 duration:-1.000000 sar:1/1
[Parsed_fade_14 @ 000001df2b62d580] type:in start_time:40.000000 duration:1.000000 alpha:1
[Parsed_fade_15 @ 000001df2b62ddc0] type:out start_time:70.000000 duration:1.000000 alpha:1
Stream mapping:
  Stream #0:0 (h264) -> overlay:main (graph 0)
  overlay (graph 0) -> Stream #0:0 (libx264)
  Stream #0:1 -> #0:1 (mp3 (native) -> aac (native))
Press [q] to stop, [?] for help
[h264 @ 000001df2b8e5040] Reinit context to 1920x1088, pix_fmt: yuv420p
[graph_1_in_0_1 @ 000001df2b62e100] tb:1/44100 samplefmt:s16p samplerate:44100 chlayout:0x3
[format_out_0_1 @ 000001df2b62e5e0] auto-inserting filter 'auto_resampler_0' between the filter 'Parsed_anull_0' and the filter 'format_out_0_1'
[auto_resampler_0 @ 000001df2b62dd00] ch:2 chl:stereo fmt:s16p r:44100Hz -> ch:2 chl:stereo fmt:fltp r:44100Hz
[Parsed_nullsrc_0 @ 000001df2b62e440] size:1920x1080 rate:25/1 duration:-1.000000 sar:1/1
[Parsed_fade_2 @ 000001df2b62df60] type:in start_time:10.000000 duration:1.000000 alpha:1
[Parsed_fade_3 @ 000001df2b62e6c0] type:out start_time:40.000000 duration:1.000000 alpha:1
[Parsed_nullsrc_4 @ 000001df2b62d9c0] size:1920x1080 rate:25/1 duration:-1.000000 sar:1/1
[Parsed_fade_6 @ 000001df2b62e520] type:in start_time:20.000000 duration:1.000000 alpha:1
[Parsed_fade_7 @ 000001df2b62d8e0] type:out start_time:50.000000 duration:1.000000 alpha:1
[Parsed_nullsrc_8 @ 000001df2b62e1e0] size:1920x1080 rate:25/1 duration:-1.000000 sar:1/1
[Parsed_fade_10 @ 000001df2b62e2a0] type:in start_time:30.000000 duration:1.000000 alpha:1
[Parsed_fade_11 @ 000001df2b62e380] type:out start_time:60.000000 duration:1.000000 alpha:1
[Parsed_nullsrc_12 @ 000001df2b62dc20] size:1920x1080 rate:25/1 duration:-1.000000 sar:1/1
[Parsed_fade_14 @ 000001df2b816f60] type:in start_time:40.000000 duration:1.000000 alpha:1
[Parsed_fade_15 @ 000001df2b817ac0] type:out start_time:70.000000 duration:1.000000 alpha:1
[graph 0 input from stream 0:0 @ 000001df2b817ba0] w:1920 h:1080 pixfmt:yuv420p tb:1/15360 fr:30/1 sar:1/1 sws_param:flags=2
[Parsed_drawbox_1 @ 000001df2b62da80] x:640 y:360 w:320 h:180 color:0xA9A610FF
[Parsed_drawbox_5 @ 000001df2b62e780] x:960 y:540 w:320 h:180 color:0x29F06EFF
[Parsed_overlay_16 @ 000001df2b817860] main w:1920 h:1080 fmt:yuva420p overlay w:1920 h:1080 fmt:yuva420p
[Parsed_overlay_16 @ 000001df2b817860] [framesync @ 000001df2b815348] Selected 1/25 time base
[Parsed_overlay_16 @ 000001df2b817860] [framesync @ 000001df2b815348] Sync level 2
[Parsed_drawbox_9 @ 000001df2b62db60] x:1280 y:720 w:320 h:180 color:0x515B51FF
[Parsed_drawbox_13 @ 000001df2b818be0] x:1600 y:900 w:320 h:180 color:0xD21092FF
[Parsed_overlay_17 @ 000001df2b816d00] main w:1920 h:1080 fmt:yuva420p overlay w:1920 h:1080 fmt:yuva420p
[Parsed_overlay_17 @ 000001df2b816d00] [framesync @ 000001df2b815e48] Selected 1/25 time base
[Parsed_overlay_17 @ 000001df2b816d00] [framesync @ 000001df2b815e48] Sync level 2
[Parsed_overlay_18 @ 000001df2b8183c0] main w:1920 h:1080 fmt:yuva420p overlay w:1920 h:1080 fmt:yuva420p
[Parsed_overlay_18 @ 000001df2b8183c0] [framesync @ 000001df2b815668] Selected 1/25 time base
[Parsed_overlay_18 @ 000001df2b8183c0] [framesync @ 000001df2b815668] Sync level 2
[Parsed_overlay_19 @ 000001df2b817d40] main w:1920 h:1080 fmt:yuv420p overlay w:1920 h:1080 fmt:yuva420p
[Parsed_overlay_19 @ 000001df2b817d40] [framesync @ 000001df2b816488] Selected 1/76800 time base
[Parsed_overlay_19 @ 000001df2b817d40] [framesync @ 000001df2b816488] Sync level 2
[libx264 @ 000001df2b62cd40] using SAR=1/1
[libx264 @ 000001df2b62cd40] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 000001df2b62cd40] profile High, level 4.0
[libx264 @ 000001df2b62cd40] 264 - core 152 r2851 ba24899 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weigh
tb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'Output.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.83.100
    Stream #0:0: Video: h264 (libx264), 1 reference frame (avc1 / 0x31637661), yuv420p(left), 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 30 fps, 15360 tbn, 30 tbc (default)
    Metadata:
      encoder         : Lavc57.107.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, delay 1024, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      encoder         : Lavc57.107.100 aac
[Parsed_overlay_19 @ 000001df2b817d40] [framesync @ 000001df2b816488] Sync level 1speed=0.78x
Past duration 0.800774 too large
*** 1 dup!5 fps= 24 q=29.0 size=   21760kB time=00:01:46.86 bitrate=1668.0kbits/s speed=0.78x
    Last message repeated 1 times
*** 1 dup!8 fps= 24 q=29.0 size=   21760kB time=00:01:47.30 bitrate=1661.3kbits/s dup=2 drop=0 speed=0.78x
    Last message repeated 2 times
*** 1 dup!3 fps= 24 q=29.0 size=   21760kB time=00:01:47.80 bitrate=1653.6kbits/s dup=5 drop=0 speed=0.781x
    Last message repeated 1 times

...

    Last message repeated 2 times
*** 1 dup!3 fps= 25 q=29.0 size=   27392kB time=00:05:05.80 bitrate= 733.8kbits/s dup=995 drop=0 speed=0.832x
    Last message repeated 1 times
*** 1 dup!6 fps= 25 q=29.0 size=   27392kB time=00:05:06.23 bitrate= 732.8kbits/s dup=997 drop=0 speed=0.832x
    Last message repeated 1 times
*** 1 dup!0 fps= 25 q=29.0 size=   27392kB time=00:05:06.70 bitrate= 731.6kbits/s dup=999 drop=0 speed=0.832x
    Last message repeated 1 times
More than 1000 frames duplicated
*** 1 dup!3 fps= 25 q=29.0 size=   27392kB time=00:05:07.13 bitrate= 730.6kbits/s dup=1001 drop=0 speed=0.832x
    Last message repeated 2 times
*** 1 dup!8 fps= 25 q=29.0 size=   27392kB time=00:05:07.63 bitrate= 729.4kbits/s dup=1004 drop=0 speed=0.832x
    Last message repeated 1 times
... 
*** 1 dup!9 fps= 25 q=29.0 size=   27904kB time=00:05:18.66 bitrate= 717.3kbits/s dup=1059 drop=0 speed=0.834x
    Last message repeated 1 times
*** 1 dup!3 fps= 25 q=29.0 size=   27904kB time=00:05:19.13 bitrate= 716.3kbits/s dup=1061 drop=0 speed=0.834x
frame= 9635 fps= 25 q=-1.0 Lsize=   28364kB time=00:05:21.06 bitrate= 723.7kbits/s dup=1062 drop=0 speed=0.837x
video:26539kB audio:1637kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.669889%
Input file #0 (Input.mp4):
  Input stream #0:0 (video): 3260 packets read (23006797 bytes); 3260 frames decoded;
  Input stream #0:1 (audio): 4025 packets read (1682285 bytes); 4025 frames decoded (4636800 samples);
  Total: 7285 packets (24689082 bytes) demuxed
Output file #0 (Output.mp4):
  Output stream #0:0 (video): 9635 frames encoded; 9635 packets muxed (27175566 bytes);
  Output stream #0:1 (audio): 4529 frames encoded (4636800 samples); 4530 packets muxed (1676214 bytes);
  Total: 14165 packets (28851780 bytes) muxed
[libx264 @ 000001df2b62cd40] frame I:39    Avg QP:16.49  size:213954
[libx264 @ 000001df2b62cd40] frame P:2446  Avg QP:17.77  size:  6277
[libx264 @ 000001df2b62cd40] frame B:7150  Avg QP:30.38  size:   486
[libx264 @ 000001df2b62cd40] consecutive B-frames:  0.8%  0.8%  0.1% 98.3%
[libx264 @ 000001df2b62cd40] mb I  I16..4: 13.2% 43.0% 43.8%
[libx264 @ 000001df2b62cd40] mb P  I16..4:  0.2%  0.2%  0.1%  P16..4:  7.1%  3.3%  1.7%  0.0%  0.0%    skip:87.5%
[libx264 @ 000001df2b62cd40] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  4.3%  0.1%  0.0%  direct: 0.0%  skip:95.5%  L0:41.9% L1:56.1% BI: 2.0%
[libx264 @ 000001df2b62cd40] 8x8 transform intra:44.1% inter:65.8%
[libx264 @ 000001df2b62cd40] coded y,uvDC,uvAC intra: 71.3% 83.5% 52.5% inter: 1.1% 1.5% 0.0%
[libx264 @ 000001df2b62cd40] i16 v,h,dc,p: 31% 28%  4% 38%
[libx264 @ 000001df2b62cd40] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 28% 20% 10%  6%  6%  7%  7%  8%  8%
[libx264 @ 000001df2b62cd40] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 33% 28%  7%  4%  6%  6%  6%  5%  5%
[libx264 @ 000001df2b62cd40] i8c dc,h,v,p: 34% 27% 28% 11%
[libx264 @ 000001df2b62cd40] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 000001df2b62cd40] ref P L0: 71.1% 14.9% 10.6%  3.4%
[libx264 @ 000001df2b62cd40] ref B L0: 93.3%  5.9%  0.8%
[libx264 @ 000001df2b62cd40] ref B L1: 97.8%  2.2%
[libx264 @ 000001df2b62cd40] kb/s:676.90
[aac @ 000001df2b6a5620] Qavg: 1165.766
Exiting normally, received signal 2.
Terminate batch job (Y/N)? y


  


• Re-solving My Search Engine Problem

  28 juillet 2012, par Multimedia Mike — General, swish-e

  14 years ago, I created a web database of 8-bit Nintendo Entertainment System games. To make it useful, I developed a very primitive search feature.

  A few months ago, I decided to create a web database of video game music. To make it useful, I knew it would need to have a search feature. I realized I needed to solve the exact same problem again.

  Requirements
  The last time I solved this problem, I came up with an excruciatingly naïve idea. Hey, it worked. I really didn’t want to deploy the same solution again because it felt so silly the first time. Surely there are many better ways to solve it now ? Many different workable software solutions that do all the hard work for me ?

  The first time I attacked this, it was 1998 and hosting resources were scarce. On my primary web host I was able to put static HTML pages, perhaps with server side includes. The web host also offered dynamic scripting capabilities via something called htmlscript (a.k.a. MIVA Script). I had a secondary web host in my ISP which allowed me to host conventional CGI scripts on a Unix host, so that’s where I hosted the search function (Perl CGI script accessing a key/value data store file).

  Nowadays, sky’s the limit. Any type of technology you want to deploy should be tractable. Still, a key requirement was that I didn’t want to pay for additional hosting resources for this silly little side project. That leaves me with options that my current shared web hosting plan allows, which includes such advanced features as PHP, Perl and Python scripts. I can also access MySQL.

  Candidates
  There are a lot of mature software packages out there which can index and search data and be plugged into a website. But a lot of them would be unworkable on my web hosting plan due to language or library package limitations. Further, a lot of them feel like overkill. At the most basic level, all I really want to do is map a series of video game titles to URLs in a website.

  Based on my research, Lucene seems to hold a fair amount of mindshare as an open source indexing and search solution. But I was unsure of my ability to run it on my hosting plan. I think MySQL does some kind of full text search, so I could have probably made a solution around that. Again, it just feels like way more power than I need for this project.

  I used Swish-e once about 3 years ago for a little project. I wasn’t confident of my ability to run that on my server either. It has a Perl API but it requires custom modules.

  My quest for a search solution grew deep enough that I started perusing a textbook on information retrieval techniques in preparation for possibly writing my own solution from scratch. However, in doing so, I figured out how I might subvert an existing solution to do what I want.

  Back to Swish-e
  Again, all I wanted to do was pull data out of a database and map that data to a URL in a website. Reading the Swish-e documentation, I learned that the software supports a mode specifically tailored for this. Rather than asking Swish-e to index a series of document files living on disk, you can specify a script for Swish-e to run and the script will generate what appears to be a set of phantom documents for Swish-e to index.
  

  When I ’add’ a game music file to the game music website, I have a scripts that scrape the metadata (game title, system, song titles, composers, company, copyright, the original file name on disk, even the ripper/dumper who extracted the chiptune in the first place) and store it all in an SQLite database. When it’s time to update the database, another script systematically generates a series of pseudo-documents that spell out the metadata for each game and prefix each document with a path name. Searching for a term in the index returns a lists of paths that contain the search term. Thus, it makes sense for that path to be a site URL.

  But what about a web script which can search this Swish-e index ? That’s when I noticed Swish-e’s C API and came up with a crazy idea : Write the CGI script directly in C. It feels like sheer madness (or at least the height of software insecurity) to write a CGI script directly in C in this day and age. But it works (with the help of cgic for input processing), just as long as I statically link the search script with libswish-e.a (and libz.a). The web host is an x86 machine, after all.

  I’m not proud of what I did here— I’m proud of how little I had to do here. The searching CGI script is all of about 30 lines of C code. The one annoyance I experienced while writing it is that I had to consult the Swish-e source code to learn how to get my search results (the "swishdocpath" key — or any other key — for SwishResultPropertyStr() is not documented). Also, the C program just does the simplest job possible, only querying the term in the index and returning the results in plaintext, in order of relevance, to the client-side JavaScript code which requested them. JavaScript gets the job of sorting and grouping the results for presentation.

  Tuning the Search
  Almost immediately, I noticed that the search engine could not find one of my favorite SNES games, U.N. Squadron. That’s because all of its associated metadata names Area 88, the game’s original title. Thus, I had to modify the metadata database to allow attaching somewhat free-form tags to games in order to compensate. In this case, an alias title would show up in the game’s pseudo-document.

  Roman numerals are still a thorn in my side, just as they were 14 years ago in my original iteration. I dealt with it back then by converting all numbers to Roman numerals during the index and searching processes. I’m not willing to do that for this case and I’m still looking for a good solution.

  Another annoying problem deals with Mega Man, a popular franchise. The proper spelling is 2 words but it’s common for people to mash it into one word, Megaman (see also : Spider-Man, Spiderman, Spider Man). The index doesn’t gracefully deal with that and I have some hacks in place to cope for the time being.

  Positive Results
  I’m pleased with the results so far, and so are the users I have heard from. I know one user expressed amazement that a search for Castlevania turned up Akumajou Densetsu, the Japanese version of Castlevania III : Dracula’s Curse. This didn’t surprise me because I manually added a hint for that mapping. (BTW, if you are a fan of Castlevania III, definitely check out the Akumajou Densetsu soundtrack which has an upgraded version of the same soundtrack using special audio channels.)

  I was a little more surprised when a user announced that searching for ’probotector’ correctly turned up Contra : Hard Corps. I looked into why this was. It turns out that the original chiptune filename was extremely descriptive : "Contra - Hard Corps [Probotector] (1994-08-08)(Konami)". The filenames themselves often carry a bunch of useful metadata which is why it’s important to index those as well.

  And of course, many rippers, dumpers, and taggers have labored for over a decade to lovingly tag these songs with as much composer information as possible, which all gets indexed. The search engine gets a lot of compliments for its ability to find many songs written by favorite composers.

• Our latest improvement to QA : Screenshot Testing

  2 octobre 2013, par benaka — Development

  Introduction to QA in Piwik

  Like any piece of good software, Piwik comes with a comprehensive QA suite that includes unit and integration tests. The unit tests make sure core components of Piwik work properly. The integration tests make sure Piwik’s tracking and report aggregation and APIs work properly.

  To complete our QA suite, we’ve recently added a new type of tests : Screenshot tests, that we use to make sure Piwik’s controller and JavaScript code works properly.

  This blog post will explain how they work and describe our experiences setting them up ; we hope to show you an example of innovative QA practices in an active open source project.

  Screenshot Tests

  As the name implies, our screenshot tests (1) first capture a screenshot of a URL, then (2) compare the result with an expected image. This lets us test the code in Piwik’s controllers and Piwik’s JavaScript simply by specifying a URL.

  Contrast this with conventional UI tests that test for page content changes. Such tests require writing large amounts of test code that, at most, check for changes in HTML. Our tests, on the otherhand, will be able to show regressions in CSS and JavaScript rendering logic with a bare minimum of testing code.

  Capturing Screenshots

  Screenshots are captured using a 3rd party tool. We tried several tools before settling on PhantomJS. PhantomJS executes a JavaScript file with an environment that allows it to create WebKit powered web views. When capturing a screenshot, we supply PhantomJS with a script that :

  • opens a web page view,
  • loads a URL,
  • waits for all AJAX requests to be completed,
  • waits for all images to be loaded
  • waits for all JavaScript to be run.

  Then it renders the completed page to an PNG file.

  • To see how we use PhantomJS see capture.js.
  • To see how we wait for AJAX requests to complete and images to load see override.js.

  Comparing Screenshots

  Once a screenshot is generated we test for UI regressions by comparing it with an expected image. There is no sort of fuzzy matching involved. We just check that the images consist of the same bytes.

  If a screenshot test fails we use ImageMagick’s compare command line tool to generate an image diff :

  

  In this example above, there was a change that caused the Search box to be hidden in the datatable. This resulted in the whole Data table report being shifted up a few pixels. The differences are visible in red color which gives rapid feedback to the developers what has changed in the last commit.

  Screenshot Tests on Travis

  We experienced trouble generating identical screenshots on different machines, so our tests were not initially automated by Travis. Once we surpassed this hurdle, we created a new github repo to store our UI tests and screenshots and then enabled the travis build for it. We also made sure that every time a commit is pushed to the Piwik repo, our travis build will push a commit to the UI test repo to run the UI tests.

  We decided to create a new repository so the main repository wouldn’t be burdened with the large screenshot files (which git would not handle very well). We also made sure the travis build would upload all the generated screenshots to a server so debugging failures would be easier.

  Problems we experienced

  Getting generated screenshots to render identically on separate machines was quite a challenge. It took months to figure out how to get it right. Here’s what we learned :

  Fonts will render identically on different machines, but different machines can pick the wrong fonts. When we first tried getting these tests to run on Travis, we noticed small differences in the way fonts were rendered on different machines. We thought this was an insurmountable problem that would occur due to the libraries installed on these machines. It turns out, the machines were just picking the wrong fonts. After installing certain fonts during our Travis build, everything started working.

  Different versions of GD can generate slightly different images. GD is used in Piwik to, among other things, generate sparkline images. Different versions of GD will result in slightly different images. They look the same to the naked eye, but some pixels will have slightly different colors. This is, unfortunately, a problem we couldn’t solve. We couldn’t make sure that everyone who runs the tests uses the same version of GD, so instead we disabled sparklines for UI testing.

  What we learned about existing screenshot capturing tools

  We tried several screenshot capturing tools before finding one that would work adequately. Here’s what we learned about them :

  • CutyCapt This is the first screenshot capturing tool we tried. CutyCapt is a C++ program that uses QtWebKit to load and take a screenshot of a page. It can’t be used to capture multiple screenshots in one run and it can’t be used to wait for all AJAX/Images/JavaScript to complete/load (at least not currently).

  • PhantomJS This is the solution we eventually chose. PhantomJS is a headless scriptable browser that currently uses WebKit as its rendering engine.

    For the most part, PhantomJS is the best solution we found. It reliably renders screenshots, allows JavaScript to be injected into pages it loads, and since it essentially just runs JavaScript code that you provide, it can be made to do whatever you want.

  • SlimerJS SlimerJS is a clone of PhantomJS that uses Gecko as the rendering engine. It is meant to function similarly to PhantomJS. Unfortunately, due to some limitations hard-coded in Mozilla’s software, we couldn’t use it.

    For one, SlimerJS is not headless. There is, apparently, no way to do that when embedding Mozilla. You can, however, run it through xvfb, however the fact that it has to create a window means some odd things can happen. When using SlimerJS, we would sometimes end up with images where tooltips would display as if the mouse was hovering over an element. This inconsistency meant we couldn’t use it for our tests.

  One tool we didn’t try was Selenium Webdriver. Although Selenium is traditionally used to create tests that check for HTML content, it can be used to generate screenshots. (Note : PhantomJS supports using a remote WebDriver.)

  Our Future Plans for Screenshot Testing

  At the moment we render a couple dozen screenshots. We test how our PHP code, JavaScript code and CSS makes Piwik’s UI look, but we don’t test how it behaves. This is our next step.

  We want to create Screenshot Unit Tests for each UI control Piwik uses (for example, the Data Table View or the Site Selector). These tests would use the Widgetize plugin to load a control by itself, then execute JavaScript that simulates events and user behavior, and finally take a screenshot. This way we can test how our code handles clicks and hovers and all sorts of other behavior.

  Screenshots Tests will make Piwik more stable and keep us agile and able to release early and often. Thank you for your support & Spreading the word about Piwik !