Recherche avancée

Sur d’autres sites (5363)

• Working on images asynchronously

  15 décembre 2013, par Mikko Koppanen — Imagick, PHP stuff

  To get my quota on buzzwords for the day we are going to look at using ZeroMQ and Imagick to create a simple asynchronous image processing system. Why asynchronous ? First of all, separating the image handling from a interactive PHP scripts allows us to scale the image processing separately from the web heads. For example we could do the image processing on separate servers, which have SSDs attached and more memory. In this example making the images available to all worker nodes is left to the reader.

  Secondly, separating the image processing from a web script can provide more responsive experience to the user. This doesn’t necessarily mean faster, but let’s say in a multiple image upload scenario this method allows the user to do something else on the site while we process the images in the background. This can be beneficial especially in cases where users upload hundreds of images at a time. To achieve a simple distributed image processing infrastructure we are going to use ZeroMQ for communicating between different components and Imagick to work on the images.

  The first part we are going to create is a simple “Worker” -process skeleton. Naturally for a live environment you would like to have more error handling and possibly use pcntl for process control, but for the sake of brewity the example is barebones :

  php

  < view plain text >

  1. < ?php
  2.  
  3. define (’THUMBNAIL_ADDR’, ’tcp ://127.0.0.1:5000’) ;
  4. define (’COLLECTOR_ADDR’, ’tcp ://127.0.0.1:5001’) ;
  5.  
  6. class Worker {
  7.  
  8.   private $in ;
  9.   private $out ;
  10.  
  11.   public function __construct ($in_addr, $out_addr)
  12.   {
  13.     $context = new ZMQContext () ;
  14.  
  15.     $this->in = new ZMQSocket ($context, ZMQ: :SOCKET_PULL) ;
  16.     $this->in->bind ($in_addr) ;
  17.  
  18.     $this->out = new ZMQSocket ($context, ZMQ: :SOCKET_PUSH) ;
  19.     $this->out->connect ($out_addr) ;
  20.   }
  21.  
  22.   public function work () {
  23.     while ($command = $this->in->recvMulti ()) {
  24.       if (isset ($this->commands [$command [0]])) {
  25.         echo "Received work" . PHP_EOL ;
  26.  
  27.         $callback = $this->commands [$command [0]] ;
  28.  
  29.         array_shift ($command) ;
  30.         $response = call_user_func_array ($callback, $command) ;
  31.  
  32.         if (is_array ($response))
  33.           $this->out->sendMulti ($response) ;
  34.         else
  35.           $this->out->send ($response) ;
  36.       }
  37.       else {
  38.         error_log ("There is no registered worker for $command [0]") ;
  39.       }
  40.     }
  41.   }
  42.  
  43.   public function register ($command, $callback)
  44.   {
  45.     $this->commands [$command] = $callback ;
  46.   }
  47. }
  48.  ?>

  The Worker class allows us to register commands with callbacks associated with them. In our case the Worker class doesn’t actually care or know about the parameters being passed to the actual callback, it just blindly passes them on. We are using two separate sockets in this example, one for incoming work requests and one for passing the results onwards. This allows us to create a simple pipeline by adding more workers in the mix. For example we could first have a watermark worker, which takes the original image and composites a watermark on it, passes the file onwards to thumbnail worker, which then creates different sizes of thumbnails and passes the final results to event collector.

  The next part we are going to create a is a simple worker script that does the actual thumbnailing of the images :

  php

  < view plain text >

  1. < ?php
  2. include __DIR__ . ’/common.php’ ;
  3.  
  4. // Create worker class and bind the inbound address to ’THUMBNAIL_ADDR’ and connect outbound to ’COLLECTOR_ADDR’
  5. $worker = new Worker (THUMBNAIL_ADDR, COLLECTOR_ADDR) ;
  6.  
  7. // Register our thumbnail callback, nothing special here
  8. $worker->register (’thumbnail’, function ($filename, $width, $height) {
  9.                   $info = pathinfo ($filename) ;
  10.  
  11.                   $out = sprintf ("%s/%s_%dx%d.%s",
  12.                           $info [’dirname’],
  13.                           $info [’filename’],
  14.                           $width,
  15.                           $height,
  16.                           $info [’extension’]) ;
  17.  
  18.                   $status = 1 ;
  19.                   $message = ’’ ;
  20.  
  21.                   try {
  22.                     $im = new Imagick ($filename) ;
  23.                     $im->thumbnailImage ($width, $height) ;
  24.                     $im->writeImage ($out) ;
  25.                   }
  26.                   catch (Exception $e) {
  27.                     $status = 0 ;
  28.                     $message = $e->getMessage () ;
  29.                   }
  30.  
  31.                   return array (
  32.                         ’status’  => $status,
  33.                         ’filename’ => $filename,
  34.                         ’thumbnail’ => $out,
  35.                         ’message’ => $message,
  36.                     ) ;
  37.                 }) ;
  38.  
  39. // Run the worker, will block
  40. echo "Running thumbnail worker.." . PHP_EOL ;
  41. $worker->work () ;

  As you can see from the code the thumbnail worker registers a callback for ‘thumbnail’ command. The callback does the thumbnailing based on input and returns the status, original filename and the thumbnail filename. We have connected our Workers “outbound” socket to event collector, which will receive the results from the thumbnail worker and do something with them. What the “something” is depends on you. For example you could push the response into a websocket to show immediate feeedback to the user or store the results into a database.

  Our example event collector will just do a var_dump on every event it receives from the thumbnailer :

  php

  < view plain text >

  1. < ?php
  2. include __DIR__ . ’/common.php’ ;
  3.  
  4. $socket = new ZMQSocket (new ZMQContext (), ZMQ: :SOCKET_PULL) ;
  5. $socket->bind (COLLECTOR_ADDR) ;
  6.  
  7. echo "Waiting for events.." . PHP_EOL ;
  8. while (($message = $socket->recvMulti ())) {
  9.   var_dump ($message) ;
  10. }
  11.  ?>

  The final piece of the puzzle is the client that pumps messages into the pipeline. The client connects to the thumbnail worker, passes on filename and desired dimensions :

  php

  < view plain text >

  1. < ?php
  2. include __DIR__ . ’/common.php’ ;
  3.  
  4. $socket = new ZMQSocket (new ZMQContext (), ZMQ: :SOCKET_PUSH) ;
  5. $socket->connect (THUMBNAIL_ADDR) ;
  6.  
  7. $socket->sendMulti (
  8.       array (
  9.         ’thumbnail’,
  10.         realpath (’./test.jpg’),
  11.         50,
  12.         50,
  13.       )
  14. ) ;
  15. echo "Sent request" . PHP_EOL ;
  16.  ?>

  After this our processing pipeline will look like this :

  

  Now, if we notice that thumbnail workers or the event collectors can’t keep up with the rate of images we are pushing through we can start scaling the pipeline by adding more processes on each layer. ZeroMQ PUSH socket will automatically round-robin between all connected nodes, which makes adding more workers and event collectors simple. After adding more workers our pipeline will look like this :

  

  Using ZeroMQ also allows us to create more flexible architectures by adding forwarding devices in the middle, adding request-reply workers etc. So, the last thing to do is to run our pipeline and see the results :

  Let’s create our test image first :

  $ convert magick:rose test.jpg
  

  From the command-line run the thumbnail script :

  $ php thumbnail.php 
  Running thumbnail worker..
  

  In a separate terminal window run the event collector :

  $ php collector.php 
  Waiting for events..
  

  And finally run the client to send the thumbnail request :

  $ php client.php 
  Sent request
  $
  

  If everything went according to the plan you should now see the following output in the event collector window :

  array(4) 
    [0]=>
    string(1) "1"
    [1]=>
    string(56) "/test.jpg"
    [2]=>
    string(62) "/test_50x50.jpg"
    [3]=>
    string(0) ""
  
  

  Happy hacking !

• Understanding the VP8 Token Tree

  7 juin 2010, par Multimedia Mike — VP8

  I got tripped up on another part of the VP8 decoding process today. So I drew a picture to help myself understand it. Then I went back and read David Conrad’s comment on my last post regarding my difficulty understanding the VP8 spec and saw that he ran into the same problem. Since we both experienced the same hindrance in trying to sort out this matter, I thought I may as well publish the picture I drew.

  VP8 defines various trees for decoding different syntax elements. There is one tree for decoding the tokens and it is expressed in the VP8 spec as such :

  PLAIN TEXT

  C :

  1. const tree_index coef_tree [2 * (num_dct_tokens - 1)] =
  2. {
  3.  -dct_eob, 2,        /* eob = "0"  */
  4.   -DCT_0, 4,        /* 0  = "10" */
  5.   -DCT_1, 6,        /* 1  = "110" */
  6.    8, 12,
  7.    -DCT_2, 10,      /* 2  = "11100" */
  8.     -DCT_3, -DCT_4,    /* 3  = "111010", 4 = "111011" */
  9.    14, 16,
  10.     -dct_cat1, -dct_cat2, /* cat1 = "111100", cat2 = "111101" */
  11.    18, 20,
  12.     -dct_cat3, -dct_cat4, /* cat3 = "1111100", cat4 = "1111101" */
  13.     -dct_cat5, -dct_cat6 /* cat4 = "1111110", cat4 = "1111111" */
  14. } ;

  Here is what the table looks like when you make a tree out of it (click for full size image) :

  
  
  

  The catch is that it makes no sense for an end-of-block (EOB) token to follow a 0 token since EOB already indicates that the remainder of the coefficients should be 0 anyway. Thus, the spec states that, "decoding of certain DCT coefficients may skip the first branch, whose preceding coefficient is a DCT_0." I confess, I didn’t understand what "skip the first branch" meant until I drew the tree.

  
  
  

  For those wondering why it might be sub-optimal (clarity-wise) for a spec to simply regurgitate vast chunks of C code, this makes a decent case. As you can see, the spec makes certain assumptions about how a binary tree should be organized in a static array (node n points to elements n*2 and n*2+1 as its branches ; leaves are either negative or 0). This is the second method I have seen ; another piece of code (not the VP8 spec) had the nodes in the first half of the array and pointed to leaves in the second half. There must be other arrangements.

• Dreamcast Operating Systems

  16 septembre 2010, par Multimedia Mike — Sega Dreamcast

  The Sega Dreamcast was famously emblazoned with a logo proudly announcing that it was compatible with Windows CE :

  
  
  

  It’s quite confusing. The console certainly doesn’t boot into some version of Windows to launch games. Apparently, there was a special version of CE developed for the DC and game companies had the option to leverage it. I do recall that some game startup screens would similarly advertise Windows CE.

  Once the homebrew community got ahold of the device, the sky was the limit. I think NetBSD was the first alternative OS to support the Dreamcast. Meanwhile, I have recollections of DC Linux and LinuxDC projects along with more generic Linux-SH and SH-Linux projects.

  
  
  

  DC Evolution hosts a disc image available for download with an unofficial version of DC Linux, assembled by one Adrian O’Grady. I figured out how to burn the disc (burning DC discs is a blog post of its own) and got it working in the console.

  It’s possible to log in directly via the physical keyboard or through a serial terminal provided that you have a coder’s cable. That reminds me– my local Fry’s had a selection of USB-to-serial cables. I think this is another area that is sufficiently commoditized that just about any cable ought to work with Linux out of the box. Or maybe I’m just extrapolating from the experience of having the cheapest cable in the selection (made by io connect) plug and play with Linux.

  
  
  

  Look ! No messy converter box in the middle as in the Belkin case. The reason I went with this cable is that the packaging claimed it was capable of up to 500 Kbits/sec. Most of the cables advertised a max of 115200 bps. I distinctly recall being able to use the DC coder’s cable at 230400 bps a long time ago. Alas, 115200 seems to be the speed limit, even with this new USB cable.

  Anyway, the distribution is based on a 2.4.5 kernel circa 2001. I tried to make PPP work over the serial cable but the kernel doesn’t have support. If you’re interested, here is some basic information about the machine from Linux’s perspective, gleaned from some simple commands. This helps remind us of a simpler time when Linux was able to run comfortably on a computer with 16 MB of RAM.

  Debian GNU/Linux testing/unstable dreamcast ttsc/1
  dreamcast login : root
  
  Linux dreamcast 2.4.5 #27 Thu May 31 07:06:51 JST 2001 sh4 unknown
  Most of the programs included with the Debian GNU/Linux system are
  
  freely redistributable ; the exact distribution terms for each program
  
  are described in the individual files in /usr/share/doc/*/copyright
  Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
  
  permitted by applicable law.
  dreamcast : # uname -a
  
  Linux dreamcast 2.4.5 #27 Thu May 31 07:06:51 JST 2001 sh4 unknown
  dreamcast : # cat /proc/cpuinfo 
  
  cpu family : SH-4
  
  cache size : 8K-byte/16K-byte
  
  bogomips : 199.47
  Machine : dreamcast
  
  CPU clock : 200.00MHz
  
  Bus clock : 100.00MHz
  
  Peripheral module clock : 50.00MHz
  dreamcast : # top -b
   09:14:54 up 14 min,  1 user,  load average : 0.04, 0.03, 0.03
  
  15 processes : 14 sleeping, 1 running, 0 zombie, 0 stopped
  
  CPU states :   1.1% user,   5.8% system,   0.0% nice,  93.1% idle
  
  Mem :     14616K total,    11316K used,     3300K free,     2296K buffers
  
  Swap :        0K total,        0K used,        0K free,     5556K cached
    PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
  
    219 root      18   0  1072 1068   868 R     5.6  7.3   0:00 top
  
      1 root       9   0   596  596   512 S     0.0  4.0   0:01 init
  
      2 root       9   0     0    0     0 SW    0.0  0.0   0:00 keventd
  
      3 root       9   0     0    0     0 SW    0.0  0.0   0:00 kswapd
  
      4 root       9   0     0    0     0 SW    0.0  0.0   0:00 kreclaimd
  
      5 root       9   0     0    0     0 SW    0.0  0.0   0:00 bdflush
  
      6 root       9   0     0    0     0 SW    0.0  0.0   0:00 kupdated
  
      7 root       9   0     0    0     0 SW    0.0  0.0   0:00 kmapled
  
     39 root       9   0   900  900   668 S     0.0  6.1   0:00 devfsd
  
     91 root       8   0   652  652   556 S     0.0  4.4   0:00 pump
  
     96 daemon     9   0   524  524   420 S     0.0  3.5   0:00 portmap
  
    149 root       9   0   944  944   796 S     0.0  6.4   0:00 syslogd
  
    152 root       9   0   604  604   456 S     0.0  4.1   0:00 klogd
  
    187 root       9   0   540  540   456 S     0.0  3.6   0:00 getty
  
    201 root       9   0  1380 1376  1112 S     0.0  9.4   0:01 bash
  

  Note that at this point I had shutdown both gpm and inetd. The rest of the processes, save for bash, are default. The above stats only report about 14 MB of RAM ; where are the other 2 MB ?

  dreamcast : # df -h
  Filesystem            Size  Used Avail Use% Mounted on
  /dev/rd/1             2.0M  560k  1.4M  28% /