Recherche avancée

Sur d’autres sites (7202)

• Multiprocess FATE Revisited

  26 juin 2010, par Multimedia Mike — FATE Server, Python

  I thought I had brainstormed a simple, elegant, multithreaded, deadlock-free refactoring for FATE in a previous post. However, I sort of glossed over the test ordering logic which I had not yet prototyped. The grim, possibly deadlock-afflicted reality is that the main thread needs to be notified as tests are completed. So, the main thread sends test specs through a queue to be executed by n tester threads and those threads send results to a results aggregator thread. Additionally, the results aggregator will need to send completed test IDs back to the main thread.

  
  
  

  But when I step back and look at the graph, I can’t rationalize why there should be a separate results aggregator thread. That was added to cut down on deadlock possibilities since the main thread and the tester threads would not be waiting for data from each other. Now that I’ve come to terms with the fact that the main and the testers need to exchange data in realtime, I think I can safely eliminate the result thread. Adding more threads is not the best way to guard against race conditions and deadlocks. Ask xine.

  
  
  

  I’m still hung up on the deadlock issue. I have these queues through which the threads communicate. At issue is the fact that they can cause a thread to block when inserting an item if the queue is "full". How full is full ? Immaterial ; seeking to answer such a question is not how you guard against race conditions. Rather, it seems to me that one side should be doing non-blocking queue operations.

  This is how I’m planning to revise the logic in the main thread :

  test_set = set of all tests to execute
  tests_pending = test_set
  tests_blocked = empty set
  tests_queue = multi-consumer queue to send test specs to tester threads
  results_queue = multi-producer queue through which tester threads send results
  while there are tests in tests_pending :
    pop a test from test_set
    if test depends on any tests that appear in tests_pending :
      add test to tests_blocked
    else :
      add test to tests_queue in a non-blocking manner
      if tests_queue is full, add test to tests_blocked
    while there are results in the results_queue :
  
      get a result from result_queue in non-blocking manner
  
      remove the corresponding test from tests_pending
  if tests_blocked is non-empty :
  
    sleep for 1 second
  
    test_set = tests_blocked
  
    tests_blocked = empty set
  
  else :
  
    insert n shutdown signals, one from each thread
  go to the top of the loop and repeat until there are no more tests
  while there are results in the results_queue :
  
    get a result from result_queue in a blocking manner
  

  Not mentioned in the pseudocode (so it doesn’t get too verbose) is logic to check whether the retrieved test result is actually an end-of-thread signal. These are accounted and the whole test process is done when one is received for each thread.

  On the tester thread side, it’s safe for them to do blocking test queue retrievals and blocking result queue insertions. The reason for the 1-second delay before resetting tests_blocked and looping again is because I want to guard against the situation where tests A and B are to be run, A depends of B running first, and while B is running (and happens to be a long encoding test), the main thread is spinning about, obsessively testing whether it’s time to insert A into the tests queue.

  It all sounds just crazy enough to work. In fact, I coded it up and it does work, sort of. The queue gets blocked pretty quickly. Instead of sleeping, I decided it’s better to perform the put operation using a 1-second timeout.

  Still, I’m paranoid about the precise operation of the IPC queue mechanism at work here. What happens if I try to stuff in a test spec that’s a bit too large ? Will the module take whatever I give it and serialize it through the queue as soon as it can ? I think an impromptu science project is in order.

  big-queue.py :

  PLAIN TEXT

  PYTHON :

  1. # !/usr/bin/python
  2.  
  3. import multiprocessing
  4. import Queue
  5.  
  6. def f(q) :
  7.   str = q.get()
  8.   print "reader function got a string of %d characters" % (len(str))
  9.  
  10. q = multiprocessing.Queue()
  11. p = multiprocessing.Process(target=f, args=(q,))
  12. p.start()
  13. try :
  14.   q.put_nowait(’a’ * 100000000)
  15. except Queue.Full :
  16.   print "queue full"

  $ ./big-queue.py
  reader function got a string of 100000000 characters
  

  Since 100 MB doesn’t even make it choke, FATE’s little test specs shouldn’t pose any difficulty.

• Anomalie #2388 : formulaires CVT multi-etapes et formulaires_editer_objet_charger

  1er novembre 2011, par Severo Lipari

  ah pardon, if (is_numeric($id_chat) && intval($id_chat)==$id_chat) $valeurs = formulaires_editer_objet_charger(’chat’,$id_chat,$id_rubrique,$lier_trad,$retour,$config_fonc,$row,$hidden) ; else $valeurs = array() ; $valeurs[’nom’] = ’’ ; $valeurs[’race’] = ’’ ; $valeurs[’date’] = ’’ ; (...)

• Simply beyond ridiculous

  7 mai 2010, par Dark Shikari — H.265, speed

  For the past few years, various improvements on H.264 have been periodically proposed, ranging from larger transforms to better intra prediction. These finally came together in the JCT-VC meeting this past April, where over two dozen proposals were made for a next-generation video coding standard. Of course, all of these were in very rough-draft form ; it will likely take years to filter it down into a usable standard. In the process, they’ll pick the most useful features (hopefully) from each proposal and combine them into something a bit more sane. But, of course, it all has to start somewhere.

  A number of features were common : larger block sizes, larger transform sizes, fancier interpolation filters, improved intra prediction schemes, improved motion vector prediction, increased internal bit depth, new entropy coding schemes, and so forth. A lot of these are potentially quite promising and resolve a lot of complaints I’ve had about H.264, so I decided to try out the proposal that appeared the most interesting : the Samsung+BBC proposal (A124), which claims compression improvements of around 40%.

  The proposal combines a bouillabaisse of new features, ranging from a 12-tap interpolation filter to 12thpel motion compensation and transforms as large as 64×64. Overall, I would say it’s a good proposal and I don’t doubt their results given the sheer volume of useful features they’ve dumped into it. I was a bit worried about complexity, however, as 12-tap interpolation filters don’t exactly scream “fast”.

  I prepared myself for the slowness of an unoptimized encoder implementation, compiled their tool, and started a test encode with their recommended settings.

  I waited. The first frame, an I-frame, completed.

  I took a nap.

  I waited. The second frame, a P-frame, was done.

  I played a game of Settlers.

  I waited. The third frame, a B-frame, was done.

  I worked on a term paper.

  I waited. The fourth frame, a B-frame, was done.

  After a full 6 hours, 8 frames had encoded. Yes, at this rate, it would take a full two weeks to encode 10 seconds of HD video. On a Core i7. This is not merely slow ; this is over 1000 times slower than x264 on “placebo” mode. This is so slow that it is not merely impractical ; it is impossible to even test. This encoder is apparently designed for some sort of hypothetical future computer from space. And word from other developers is that the Intel proposal is even slower.

  This has led me to suspect that there is a great deal of cheating going on in the H.265 proposals. The goal of the proposals, of course, is to pick the best feature set for the next generation video compression standard. But there is an extra motivation : organizations whose features get accepted get patents on the resulting standard, and thus income. With such large sums of money in the picture, dishonesty becomes all the more profitable.

  There is a set of rules, of course, to limit how the proposals can optimize their encoders. If different encoders use different optimization techniques, the results will no longer be comparable — remember, they are trying to compare compression features, not methods of optimizing encoder-side decisions. Thus all encoders are required to use a constant quantizer, specified frame types, and so forth. But there are no limits on how slow an encoder can be or what algorithms it can use.

  It would be one thing if the proposed encoder was a mere 10 times slower than the current reference ; that would be reasonable, given the low level of optimization and higher complexity of the new standard. But this is beyond ridiculous. With the prize given to whoever can eke out the most PSNR at a given quantizer at the lowest bitrate (with no limits on speed), we’re just going to get an arms race of slow encoders, with every company trying to use the most ridiculous optimizations possible, even if they involve encoding the frame 100,000 times over to choose the optimal parameters. And the end result will be as I encountered here : encoders so slow that they are simply impossible to even test.

  Such an arms race certainly does little good in optimizing for reality where we don’t have 30 years to encode an HD movie : a feature that gives great compression improvements is useless if it’s impossible to optimize for in a reasonable amount of time. Certainly once the standard is finalized practical encoders will be written — but it makes no sense to optimize the standard for a use-case that doesn’t exist. And even attempting to “optimize” anything is difficult when encoding a few seconds of video takes weeks.

  Update : The people involved have contacted me and insist that there was in fact no cheating going on. This is probably correct ; the problem appears to be that the rules that were set out were simply not strict enough, making many changes that I would intuitively consider “cheating” to be perfectly allowed, and thus everyone can do it.

  I would like to apologize if I implied that the results weren’t valid ; they are — the Samsung-BBC proposal is definitely one of the best, which is why I picked it to test with. It’s just that I think any situation in which it’s impossible to test your own software is unreasonable, and thus the entire situation is an inherently broken one, given the lax rules, slow baseline encoder, and no restrictions on compute time.