Recherche avancée

Sur d’autres sites (6685)

• build video from images with a bash for loop using ffmpeg

  10 septembre 2021, par TheCodeNovice

  I am trying to take 50K images and turn them into a movie using ffmpeg. I am running this on a HPC setup, hence the slurm commands. My attempt does not work since I am running into a hard limit due to the shear volume of images. I cannot just list a start number since the pipeline will reject some images so I do not have a proper sequence.

  


  I know a loop could circumvent both issues but I am not sure how to use that with ffmpeg so that it builds one long movie.

  


  The cat command has worked for shorter movies but i just have too many images now.

  


  #!/bin/bash&#xA;&#xA;img_dir=&#x27;foo/bar/1/2/123456&#x27;&#xA;folder=&#x27;fooo&#x27;&#xA;#BATCH -p general&#xA;#SBATCH -N 1&#xA;#SBATCH -t 03-00:00:00&#xA;#SBATCH --mem=8g&#xA;#SBATCH -n 1&#xA;#SBATCH --mail-type=BEGIN,REQUEUE,END,FAIL,REQUEUE   &#xA;#SBATCH --mail-user=<snip>&#xA;&#xA;singularity exec /$img_dir/foo_container cat /$img_dir/processed_images/$folder/*.jpeg | ffmpeg -f image2pipe -i pipe:.jpeg -vf "crop=trunc(iw/2)*2:trunc(ih/2)*2" /$img_dir/processed_images/$folder/$folder.mp4&#xA;&#xA;</snip>

  


• CD-R Read Speed Experiments

  21 mai 2011, par Multimedia Mike — Science Projects, Sega Dreamcast

  I want to know how fast I can really read data from a CD-R. Pursuant to my previous musings on this subject, I was informed that it is inadequate to profile reading just any file from a CD-R since data might be read faster or slower depending on whether the data is closer to the inside or the outside of the disc.

  Conclusion / Executive Summary
  It is 100% true that reading data from the outside of a CD-R is faster than reading data from the inside. Read on if you care to know the details of how I arrived at this conclusion, and to find out just how much speed advantage there is to reading from the outside rather than the inside.

  Science Project Outline

  • Create some sample CD-Rs with various properties
  • Get a variety of optical drives
  • Write a custom program that profiles the read speed

  Creating The Test Media
  It’s my understanding that not all CD-Rs are created equal. Fortunately, I have 3 spindles of media handy : Some plain-looking Memorex discs, some rather flamboyant Maxell discs, and those 80mm TDK discs :

  
  
  

  My approach for burning is to create a single file to be burned into a standard ISO-9660 filesystem. The size of the file will be the advertised length of the CD-R minus 1 megabyte for overhead— so, 699 MB for the 120mm discs, 209 MB for the 80mm disc. The file will contain a repeating sequence of 0..0xFF bytes.

  Profiling
  I don’t want to leave this to the vagaries of any filesystem handling layer so I will conduct this experiment at the sector level. Profiling program outline :

  • Read the CD-ROM TOC and get the number of sectors that comprise the data track
  • Profile reading the first 20 MB of sectors
  • Profile reading 20 MB of sectors in the middle of the track
  • Profile reading the last 20 MB of sectors

  Unfortunately, I couldn’t figure out the raw sector reading on modern Linux incarnations (which is annoying since I remember it being pretty straightforward years ago). So I left it to the filesystem after all. New algorithm :

  • Open the single, large file on the CD-R and query the file length
  • Profile reading the first 20 MB of data, 512 kbytes at a time
  • Profile reading 20 MB of sectors in the middle of the track (starting from filesize / 2 - 10 MB), 512 kbytes at a time
  • Profile reading the last 20 MB of sectors (starting from filesize - 20MB), 512 kbytes at a time

  Empirical Data
  I tested the program in Linux using an LG Slim external multi-drive (seen at the top of the pile in this post) and one of my Sega Dreamcast units. I gathered the median value of 3 runs for each area (inner, middle, and outer). I also conducted a buffer flush in between Linux runs (as root : 'sync; echo 3 > /proc/sys/vm/drop_caches').

  LG Slim external multi-drive (reading from inner, middle, and outer areas in kbytes/sec) :

  • TDK-80mm : 721, 897, 1048
  • Memorex-120mm : 1601, 2805, 3623
  • Maxell-120mm : 1660, 2806, 3624

  So the 120mm discs can range from about 10.5X all the way up to a full 24X on this drive. For whatever reason, the 80mm disc fares a bit worse — even at the inner track — with a range of 4.8X - 7X.

  Sega Dreamcast (reading from inner, middle, and outer areas in kbytes/sec) :

  • TDK-80mm : 502, 632, 749
  • Memorex-120mm : 499, 889, 1143
  • Maxell-120mm : 500, 890, 1156

  It’s interesting that the 80mm disc performed comparably to the 120mm discs in the Dreamcast, in contrast to the LG Slim drive. Also, the results are consistent with my previous profiling experiments, which largely only touched the inner area. The read speeds range from 3.3X - 7.7X. The middle of a 120mm disc reads at about 6X.

  Implications
  A few thoughts regarding these results :

  • Since the very definition of 1X is the minimum speed necessary to stream data from an audio CD, then presumably, original 1X CD-ROM drives would have needed to be capable of reading 1X from the inner area. I wonder what the max read speed at the outer edges was ? It’s unlikely I would be able to get a 1X drive working easily in this day and age since the earliest CD-ROM drives required custom controllers.
  • I think 24X is the max rated read speed for CD-Rs, at least for this drive. This implies that the marketing literature only cites the best possible numbers. I guess this is no surprise, similar to how monitors and TVs have always been measured by their diagonal dimension.
  • Given this data, how do you engineer an ISO-9660 filesystem image so that the timing-sensitive multimedia files live on the outermost track ? In the Dreamcast case, if you can guarantee your FMV files will live somewhere between the middle and the end of the disc, you should be able to count on a bitrate of at least 900 kbytes/sec.

  Source Code
  Here is the program I wrote for profiling. Note that the filename is hardcoded (#define FILENAME). Compiling for Linux is a simple 'gcc -Wall profile-cdr.c -o profile-cdr'. Compiling for Dreamcast is performed in the standard KallistiOS manner (people skilled in the art already know what they need to know) ; the only variation is to compile with the '-D_arch_dreamcast' flag, which the default KOS environment adds anyway.

  PLAIN TEXT

  C :

  1. #ifdef _arch_dreamcast
  2.   #include <kos .h>
  3.  
  4.   /* map I/O functions to their KOS equivalents */
  5.   #define open fs_open
  6.   #define lseek fs_seek
  7.   #define read fs_read
  8.   #define close fs_close
  9.  
  10.   #define FILENAME "/cd/bigfile"
  11. #else
  12.   #include <stdio .h>
  13.   #include <sys /types.h>
  14.   #include </sys><sys /stat.h>
  15.   #include </sys><sys /time.h>
  16.   #include <fcntl .h>
  17.   #include <unistd .h>
  18.  
  19.   #define FILENAME "/media/Full disc/bigfile"
  20. #endif
  21.  
  22. /* Get a current absolute millisecond count ; it doesn’t have to be in
  23. * reference to anything special. */
  24. unsigned int get_current_milliseconds()
  25. {
  26. #ifdef _arch_dreamcast
  27.   return timer_ms_gettime64() ;
  28. #else
  29.   struct timeval tv ;
  30.   gettimeofday(&tv, NULL) ;
  31.   return tv.tv_sec * 1000 + tv.tv_usec / 1000 ;
  32. #endif
  33. }
  34.  
  35. #define READ_SIZE (20 * 1024 * 1024)
  36. #define READ_BUFFER_SIZE (512 * 1024)
  37.  
  38. int main()
  39. {
  40.   int i, j ;
  41.   int fd ;
  42.   char read_buffer[READ_BUFFER_SIZE] ;
  43.   off_t filesize ;
  44.   unsigned int start_time, end_time ;
  45.  
  46.   fd = open(FILENAME, O_RDONLY) ;
  47.   if (fd == -1)
  48.   {
  49.     printf("could not open %s\n", FILENAME) ;
  50.     return 1 ;
  51.   }
  52.   filesize = lseek(fd, 0, SEEK_END) ;
  53.  
  54.   for (i = 0 ; i <3 ; i++)
  55.   {
  56.     if (i == 0)
  57.     {
  58.       printf("reading inner 20 MB...\n") ;
  59.       lseek(fd, 0, SEEK_SET) ;
  60.     }
  61.     else if (i == 1)
  62.     {
  63.       printf("reading middle 20 MB...\n") ;
  64.       lseek(fd, (filesize / 2) - (READ_SIZE / 2), SEEK_SET) ;
  65.     }
  66.     else
  67.     {
  68.       printf("reading outer 20 MB...\n") ;
  69.       lseek(fd, filesize - READ_SIZE, SEEK_SET) ;
  70.     }
  71.     /* read 20 MB ; 40 chunks of 1/2 MB */
  72.     start_time = get_current_milliseconds() ;
  73.     for (j = 0 ; j <(READ_SIZE / READ_BUFFER_SIZE) ; j++)
  74.       if (read(fd, read_buffer, READ_BUFFER_SIZE) != READ_BUFFER_SIZE)
  75.       {
  76.         printf("read error\n") ;
  77.         break ;
  78.       }
  79.     end_time = get_current_milliseconds() ;
  80.     printf("%d - %d = %d ms => %d kbytes/sec\n",
  81.       end_time, start_time, end_time - start_time,
  82.       READ_SIZE / (end_time - start_time)) ;
  83.   }
  84.  
  85.   close(fd) ;
  86.  
  87.   return 0 ;
  88. }

• Multiprocess FATE Revisited

  26 juin 2010, par Multimedia Mike — FATE Server, Python

  I thought I had brainstormed a simple, elegant, multithreaded, deadlock-free refactoring for FATE in a previous post. However, I sort of glossed over the test ordering logic which I had not yet prototyped. The grim, possibly deadlock-afflicted reality is that the main thread needs to be notified as tests are completed. So, the main thread sends test specs through a queue to be executed by n tester threads and those threads send results to a results aggregator thread. Additionally, the results aggregator will need to send completed test IDs back to the main thread.

  
  
  

  But when I step back and look at the graph, I can’t rationalize why there should be a separate results aggregator thread. That was added to cut down on deadlock possibilities since the main thread and the tester threads would not be waiting for data from each other. Now that I’ve come to terms with the fact that the main and the testers need to exchange data in realtime, I think I can safely eliminate the result thread. Adding more threads is not the best way to guard against race conditions and deadlocks. Ask xine.

  
  
  

  I’m still hung up on the deadlock issue. I have these queues through which the threads communicate. At issue is the fact that they can cause a thread to block when inserting an item if the queue is "full". How full is full ? Immaterial ; seeking to answer such a question is not how you guard against race conditions. Rather, it seems to me that one side should be doing non-blocking queue operations.

  This is how I’m planning to revise the logic in the main thread :

  test_set = set of all tests to execute
  tests_pending = test_set
  tests_blocked = empty set
  tests_queue = multi-consumer queue to send test specs to tester threads
  results_queue = multi-producer queue through which tester threads send results
  while there are tests in tests_pending :
    pop a test from test_set
    if test depends on any tests that appear in tests_pending :
      add test to tests_blocked
    else :
      add test to tests_queue in a non-blocking manner
      if tests_queue is full, add test to tests_blocked
    while there are results in the results_queue :
  
      get a result from result_queue in non-blocking manner
  
      remove the corresponding test from tests_pending
  if tests_blocked is non-empty :
  
    sleep for 1 second
  
    test_set = tests_blocked
  
    tests_blocked = empty set
  
  else :
  
    insert n shutdown signals, one from each thread
  go to the top of the loop and repeat until there are no more tests
  while there are results in the results_queue :
  
    get a result from result_queue in a blocking manner
  

  Not mentioned in the pseudocode (so it doesn’t get too verbose) is logic to check whether the retrieved test result is actually an end-of-thread signal. These are accounted and the whole test process is done when one is received for each thread.

  On the tester thread side, it’s safe for them to do blocking test queue retrievals and blocking result queue insertions. The reason for the 1-second delay before resetting tests_blocked and looping again is because I want to guard against the situation where tests A and B are to be run, A depends of B running first, and while B is running (and happens to be a long encoding test), the main thread is spinning about, obsessively testing whether it’s time to insert A into the tests queue.

  It all sounds just crazy enough to work. In fact, I coded it up and it does work, sort of. The queue gets blocked pretty quickly. Instead of sleeping, I decided it’s better to perform the put operation using a 1-second timeout.

  Still, I’m paranoid about the precise operation of the IPC queue mechanism at work here. What happens if I try to stuff in a test spec that’s a bit too large ? Will the module take whatever I give it and serialize it through the queue as soon as it can ? I think an impromptu science project is in order.

  big-queue.py :

  PLAIN TEXT

  PYTHON :

  1. # !/usr/bin/python
  2.  
  3. import multiprocessing
  4. import Queue
  5.  
  6. def f(q) :
  7.   str = q.get()
  8.   print "reader function got a string of %d characters" % (len(str))
  9.  
  10. q = multiprocessing.Queue()
  11. p = multiprocessing.Process(target=f, args=(q,))
  12. p.start()
  13. try :
  14.   q.put_nowait(’a’ * 100000000)
  15. except Queue.Full :
  16.   print "queue full"

  $ ./big-queue.py
  reader function got a string of 100000000 characters
  

  Since 100 MB doesn’t even make it choke, FATE’s little test specs shouldn’t pose any difficulty.