Recherche avancée

Sur d’autres sites (12128)

• Why does the frame count change when scaling with FFmpeg ?

  22 octobre 2016, par ajmicek

  I use this to scale 1920x1080 H.264 videos :

  ffmpeg -i IMG_1438.MOV -threads 2 -vf scale=-2:600 IMG_1438_scaledTo600.MOV

  And it works great ! But here is my question : most of the time, the frame rate stays exactly the same between the original file and the scaled file. For example :

  $ mediainfo -F IMG_1426.MOV | grep Frame\ rate
  
  Frame rate                               : 29.970
  
  Frame rate                               : 29.970 FPS
  
  Frame rate mode                          : VFR
  
  Frame rate mode                          : Variable
  
  Frame rate                               : 29.970
  
  Frame rate                               : 29.970 (29970/1000) FPS
  
  
  
  $ mediainfo -F IMG_1426_scaledTo600.MOV | grep Frame\ rate
  
  Frame rate                               : 29.970
  
  Frame rate                               : 29.970 FPS
  
  Frame rate mode                          : CFR
  
  Frame rate mode                          : Constant
  
  Frame rate                               : 29.970
  
  Frame rate                               : 29.970 (30000/1001) FPS

  But sometimes, the frame rate increases dramatically :

  $ mediainfo -F IMG_1438.MOV | grep Frame\ rate
  
  Frame rate                               : 25.044
  
  Frame rate                               : 25.044 FPS
  
  Frame rate mode                          : VFR
  
  Frame rate mode                          : Variable
  
  Frame rate                               : 25.044
  
  Frame rate                               : 25.044 FPS
  
  
  
  $ mediainfo -F IMG_1438_scaledTo600.MOV | grep Frame\ rate
  
  Frame rate                               : 120.000
  
  Frame rate                               : 120.000 FPS
  
  Frame rate mode                          : CFR
  
  Frame rate mode                          : Constant
  
  Frame rate                               : 120.000
  
  Frame rate                               : 120.000 FPS

  What should I know about FFmpeg or libx264 or libswscale that will help me understand why this happens ? (Hoping to hear from LordNeckbeard, in particular).

  mediainfo IMG_1438.MOV --Full outputs :

  General
  
  Count                                    : 327
  
  Count of stream of this kind             : 1
  
  Kind of stream                           : General
  
  Kind of stream                           : General
  
  Stream identifier                        : 0
  
  Count of video streams                   : 1
  
  Count of audio streams                   : 1
  
  OtherCount                               : 2
  
  Video_Format_List                        : AVC
  
  Video_Format_WithHint_List               : AVC
  
  Codecs Video                             : AVC
  
  Audio_Format_List                        : AAC
  
  Audio_Format_WithHint_List               : AAC
  
  Audio codecs                             : AAC LC
  
  Complete name                            : IMG_1438.MOV
  
  File name                                : IMG_1438
  
  File extension                           : MOV
  
  Format                                   : MPEG-4
  
  Format                                   : MPEG-4
  
  Format/Extensions usually used           : mp4 m4v m4a m4b m4p 3gpp 3gp 3gpp2 3g2 k3g jpm jpx mqv ismv isma f4v
  
  Commercial name                          : MPEG-4
  
  Format profile                           : QuickTime
  
  Internet media type                      : video/mp4
  
  Codec ID                                 : qt  
  
  Codec ID                                 : qt   0000.00 (qt  )
  
  Codec ID/Url                             : http://www.apple.com/quicktime/download/standalone.html
  
  CodecID_Version                          : 0000.00
  
  CodecID_Compatible                       : qt  
  
  Codec                                    : MPEG-4
  
  Codec                                    : MPEG-4
  
  Codec/Extensions usually used            : mp4 m4v m4a m4b m4p 3gpp 3gp 3gpp2 3g2 k3g jpm jpx mqv ismv isma f4v
  
  File size                                : 113990140
  
  File size                                : 109 MiB
  
  File size                                : 109 MiB
  
  File size                                : 109 MiB
  
  File size                                : 109 MiB
  
  File size                                : 108.7 MiB
  
  Duration                                 : 52268
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 00:00:52.268
  
  Duration                                 : 00:00:52:09
  
  Duration                                 : 00:00:52.268 (00:00:52:09)
  
  Overall bit rate                         : 17447026
  
  Overall bit rate                         : 17.4 Mb/s
  
  Frame rate                               : 25.044
  
  Frame rate                               : 25.044 FPS
  
  Frame count                              : 1309
  
  Stream size                              : 56670
  
  Stream size                              : 55.3 KiB (0%)
  
  Stream size                              : 55 KiB
  
  Stream size                              : 55 KiB
  
  Stream size                              : 55.3 KiB
  
  Stream size                              : 55.34 KiB
  
  Stream size                              : 55.3 KiB (0%)
  
  Proportion of this stream                : 0.00050
  
  HeaderSize                               : 28
  
  DataSize                                 : 113966271
  
  FooterSize                               : 23841
  
  IsStreamable                             : No
  
  Encoded date                             : UTC 2016-10-08 22:51:19
  
  Tagged date                              : UTC 2016-10-08 22:52:12
  
  File last modification date              : UTC 2016-10-08 22:51:19
  
  File last modification date (local)      : 2016-10-08 17:51:19
  
  Writing library                          : Apple QuickTime
  
  Writing library                          : Apple QuickTime
  
  Encoded_Library_Name                     : Apple QuickTime
  
  com.apple.quicktime.make                 : Apple
  
  com.apple.quicktime.model                : iPhone 5
  
  com.apple.quicktime.software             : 10.0.2
  
  com.apple.quicktime.creationdate         : 2016-10-08T17:51:19-0500
  
  
  
  Video
  
  Count                                    : 334
  
  Count of stream of this kind             : 1
  
  Kind of stream                           : Video
  
  Kind of stream                           : Video
  
  Stream identifier                        : 0
  
  StreamOrder                              : 0
  
  ID                                       : 1
  
  ID                                       : 1
  
  Format                                   : AVC
  
  Format/Info                              : Advanced Video Codec
  
  Format/Url                               : http://developers.videolan.org/x264.html
  
  Commercial name                          : AVC
  
  Format profile                           : High@L4.1
  
  Format settings                          : CABAC / 1 Ref Frames
  
  Format settings, CABAC                   : Yes
  
  Format settings, CABAC                   : Yes
  
  Format settings, ReFrames                : 1
  
  Format settings, ReFrames                : 1 frame
  
  Internet media type                      : video/H264
  
  Codec ID                                 : avc1
  
  Codec ID/Info                            : Advanced Video Coding
  
  Codec ID/Url                             : http://www.apple.com/quicktime/download/standalone.html
  
  Codec                                    : AVC
  
  Codec                                    : AVC
  
  Codec/Family                             : AVC
  
  Codec/Info                               : Advanced Video Codec
  
  Codec/Url                                : http://developers.videolan.org/x264.html
  
  Codec/CC                                 : avc1
  
  Codec profile                            : High@L4.1
  
  Codec settings                           : CABAC / 1 Ref Frames
  
  Codec settings, CABAC                    : Yes
  
  Codec_Settings_RefFrames                 : 1
  
  Duration                                 : 52268
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 00:00:52.268
  
  Duration                                 : 00:00:52:09
  
  Duration                                 : 00:00:52.268 (00:00:52:09)
  
  Bit rate                                 : 17375530
  
  Bit rate                                 : 17.4 Mb/s
  
  Width                                    : 1920
  
  Width                                    : 1 920 pixels
  
  Height                                   : 1080
  
  Height                                   : 1 080 pixels
  
  Stored_Height                            : 1088
  
  Sampled_Width                            : 1920
  
  Sampled_Height                           : 1080
  
  Pixel aspect ratio                       : 1.000
  
  Display aspect ratio                     : 1.778
  
  Display aspect ratio                     : 16:9
  
  Rotation                                 : 90.000
  
  Rotation                                 : 90°
  
  Frame rate mode                          : VFR
  
  Frame rate mode                          : Variable
  
  Frame rate                               : 25.044
  
  Frame rate                               : 25.044 FPS
  
  Minimum frame rate                       : 23.077
  
  Minimum frame rate                       : 23.077 FPS
  
  Maximum frame rate                       : 30.000
  
  Maximum frame rate                       : 30.000 FPS
  
  Frame count                              : 1309
  
  Resolution                               : 8
  
  Resolution                               : 8 bits
  
  Colorimetry                              : 4:2:0
  
  Color space                              : YUV
  
  Chroma subsampling                       : 4:2:0
  
  Chroma subsampling                       : 4:2:0
  
  Bit depth                                : 8
  
  Bit depth                                : 8 bits
  
  Scan type                                : Progressive
  
  Scan type                                : Progressive
  
  Interlacement                            : PPF
  
  Interlacement                            : Progressive
  
  Bits/(Pixel*Frame)                       : 0.335
  
  Stream size                              : 113523046
  
  Stream size                              : 108 MiB (100%)
  
  Stream size                              : 108 MiB
  
  Stream size                              : 108 MiB
  
  Stream size                              : 108 MiB
  
  Stream size                              : 108.3 MiB
  
  Stream size                              : 108 MiB (100%)
  
  Proportion of this stream                : 0.99590
  
  Title                                    : Core Media Video
  
  Encoded date                             : UTC 2016-10-08 22:51:19
  
  Tagged date                              : UTC 2016-10-08 22:52:12
  
  Color range                              : Limited
  
  colour_description_present               : Yes
  
  Color primaries                          : BT.709
  
  Transfer characteristics                 : BT.709
  
  Matrix coefficients                      : BT.709
  
  
  
  Audio
  
  Count                                    : 272
  
  Count of stream of this kind             : 1
  
  Kind of stream                           : Audio
  
  Kind of stream                           : Audio
  
  Stream identifier                        : 0
  
  StreamOrder                              : 1
  
  ID                                       : 2
  
  ID                                       : 2
  
  Format                                   : AAC
  
  Format/Info                              : Advanced Audio Codec
  
  Commercial name                          : AAC
  
  Format profile                           : LC
  
  Codec ID                                 : 40
  
  Codec                                    : AAC LC
  
  Codec                                    : AAC LC
  
  Codec/Family                             : AAC
  
  Codec/CC                                 : 40
  
  Duration                                 : 52268
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 00:00:52.268
  
  Duration                                 : 00:00:52:15
  
  Duration                                 : 00:00:52.268 (00:00:52:15)
  
  Source duration                          : 52338
  
  Source duration                          : 52 s 338 ms
  
  Source duration                          : 52 s 338 ms
  
  Source duration                          : 52 s 338 ms
  
  Source duration                          : 00:00:52.338
  
  Bit rate mode                            : CBR
  
  Bit rate mode                            : Constant
  
  Bit rate                                 : 64000
  
  Bit rate                                 : 64.0 kb/s
  
  Channel(s)                               : 1
  
  Channel(s)                               : 1 channel
  
  Channel positions                        : Front: C
  
  Channel positions                        : 1/0/0
  
  ChannelLayout                            : C
  
  Samples per frame                        : 1024
  
  Sampling rate                            : 44100
  
  Sampling rate                            : 44.1 kHz
  
  Samples count                            : 2305019
  
  Frame rate                               : 43.066
  
  Frame rate                               : 43.066 FPS (1024 spf)
  
  Frame count                              : 2251
  
  Source frame count                       : 2254
  
  Compression mode                         : Lossy
  
  Compression mode                         : Lossy
  
  Stream size                              : 410424
  
  Stream size                              : 401 KiB (0%)
  
  Stream size                              : 401 KiB
  
  Stream size                              : 401 KiB
  
  Stream size                              : 401 KiB
  
  Stream size                              : 400.8 KiB
  
  Stream size                              : 401 KiB (0%)
  
  Proportion of this stream                : 0.00360
  
  Source stream size                       : 410894
  
  Source stream size                       : 401 KiB (0%)
  
  Source stream size                       : 401 KiB
  
  Source stream size                       : 401 KiB
  
  Source stream size                       : 401 KiB
  
  Source stream size                       : 401.3 KiB
  
  Source stream size                       : 401 KiB (0%)
  
  Source_StreamSize_Proportion             : 0.00360
  
  Title                                    : Core Media Audio
  
  Encoded date                             : UTC 2016-10-08 22:51:19
  
  Tagged date                              : UTC 2016-10-08 22:52:12
  
  
  
  Other #1
  
  Count                                    : 112
  
  Count of stream of this kind             : 2
  
  Kind of stream                           : Other
  
  Kind of stream                           : Other
  
  Stream identifier                        : 0
  
  Stream identifier                        : 1
  
  Type                                     : meta
  
  Duration                                 : 52268
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 00:00:52.268
  
  Duration                                 : 00:00:52.268
  
  Frame count                              : 6
  
  Bit rate mode                            : VBR
  
  
  
  Other #2
  
  Count                                    : 112
  
  Count of stream of this kind             : 2
  
  Kind of stream                           : Other
  
  Kind of stream                           : Other
  
  Stream identifier                        : 1
  
  Stream identifier                        : 2
  
  Type                                     : meta
  
  Duration                                 : 52268
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 52 s 268 ms
  
  Duration                                 : 00:00:52.268
  
  Duration                                 : 00:00:52.268
  
  Frame count                              : 1
  
  Bit rate mode                            : CBR

  and ffprobe IMG_1438.MOV outputs :

  ffprobe version 3.1.3 Copyright (c) 2007-2016 the FFmpeg developers
  
    built with Apple LLVM version 7.3.0 (clang-703.0.31)
  
    configuration: --prefix=/usr/local/Cellar/ffmpeg/3.1.3 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-opencl --enable-libx264 --enable-libmp3lame --enable-libxvid --disable-lzma --enable-vda
  
    libavutil      55. 28.100 / 55. 28.100
  
    libavcodec     57. 48.101 / 57. 48.101
  
    libavformat    57. 41.100 / 57. 41.100
  
    libavdevice    57.  0.101 / 57.  0.101
  
    libavfilter     6. 47.100 /  6. 47.100
  
    libavresample   3.  0.  0 /  3.  0.  0
  
    libswscale      4.  1.100 /  4.  1.100
  
    libswresample   2.  1.100 /  2.  1.100
  
    libpostproc    54.  0.100 / 54.  0.100
  
  Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'IMG_1438.MOV':
  
    Metadata:
  
      major_brand     : qt  
  
      minor_version   : 0
  
      compatible_brands: qt  
  
      creation_time   : 2016-10-08 22:51:19
  
      com.apple.quicktime.make: Apple
  
      com.apple.quicktime.model: iPhone 5
  
      com.apple.quicktime.software: 10.0.2
  
      com.apple.quicktime.creationdate: 2016-10-08T17:51:19-0500
  
    Duration: 00:00:52.27, start: 0.000000, bitrate: 17446 kb/s
  
      Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080, 17375 kb/s, 25.04 fps, 120 tbr, 600 tbn, 1200 tbc (default)
  
      Metadata:
  
        rotate          : 90
  
        creation_time   : 2016-10-08 22:51:19
  
        handler_name    : Core Media Data Handler
  
        encoder         : H.264
  
      Side data:
  
        displaymatrix: rotation of -90.00 degrees
  
      Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, mono, fltp, 62 kb/s (default)
  
      Metadata:
  
        creation_time   : 2016-10-08 22:51:19
  
        handler_name    : Core Media Data Handler
  
      Stream #0:2(und): Data: none (mebx / 0x7862656D), 0 kb/s (default)
  
      Metadata:
  
        creation_time   : 2016-10-08 22:51:19
  
        handler_name    : Core Media Data Handler
  
      Stream #0:3(und): Data: none (mebx / 0x7862656D), 0 kb/s (default)
  
      Metadata:
  
        creation_time   : 2016-10-08 22:51:19
  
        handler_name    : Core Media Data Handler
  
  Unsupported codec with id 0 for input stream 2
  
  Unsupported codec with id 0 for input stream 3

  UPDATE
  To clarify : my video above, the one with the high framerate (120 FPS) output after scaling, plays perfectly before and after scaling with FFmpeg (no sync issues, and 120 FPS is only about 14% larger in file size), I am simply trying to understand why this increase in framerate happens (just a little beyond Mulvya’s note that the framerate stored in the container is wrong).

  From a programming perspective, the initial issue I ran into was that I was using frame= from FFmpeg’s sterr console output to determine progress, which reports erroneous results when the frame count increases dramatically on output ("I’m 372% done encoding ?!") ; I have since read another stackoverflow answer and changed my code to use time=, which appears to be a more robust way for me to display FFmpeg progress. (Also, there is FFmpeg’s -progress option, of course).

  Improving on the original command

  My new command to scale, preserve a useful framerate, and optimize threads :

  ffmpeg -i IMG_1438.MOV -vf scale=-2:600 -r 30 -vsync 0 IMG_1438_scaledTo600.MOV

  Where 30 is the "Maximum frame rate" from mediainfo.

  Thanks to help in the comments, I now know I do not fully understand FFmpeg’s use of three different time bases for timestamps : tbn, tbc, and tbr.
  They were explained by Robert Swain in 2009 and his explanation was also used to answer a Stackoverflow question about tbn, tbc, tbr.

  It sounds to me, as I’m pulling together comments from Mulvya below and Michael Rampe at another forum, that tbr is guessed ; it is frequently but not always the best value to use when changing from a variable to a constant frame rate video.

  Which leaves these 2 questions...

  (1) tbr is incorrect when "field rate and frame rate" differ ? Does this happen a lot ?
  (2) Is -r 30 where 30 is the maximum frame rate reported by mediainfo the best way to do it for most codec/container combinations ? (Or should I only use this method when I am scaling a H.264/MPEG-4 AVC video ?)

• Processing Big Data Problems

  8 janvier 2011, par Multimedia Mike — Big Data

  I’m becoming more interested in big data problems, i.e., extracting useful information out of absurdly sized sets of input data. I know it’s a growing field and there is a lot to read on the subject. But you know how I roll— just think of a problem to solve and dive right in.

  Here’s how my adventure unfolded.

  The Corpus
  I need to run a command line program on a set of files I have collected. This corpus is on the order of 350,000 files. The files range from 7 bytes to 175 MB. Combined, they occupy around 164 GB of storage space.

  Oh, and said storage space resides on an external, USB 2.0-connected hard drive. Stop laughing.

  A file is named according to the SHA-1 hash of its data. The files are organized in a directory hierarchy according to the first 6 hex digits of the SHA-1 hash (e.g., a file named a4d5832f... is stored in a4/d5/83/a4d5832f...). All of this file hash, path, and size information is stored in an SQLite database.

  First Pass
  I wrote a Python script that read all the filenames from the database, fed them into a pool of worker processes using Python’s multiprocessing module, and wrote some resulting data for each file back to the SQLite database. My Eee PC has a single-core, hyperthreaded Atom which presents 2 CPUs to the system. Thus, 2 worker threads crunched the corpus. It took awhile. It took somewhere on the order of 9 or 10 or maybe even 12 hours. It took long enough that I’m in no hurry to re-run the test and get more precise numbers.

  At least I extracted my initial set of data from the corpus. Or did I ?

  Think About The Future
  
  A few days later, I went back to revisit the data only to notice that the SQLite database was corrupted. To add insult to that bit of injury, the script I had written to process the data was also completely corrupted (overwritten with something unrelated to Python code). BTW, this is was on a RAID brick configured for redundancy. So that’s strike 3 in my personal dealings with RAID technology.

  I moved the corpus to a different external drive and also verified the files after writing (easy to do since I already had the SHA-1 hashes on record).

  The corrupted script was pretty simple to rewrite, even a little better than before. Then I got to re-run it. However, this run was on a faster machine, a hyperthreaded, quad-core beast that exposes 8 CPUs to the system. The reason I wasn’t too concerned about the poor performance with my Eee PC is that I knew I was going to be able to run in on this monster later.

  So I let the rewritten script rip. The script gave me little updates regarding its progress. As it did so, I ran some rough calculations and realized that it wasn’t predicted to finish much sooner than it would have if I were running it on the Eee PC.

  Limiting Factors
  It had been suggested to me that I/O bandwidth of the external USB drive might be a limiting factor. This is when I started to take that idea very seriously.

  The first idea I had was to move the SQLite database to a different drive. The script records data to the database for every file processed, though it only commits once every 100 UPDATEs, so at least it’s not constantly syncing the disc. I ran before and after tests with a small subset of the corpus and noticed a substantial speedup thanks to this policy chance.

  Then I remembered hearing something about "atime" which is access time. Linux filesystems, per default, record the time that a file was last accessed. You can watch this in action by running 'stat <file> ; cat <file> > /dev/null ; stat <file>' and observe that the "Access" field has been updated to NOW(). This also means that every single file that gets read from the external drive still causes an additional write. To avoid this, I started mounting the external drive with '-o noatime' which instructs Linux not to record "last accessed" time for files.

  On the limited subset test, this more than doubled script performance. I then wondered about mounting the external drive as read-only. This had the same performance as noatime. I thought about using both options together but verified that access times are not updated for a read-only filesystem.

  A Note On Profiling
  Once you start accessing files in Linux, those files start getting cached in RAM. Thus, if you profile, say, reading a gigabyte file from a disk and get 31 MB/sec, and then repeat the same test, you’re likely to see the test complete instantaneously. That’s because the file is already sitting in memory, cached. This is useful in general application use, but not if you’re trying to profile disk performance.

  Thus, in between runs, do (as root) 'sync; echo 3 > /proc/sys/vm/drop_caches' in order to wipe caches (explained here).

  Even Better ?
  I re-ran the test using these little improvements. Now it takes somewhere around 5 or 6 hours to run.

  I contrived an artificially large file on the external drive and did some 'dd' tests to measure what the drive could really do. The drive consistently measured a bit over 31 MB/sec. If I could read and process the data at 30 MB/sec, the script would be done in about 95 minutes.

  But it’s probably rather unreasonable to expect that kind of transfer rate for lots of smaller files scattered around a filesystem. However, it can’t be that helpful to have 8 different processes constantly asking the HD for 8 different files at any one time.

  So I wrote a script called stream-corpus.py which simply fetched all the filenames from the database and loaded the contents of each in turn, leaving the data to be garbage-collected at Python’s leisure. This test completed in 174 minutes, just shy of 3 hours. I computed an average read speed of around 17 MB/sec.

  Single-Reader Script
  I began to theorize that if I only have one thread reading, performance should improve greatly. To test this hypothesis without having to do a lot of extra work, I cleared the caches and ran stream-corpus.py until 'top' reported that about half of the real memory had been filled with data. Then I let the main processing script loose on the data. As both scripts were using sorted lists of files, they iterated over the filenames in the same order.

  Result : The processing script tore through the files that had obviously been cached thanks to stream-corpus.py, degrading drastically once it had caught up to the streaming script.

  Thus, I was incented to reorganize the processing script just slightly. Now, there is a reader thread which reads each file and stuffs the name of the file into an IPC queue that one of the worker threads can pick up and process. Note that no file data is exchanged between threads. No need— the operating system is already implicitly holding onto the file data, waiting in case someone asks for it again before something needs that bit of RAM. Technically, this approach accesses each file multiple times. But it makes little practical difference thanks to caching.

  Result : About 183 minutes to process the complete corpus (which works out to a little over 16 MB/sec).

  Why Multiprocess
  Is it even worthwhile to bother multithreading this operation ? Monitoring the whole operation via 'top', most instances of the processing script are barely using any CPU time. Indeed, it’s likely that only one of the worker threads is doing any work most of the time, pulling a file out of the IPC queue as soon the reader thread triggers its load into cache. Right now, the processing is usually pretty quick. There are cases where the processing (external program) might hang (one of the reasons I’m running this project is to find those cases) ; the multiprocessing architecture at least allows other processes to take over until a hanging process is timed out and killed by its monitoring process.

  Further, the processing is pretty simple now but is likely to get more intensive in future iterations. Plus, there’s the possibility that I might move everything onto a more appropriately-connected storage medium which should help alleviate the bottleneck bravely battled in this post.

  There’s also the theoretical possibility that the reader thread could read too far ahead of the processing threads. Obviously, that’s not too much of an issue in the current setup. But to guard against it, the processes could share a variable that tracks the total number of bytes that have been processed. The reader thread adds filesizes to the count while the processing threads subtract file sizes. The reader thread would delay reading more if the number got above a certain threshold.

  Leftovers
  I wondered if the order of accessing the files mattered. I didn’t write them to the drive in any special order. The drive is formatted with Linux ext3. I ran stream-corpus.py on all the filenames sorted by filename (remember the SHA-1 naming convention described above) and also by sorting them randomly.

  Result : It helps immensely for the filenames to be sorted. The sorted variant was a little more than twice as fast as the random variant. Maybe it has to do with accessing all the files in a single directory before moving onto another directory.

  Further, I have long been under the impression that the best read speed you can expect from USB 2.0 was 27 Mbytes/sec (even though 480 Mbit/sec is bandied about in relation to the spec). This comes from profiling I performed with an external enclosure that supports both USB 2.0 and FireWire-400 (and eSata). FW-400 was able to read the same file at nearly 40 Mbytes/sec that USB 2.0 could only read at 27 Mbytes/sec. Other sources I have read corroborate this number. But this test (using different hardware), achieved over 31 Mbytes/sec.

• Révision 20021 : retour sur r20020 : il faut tout de meme verifier l’existence d’un champ statut ...

  22 novembre 2012, par cedric -

  + ajout d’un argument $set dans objet_inserer qui permet d’initialiser certains champs a la creation (exemple d’une cle unique : on ne veut pas passer par une valeur vide car en cas d’insertions concourantes on risque l’echec, ce qui n’est pas le cas si on fournit directement la valeur lors de (...)