Recherche avancée

Sur d’autres sites (13864)

• VP8 : a retrospective

  13 juillet 2010, par Dark Shikari — DCT, VP8, speed

  I’ve been working the past few weeks to help finish up the ffmpeg VP8 decoder, the first community implementation of On2′s VP8 video format. Now that I’ve written a thousand or two lines of assembly code and optimized a good bit of the C code, I’d like to look back at VP8 and comment on a variety of things — both good and bad — that slipped the net the first time, along with things that have changed since the time of that blog post.

  These are less-so issues related to compression — that issue has been beaten to death, particularly in MSU’s recent comparison, where x264 beat the crap out of VP8 and the VP8 developers pulled a Pinocchio in the developer comments. But that was expected and isn’t particularly interesting, so I won’t go into that. VP8 doesn’t have to be the best in the world in order to be useful.

  When the ffmpeg VP8 decoder is complete (just a few more asm functions to go), we’ll hopefully be able to post some benchmarks comparing it to libvpx.

  1. The spec, er, I mean, bitstream guide.

  Google has reneged on their claim that a spec existed at all and renamed it a “bitstream guide”. This is probably after it was found that — not merely was it incomplete — but at least a dozen places in the spec differed wildly from what was actually in their own encoder and decoder software ! The deblocking filter, motion vector clamping, probability tables, and many more parts simply disagreed flat-out with the spec. Fortunately, Ronald Bultje, one of the main authors of the ffmpeg VP8 decoder, is rather skilled at reverse-engineering, so we were able to put together a matching implementation regardless.

  Most of the differences aren’t particularly important — they don’t have a huge effect on compression or anything — but make it vastly more difficult to implement a “working” VP8 decoder, or for that matter, decide what “working” really is. For example, Google’s decoder will, if told to “swap the ALT and GOLDEN reference frames”, overwrite both with GOLDEN, because it first sets GOLDEN = ALT, and then sets ALT = GOLDEN. Is this a bug ? Or is this how it’s supposed to work ? It’s hard to tell — there isn’t a spec to say so. Google says that whatever libvpx does is right, but I doubt they intended this.

  I expect a spec will eventually be written, but it was a bit obnoxious of Google — both to the community and to their own developers — to release so early that they didn’t even have their own documentation ready.

  2. The TM intra prediction mode.

  One thing I glossed over in the original piece was that On2 had added an extra intra prediction mode to the standard batch that H.264 came with — they replaced Planar with “TM pred”. For i4x4, which didn’t have a Planar mode, they just added it without replacing an old one, resulting in a total of 10 modes to H.264′s 9. After understanding and writing assembly code for TM pred, I have to say that it is quite a cool idea. Here’s how it works :

  1. Let us take a block of size 4×4, 8×8, or 16×16.

  2. Define the pixels bordering the top of this block (starting from the left) as T[0], T[1], T[2]…

  3. Define the pixels bordering the left of this block (starting from the top) as L[0], L[1], L[2]…

  4. Define the pixel above the top-left of the block as TL.

  5. Predict every pixel <X,Y> in the block to be equal to clip3( T[X] + L[Y] – TL, 0, 255).

  It’s effectively a generalization of gradient prediction to the block level — predict each pixel based on the gradient between its top and left pixels, and the topleft. According to the VP8 devs, it’s chosen by the encoder quite a lot of the time, which isn’t surprising ; it seems like a pretty good idea. As just one more intra pred mode, it’s not going to do magic for compression, but it’s a cool idea and elegantly simple.

  3. Performance and the deblocking filter.
  

  On2 advertised for quite some that VP8′s goal was to be significantly faster to decode than H.264. When I saw the spec, I waited for the punchline, but apparently they were serious. There’s nothing wrong with being of similar speed or a bit slower — but I was rather confused as to the fact that their design didn’t match their stated goal at all. What apparently happened is they had multiple profiles of VP8 — high and low complexity profiles. They marketed the performance of the low complexity ones while touting the quality of the high complexity ones, a tad dishonest. More importantly though, practically nobody is using the low complexity modes, so anyone writing a decoder has to be prepared to handle the high complexity ones, which are the default.

  The primary time-eater here is the deblocking filter. VP8, being an H.264 derivative, has much the same problem as H.264 does in terms of deblocking — it spends an absurd amount of time there. As I write this post, we’re about to finish some of the deblocking filter asm code, but before it’s committed, up to 70% or more of total decoding time is spent in the deblocking filter ! Like H.264, it suffers from the 4×4 transform problem : a 4×4 transform requires a total of 8 length-16 and 8 length-8 loopfilter calls per macroblock, while Theora, with only an 8×8 transform, requires half that.

  This problem is aggravated in VP8 by the fact that the deblocking filter isn’t strength-adaptive ; if even one 4×4 block in a macroblock contains coefficients, every single edge has to be deblocked. Furthermore, the deblocking filter itself is quite complicated ; the “inner edge” filter is a bit more complex than H.264′s and the “macroblock edge” filter is vastly more complicated, having two entirely different codepaths chosen on a per-pixel basis. Of course, in SIMD, this means you have to do both and mask them together at the end.

  There’s nothing wrong with a good-but-slow deblocking filter. But given the amount of deblocking one needs to do in a 4×4-transform-based format, it might have been a better choice to make the filter simpler. It’s pretty difficult to beat H.264 on compression, but it’s certainly not hard to beat it on speed — and yet it seems VP8 missed a perfectly good chance to do so. Another option would have been to pick an 8×8 transform instead of 4×4, reducing the amount of deblocking by a factor of 2.

  And yes, there’s a simple filter available in the low complexity profile, but it doesn’t help if nobody uses it.

  4. Tree-based arithmetic coding.

  Binary arithmetic coding has become the standard entropy coding method for a wide variety of compressed formats, ranging from LZMA to VP6, H.264 and VP8. It’s simple, relatively fast compared to other arithmetic coding schemes, and easy to make adaptive. The problem with this is that you have to come up with a method for converting non-binary symbols into a list of binary symbols, and then choosing what probabilities to use to code each one. Here’s an example from H.264, the sub-partition mode symbol, which is either 8×8, 8×4, 4×8, or 4×4. encode_decision( context, bit ) writes a binary decision (bit) into a numbered context (context).

  8×8 : encode_decision( 21, 0 ) ;

  8×4 : encode_decision( 21, 1 ) ; encode_decision( 22, 0 ) ;

  4×8 : encode_decision( 21, 1 ) ; encode_decision( 22, 1 ) ; encode_decision( 23, 1 ) ;

  4×4 : encode_decision( 21, 1 ) ; encode_decision( 22, 1 ) ; encode_decision( 23, 0 ) ;

  As can be seen, this is clearly like a Huffman tree. Wouldn’t it be nice if we could represent this in the form of an actual tree data structure instead of code ? On2 thought so — they designed a simple system in VP8 that allowed all binarization schemes in the entire format to be represented as simple tree data structures. This greatly reduces the complexity — not speed-wise, but implementation-wise — of the entropy coder. Personally, I quite like it.

  5. The inverse transform ordering.

  I should at some point write a post about common mistakes made in video formats that everyone keeps making. These are not issues that are patent worries or huge issues for compression — just stupid mistakes that are repeatedly made in new video formats, probably because someone just never asked the guy next to him “does this look stupid ?” before sticking it in the spec.

  One common mistake is the problem of transform ordering. Every sane 2D transform is “separable” — that is, it can be done by doing a 1D transform vertically and doing the 1D transform again horizontally (or vice versa). The original iDCT as used in JPEG, H.263, and MPEG-1/2/4 was an “idealized” iDCT — nobody had to use the exact same iDCT, theirs just had to give very close results to a reference implementation. This ended up resulting in a lot of practical problems. It was also slow ; the only way to get an accurate enough iDCT was to do all the intermediate math in 32-bit.

  Practically every modern format, accordingly, has specified an exact iDCT. This includes H.264, VC-1, RV40, Theora, VP8, and many more. Of course, with an exact iDCT comes an exact ordering — while the “real” iDCT can be done in any order, an exact iDCT usually requires an exact order. That is, it specifies horizontal and then vertical, or vertical and then horizontal.

  All of these transforms end up being implemented in SIMD. In SIMD, a vertical transform is generally the only option, so a transpose is added to the process instead of doing a horizontal transform. Accordingly, there are two ways to do it :

  1. Transpose, vertical transform, transpose, vertical transform.

  2. Vertical transform, transpose, vertical transform, transpose.

  These may seem to be equally good, but there’s one catch — if the transpose is done first, it can be completely eliminated by merging it into the coefficient decoding process. On many modern CPUs, particularly x86, transposes are very expensive, so eliminating one of the two gives a pretty significant speed benefit.

  H.264 did it way 1).

  VC-1 did it way 1).

  Theora (inherited from VP3) did it way 1).

  But no. VP8 has to do it way 2), where you can’t eliminate the transpose. Bah. It’s not a huge deal ; probably only 1-2% overall at most speed-wise, but it’s just a needless waste. What really bugs me is that VP3 got it right — why in the world did they screw it up this time around if they got it right beforehand ?

  RV40 is the other modern format I know that made this mistake.

  (NB : You can do transforms without a transpose, but it’s generally not worth it unless the intermediate needs 32-bit math, as in the case of the “real” iDCT.)

  6. Not supporting interlacing.

  THANK YOU THANK YOU THANK YOU THANK YOU THANK YOU THANK YOU THANK YOU.

  Interlacing was the scourge of H.264. It weaseled its way into every nook and cranny of the spec, making every decoder a thousand lines longer. H.264 even included a highly complicated — and effective — dedicated interlaced coding scheme, MBAFF. The mere existence of MBAFF, despite its usefulness for broadcasters and others still stuck in the analog age with their 1080i, 576i , and 480i content, was a blight upon the video format.

  VP8 has once and for all avoided it.

  And if anyone suggests adding interlaced support to the experimental VP8 branch, find a straightjacket and padded cell for them before they cause any real damage.

• How to convert rtmp hevc video stream to srt av1 endpoint with ffmpeg ?

  20 juin 2024, par Lulík

  i want use ffmpeg to listen rtmp stream and send to srt endpoint.

  &#xA;

  Flow : smartphone (camera) -> laptop (ffmpeg script) -> desktop (obs studio)

  &#xA;

  ffmpeg script show warning message and in obs stuido i can see any video only audio.

  &#xA;

  Thank you in advance.

  &#xA;

  Console output while running script (error in the end is bcs i stoped sending data from phone) :

  &#xA;

  ffmpeg version git-2024-06-20-8d6014d Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --enable-libsvtav1 --enable-libsrt
  libavutil      59. 24.100 / 59. 24.100
  libavcodec     61.  8.100 / 61.  8.100
  libavformat    61.  3.104 / 61.  3.104
  libavdevice    61.  2.100 / 61.  2.100
  libavfilter    10.  2.102 / 10.  2.102
  libswscale      8.  2.100 /  8.  2.100
  libswresample   5.  2.100 /  5.  2.100
Input #0, flv, from 'rtmp://192.168.0.194/s/streamKey':
  Duration: 00:00:00.00, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: hevc (Main), yuv420p(tv, smpte170m/bt470bg/smpte170m), 1080x1920, 10240 kb/s, 30 fps, 120 tbr, 1k tbn
  Stream #0:1: Audio: aac (LC), 44100 Hz, stereo, fltp, 131 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> av1 (libsvtav1))
  Stream #0:1 -> #0:1 (aac (native) -> mp2 (native))
Press [q] to stop, [?] for help
Svt[info]: -------------------------------------------
Svt[info]: SVT [version]:   SVT-AV1 Encoder Lib 595a874
Svt[info]: SVT [build]  :   GCC 12.2.0   64 bit
Svt[info]: LIB Build date: Jun 20 2024 14:25:08
Svt[info]: -------------------------------------------
Svt[info]: Number of logical cores available: 12
Svt[info]: Number of PPCS 76
Svt[info]: [asm level on system : up to avx2]
Svt[info]: [asm level selected : up to avx2]
Svt[info]: -------------------------------------------
Svt[info]: SVT [config]: main profile   tier (auto) level (auto)
Svt[info]: SVT [config]: width / height / fps numerator / fps denominator       : 1080 / 1920 / 120 / 1
Svt[info]: SVT [config]: bit-depth / color format                   : 8 / YUV420
Svt[info]: SVT [config]: preset / tune / pred struct                    : 10 / PSNR / random access
Svt[info]: SVT [config]: gop size / mini-gop size / key-frame type          : 641 / 16 / key frame
Svt[info]: SVT [config]: BRC mode / rate factor                     : CRF / 35 
Svt[info]: SVT [config]: AQ mode / variance boost                   : 2 / 0
Svt[info]: -------------------------------------------
Svt[warn]: Failed to set thread priority: Invalid argument
Output #0, mpegts, to 'srt://192.168.0.167:9998?mode=caller':
  Metadata:
    encoder         : Lavf61.3.104
  Stream #0:0: Video: av1, yuv420p(tv, smpte170m/bt470bg/smpte170m, progressive), 1080x1920, q=2-31, 120 fps, 90k tbn
      Metadata:
        encoder         : Lavc61.8.100 libsvtav1
  Stream #0:1: Audio: mp2, 44100 Hz, stereo, s16, 384 kb/s
      Metadata:
        encoder         : Lavc61.8.100 mp2
[mpegts @ 0x55ec921d9540] Stream 0, codec av1, is muxed as a private data stream and may not be recognized upon reading.
[in#0/flv @ 0x55ec9219cc40] Error during demuxing: Input/output error1990.7kbits/s speed=0.967x    
[out#0/mpegts @ 0x55ec922247c0] video:4431KiB audio:1138KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 6.374870%
frame=  723 fps= 31 q=35.0 Lsize=    5923KiB time=00:00:24.12 bitrate=2011.3kbits/s speed=1.04x


  &#xA;

  I send video stream from mobile app over rtmp encoded with hevc to my laptop where running script ffmpeg -f flv -listen 1 -i rtmp://192.168.0.194/s/streamKey -c:v libsvtav1 -f mpegts srt://192.168.0.167:9998?mode=caller. On the desktop i have obs with media source input srt://192.168.0.167:9998?mode=listener.

  &#xA;

  When i run ffmpeg script without video codec option (-c:v libsvtav1) its working fine and in obs i can see video from my phone camera. With the option i can not see video only audio.&#xA;I clearly dont understand warning message : [mpegts @ 0x55ec921d9540] Stream 0, codec av1, is muxed as a private data stream and may not be recognized upon reading..&#xA;Do I need specify codec (av1) in obs media source or my ffmpeg script is wrong ?

  &#xA;

• Optical Drive Value Proposition

  28 août 2010, par Multimedia Mike — General

  I have the absolute worst luck in the optical drive department. Ever since I started building my own computers in 1995 — close to the beginning of the CD-ROM epoch — I have burned through a staggering number of optical drives. Seriously, especially in the time period between about 1995-1998, I was going through a new drive every 4-6 months or so. This was also during that CD-ROM speed race where the the drive packages kept advertising loftier ‘X’ speed ratings. I didn’t play a lot of CD-ROM games during that timeframe, though I did listen to quite a few audio CDs through the computer.

  
  
  

  I use “optical drive” as a general term to describe CD-ROM drives, CD-R/RW drives, DVD-ROM drives, DVD-R/RW drives, and drives capable of doing any combination of reading and writing CDs and DVDs. In my observation, optical media seems to be falling out of favor somewhat, giving way to online digital distribution for things like games and software, as well as flash drives and external hard drives vs. recordable or rewritable media for backup and sneakernet duty. Somewhere along the line, I started to buy computers that didn’t even have optical drives. That’s why I have purchased at least 2 external USB drives (seen in the picture above). I don’t have much confidence that either works correctly. My main desktop until recently, a Mac Mini, has an internal optical drive that grew flaky and unreliable a few months after the unit was purchased.

  I just have really rotten luck with optical drives. The most reliable drive in my house is the one on the headless machine that, until recently, was the main workhorse on the FATE farm. The eject switch didn’t work correctly so I have to log in remotely, 'sudo eject', walk to the other room, pop in the disc, walk back to the other room, and work with the disc.

  Maybe optical media is on its way out, but I still have many hundreds of CD-ROMs. Perhaps I should move forward on this brainstorm to archive all of my optical discs on hard drives (and then think of some data mining experiments, just for the academic appeal), before it’s too late ; optical discs don’t last forever.

  So if I needed a good optical drive, what should I consider ? I’ve always been the type to go cheap, I admit. Many of my optical drives were on the lower end of the cost spectrum, which might have played some role in their rapid replacement. However, I’m not sold on the idea that I’m getting quality just because I’m paying a higher price. That LG unit at the top of the pile up there was relatively pricey and still didn’t fare well in the long (or even medium) term.

  Come to think of it, I used to have a ridiculous stockpile of castoff (but somehow still functional) optical drives. So many, in fact, that in 2004 I had a full size PC tower that I filled with 4 working drives, just because I could. Okay, I admit that there was a period where I had some reliable drives.

  That might be an idea, actually– throw together such a computer for heavy duty archival purposes. I visited Weird Stuff Warehouse today (needed some PC100 RAM for an old machine and they came through) and I think I could put together such a box rather cheaply.

  It’s a dirty job, but… well, you know the rest.