Recherche avancée

Sur d’autres sites (6307)

• Anatomy of an optimization : H.264 deblocking

  26 mai 2010, par Dark Shikari — H.264, assembly, development, speed, x264

  As mentioned in the previous post, H.264 has an adaptive deblocking filter. But what exactly does that mean — and more importantly, what does it mean for performance ? And how can we make it as fast as possible ? In this post I’ll try to answer these questions, particularly in relation to my recent deblocking optimizations in x264.

  H.264′s deblocking filter has two steps : strength calculation and the actual filter. The first step calculates the parameters for the second step. The filter runs on all the edges in each macroblock. That’s 4 vertical edges of length 16 pixels and 4 horizontal edges of length 16 pixels. The vertical edges are filtered first, from left to right, then the horizontal edges, from top to bottom (order matters !). The leftmost edge is the one between the current macroblock and the left macroblock, while the topmost edge is the one between the current macroblock and the top macroblock.

  Here’s the formula for the strength calculation in progressive mode. The highest strength that applies is always selected.

  If we’re on the edge between an intra macroblock and any other macroblock : Strength 4
  If we’re on an internal edge of an intra macroblock : Strength 3
  If either side of a 4-pixel-long edge has residual data : Strength 2
  If the motion vectors on opposite sides of a 4-pixel-long edge are at least a pixel apart (in either x or y direction) or the reference frames aren’t the same : Strength 1
  Otherwise : Strength 0 (no deblocking)

  These values are then thrown into a lookup table depending on the quantizer : higher quantizers have stronger deblocking. Then the actual filter is run with the appropriate parameters. Note that Strength 4 is actually a special deblocking mode that performs a much stronger filter and affects more pixels.

  One can see somewhat intuitively why these strengths are chosen. The deblocker exists to get rid of sharp edges caused by the block-based nature of H.264, and so the strength depends on what exists that might cause such sharp edges. The strength calculation is a way to use existing data from the video stream to make better decisions during the deblocking process, improving compression and quality.

  Both the strength calculation and the actual filter (not described here) are very complex if naively implemented. The latter can be SIMD’d with not too much difficulty ; no H.264 decoder can get away with reasonable performance without such a thing. But what about optimizing the strength calculation ? A quick analysis shows that this can be beneficial as well.

  Since we have to check both horizontal and vertical edges, we have to check up to 32 pairs of coefficient counts (for residual), 16 pairs of reference frame indices, and 128 motion vector values (counting x and y as separate values). This is a lot of calculation ; a naive implementation can take 500-1000 clock cycles on a modern CPU. Of course, there’s a lot of shortcuts we can take. Here’s some examples :

  • If the macroblock uses the 8×8 transform, we only need to check 2 edges in each direction instead of 4, because we don’t deblock inside of the 8×8 blocks.
  • If the macroblock is a P-skip, we only have to check the first edge in each direction, since there’s guaranteed to be no motion vector differences, reference frame differences, or residual inside of the macroblock.
  • If the macroblock has no residual at all, we can skip that check.
  • If we know the partition type of the macroblock, we can do motion vector checks only along the edges of the partitions.
  • If the effective quantizer is so low that no deblocking would be performed no matter what, don’t bother calculating the strength.

  But even all of this doesn’t save us from ourselves. We still have to iterate over a ton of edges, checking each one. Stuff like the partition-checking logic greatly complicates the code and adds overhead even as it reduces the number of checks. And in many cases decoupling the checks to add such logic will make it slower : if the checks are coupled, we can avoid doing a motion vector check if there’s residual, since Strength 2 overrides Strength 1.

  But wait. What if we could do this in SIMD, just like the actual loopfilter itself ? Sure, it seems more of a problem for C code than assembly, but there aren’t any obvious things in the way. Many years ago, Loren Merritt (pengvado) wrote the first SIMD implementation that I know of (for ffmpeg’s decoder) ; it is quite fast, so I decided to work on porting the idea to x264 to see if we could eke out a bit more speed here as well.

  Before I go over what I had to do to make this change, let me first describe how deblocking is implemented in x264. Since the filter is a loopfilter, it acts “in loop” and must be done in both the encoder and decoder — hence why x264 has it too, not just decoders. At the end of encoding one row of macroblocks, x264 goes back and deblocks the row, then performs half-pixel interpolation for use in encoding the next frame.

  We do it per-row for reasons of cache coherency : deblocking accesses a lot of pixels and a lot of code that wouldn’t otherwise be used, so it’s more efficient to do it in a single pass as opposed to deblocking each macroblock immediately after encoding. Then half-pixel interpolation can immediately re-use the resulting data.

  Now to the change. First, I modified deblocking to implement a subset of the macroblock_cache_load function : spend an extra bit of effort loading the necessary data into a data structure which is much simpler to address — as an assembly implementation would need (x264_macroblock_cache_load_deblock). Then I massively cleaned up deblocking to move all of the core strength-calculation logic into a single, small function that could be converted to assembly (deblock_strength_c). Finally, I wrote the assembly functions and worked with Loren to optimize them. Here’s the result.

  And the timings for the resulting assembly function on my Core i7, in cycles :

  deblock_strength_c : 309
  deblock_strength_mmx : 79
  deblock_strength_sse2 : 37
  deblock_strength_ssse3 : 33

  Now that is a seriously nice improvement. 33 cycles on average to perform that many comparisons–that’s absurdly low, especially considering the SIMD takes no branchy shortcuts : it always checks every single edge ! I walked over to my performance chart and happily crossed off a box.

  But I had a hunch that I could do better. Remember, as mentioned earlier, we’re reloading all that data back into our data structures in order to address it. This isn’t that slow, but takes enough time to significantly cut down on the gain of the assembly code. And worse, less than a row ago, all this data was in the correct place to be used (when we just finished encoding the macroblock) ! But if we did the deblocking right after encoding each macroblock, the cache issues would make it too slow to be worth it (yes, I tested this). So I went back to other things, a bit annoyed that I couldn’t get the full benefit of the changes.

  Then, yesterday, I was talking with Pascal, a former Xvid dev and current video hacker over at Google, about various possible x264 optimizations. He had seen my deblocking changes and we discussed that a bit as well. Then two lines hit me like a pile of bricks :

  <_skal_> tried computing the strength at least ?
  <_skal_> while it’s fresh

  Why hadn’t I thought of that ? Do the strength calculation immediately after encoding each macroblock, save the result, and then go pick it up later for the main deblocking filter. Then we can use the data right there and then for strength calculation, but we don’t have to do the whole deblock process until later.

  I went and implemented it and, after working my way through a horde of bugs, eventually got a working implementation. A big catch was that of slices : deblocking normally acts between slices even though normal encoding does not, so I had to perform extra munging to get that to work. By midday today I was able to go cross yet another box off on the performance chart. And now it’s committed.

  Sometimes chatting for 10 minutes with another developer is enough to spot the idea that your brain somehow managed to miss for nearly a straight week.

  NB : the performance chart is on a specific test clip at a specific set of settings (super fast settings) relevant to the company I work at, so it isn’t accurate nor complete for, say, default settings.

  Update : Here’s a higher resolution version of the current chart, as requested in the comments.
  

  http://git.videolan.org/?p=x264.git ;a=commit ;h=b9ec3ea27812288485329ac7771143021cd4917b

• Padding thumbnail with color

  9 juin 2010, par Mikko Koppanen — Imagick, PHP stuff

  I know, it’s been a while since I last blogged. This is because a lot of things are happening in my personal life. I recently relocated to London from Finland and started a new job. Things are quite busy but I will try to post an example now and then. In the meanwhile I would like to hear about sites using Imagick, so if your project is not super secret please post an url and maybe a small explanation what you’re doing with Imagick on the site. This is purely for my personal interest.

  Anyway, to the point. Today’s example originates from a question asked by a user. How do I thumbnail the image inside given dimensions proportionally and fill the “blank” areas with a color ? Well, the answer is here

  The code is for Imagick 2.1.0 but adapting to older versions should not be hard.

  php

  < view plain text >

  1. < ?php
  2. /* Define width and height of the thumbnail */
  3. $width = 100 ;
  4. $height = 100 ;
  5.  
  6. /* Instanciate and read the image in */
  7. $im = new Imagick( "test.png" ) ;
  8.  
  9. /* Fit the image into $width x $height box
  10.  The third parameter fits the image into a "bounding box" */
  11. $im->thumbnailImage( $width, $height, true ) ;
  12.  
  13. /* Create a canvas with the desired color */
  14. $canvas = new Imagick() ;
  15. $canvas->newImage( $width, $height, ’pink’, ’png’ ) ;
  16.  
  17. /* Get the image geometry */
  18. $geometry = $im->getImageGeometry() ;
  19.  
  20. /* The overlay x and y coordinates */
  21. $x = ( $width - $geometry[’width’] ) / 2 ;
  22. $y = ( $height - $geometry[’height’] ) / 2 ;
  23.  
  24. /* Composite on the canvas */
  25. $canvas->compositeImage( $im, imagick: :COMPOSITE_OVER, $x, $y ) ;
  26.  
  27. /* Output the image*/
  28. header( "Content-Type : image/png" ) ;
  29. echo $canvas ;
  30.  
  31.  ?>

  The source image :
  

  The resulting image :
  

• Raspberry Pi 4 live streaming with ffmpeg [closed]

  12 décembre 2019, par Berri

  So speedify created a blog post and youtube video about making an IRL streaming backpack using the Elgato Cam Link 4k, Raspberry Pi 4, and ffmpeg.

  They gave pretty detailed instructions, and included downloads to prebuilt scripts/commands to get it all running once put together.
  Blog post :
  https://speedify.com/blog/how-to/build-irl-streaming-backpack-complete-guide/

  ffmpeg command from post :

  ffmpeg_command = “/home/pi/bin/ffmpeg -nostdin -re -f v4l2 -s ‘1280×720’ -framerate 24 -i /dev/video0 -f alsa -ac 2 -i hw:CARD=Link,DEV=0 -vcodec libx264 -framerate 24 -rtbufsize 1500k -s 1280×720 -preset ultrafast -pix_fmt yuv420p -crf 17 -force_key_frames ‘expr:gte(t,n_forced*2)’ -minrate 850k -maxrate 1000k -b:v 1000k -bufsize 1000k -acodec libmp3lame -rtbufsize 1500k -b 96k -ar 44100 -f flv – | ffmpeg -f flv -i – -c copy -f flv -drop_pkts_on_overflow 1 -attempt_recovery 1 -recovery_wait_time 1 rtmp://live.twitch.tv/app/live_” + streamKey + “‘\n” 

  I replaced -i hw:card=link,dev=0 in that command with -i hw:2,0 because -i hw:card=link,dev=0 gave me "file does not exist" errors in the log. "streamkey" is filled with the appropriate key for my twitch.

  Github Resources + Instructions used :
  https://github.com/speedify/rpi-streaming-experiment

  I’m using all the exact same hardware as outlined in the post, and have gotten everything installed correctly as far as I can tell.
  But when I go to run the ffmpeg command, it seems like nothing actually gets sent over to twitch correctly.

  The log after trying to run it looks like this.
  If anybody has any insight as to what may be going wrong, it would be greatly appreciated.

  Starting ffmpeg
  
  ffmpeg version N-95970-gd5274f8 Copyright (c) 2000-2019 the FFmpeg developers
  
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)  
  
  configuration: --prefix=/home/pi/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/home/pi/ffmpeg_build/include --extra-ldflags=-L/home/pi/ffmpeg_build/lib --extra-libs='-lpthread -lm' --bindir=/home/pi/bin --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-nonfree  
  
  libavutil 56. 36.101 / 56. 36.101  
  
  libavcodec 58. 64.101 / 58. 64.101  
  
  ffmpeg version N-95970-gd5274f8 libavformat 58. 35.101 / 58. 35.101  
  
  Copyright (c) 2000-2019 the FFmpeg developers libavdevice 58. 9.101 / 58. 9.101  
  
  libavfilter 7. 67.100 / 7. 67.100  
  
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)  
  
  libswscale 5. 6.100 / 5. 6.100  
  
  configuration: --prefix=/home/pi/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/home/pi/ffmpeg_build/include --extra-ldflags=-L/home/pi/ffmpeg_build/lib --extra-libs='-lpthread -lm' --bindir=/home/pi/bin --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-nonfree  
  
  libswresample 3. 6.100 / 3. 6.100  
  
  libpostproc 55. 6.100 / 55. 6.100  
  
  libavutil 56. 36.101 / 56. 36.101  
  
  libavcodec 58. 64.101 / 58. 64.101  
  
  libavformat 58. 35.101 / 58. 35.101  
  
  libavdevice 58. 9.101 / 58. 9.101  
  
  libavfilter 7. 67.100 / 7. 67.100  
  
  libswscale 5. 6.100 / 5. 6.100  
  
  libswresample 3. 6.100 / 3. 6.100  
  
  libpostproc 55. 6.100 / 55. 6.100  
  
  [video4linux2,v4l2 @ 0x2aac5e0] The V4L2 driver changed the video from 1280x720 to 1920x1080  
  
  [video4linux2,v4l2 @ 0x2aac5e0] The driver changed the time per frame from 1/24 to 117/7013  
  
  [video4linux2,v4l2 @ 0x2aac5e0] Dequeued v4l2 buffer contains 4147200 bytes, but 3110400 were expected. Flags: 0x00012001.  
  
  Input #0, video4linux2,v4l2, from '/dev/video0':  
  
  Duration: N/A, start: 4683.201589, bitrate: 1491503 kb/s  
  
  Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1920x1080, 1491503 kb/s, 59.94 fps, 59.94 tbr, 1000k tbn, 1000k tbc  
  
  Guessed Channel Layout for Input Stream #1.0 : stereo  
  
  Input #1, alsa, from 'hw:2,0':  
  
  Duration: N/A, start: 1576099663.557438, bitrate: 1536 kb/s  
  
  Stream #1:0: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s  
  
  Please use -b:a or -b:v, -b is ambiguous  
  
  Stream mapping:  
  
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))  
  
  Stream #1:0 -> #0:1 (pcm_s16le (native) -> mp3 (libmp3lame))  
  
  [video4linux2,v4l2 @ 0x2aac5e0] Dequeued v4l2 buffer contains 4147200 bytes, but 3110400 were expected. Flags: 0x00012001.  
  
  Last message repeated 9 times
  
  [video4linux2,v4l2 @ 0x2aac5e0] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)  
  
  [video4linux2,v4l2 @ 0x2aac5e0] Dequeued v4l2 buffer contains 4147200 bytes, but 3110400 were expected. Flags: 0x00012001.  
  
  Last message repeated 28 times  
  
  terminated script  
  
  pipe:: could not find codec parameters  
  
  Exiting normally, received signal 15.  
  
  Last message repeated 15 times  
  
  [alsa @ 0x2aaf2c0] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)  
  
  Finishing stream 0:0 without any data written to it.  
  
  [libx264 @ 0x2abee40] using cpu capabilities: ARMv6 NEON  
  
  [libx264 @ 0x2abee40] profile Constrained Baseline, level 3.2, 4:2:0, 8-bit  
  
  [libx264 @ 0x2abee40] 264 - core 158 - H.264/MPEG-4 AVC codec - Copyleft 2003-2019 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc_lookahead=0 rc=crf mbtree=0 crf=17.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 vbv_maxrate=1000 vbv_bufsize=1000 crf_max=0.0 nal_hrd=none filler=0 ip_ratio=1.40 aq=0  
  
  Finishing stream 0:1 without any data written to it.  
  
  Output #0, flv, to 'pipe:':  
  
  Metadata:  
  
  encoder : Lavf58.35.101  
  
  Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuv420p, 1280x720, q=-1--1, 96 kb/s, 59.94 fps, 1k tbn, 59.94 tbc  
  
  Metadata:  
  
  encoder : Lavc58.64.101 libx264  
  
  Side data:  
  
  cpb: bitrate max/min/avg: 1000000/0/96000 buffer size: 1000000 vbv_delay: N/A  
  
  Stream #0:1: Audio: mp3 (libmp3lame) ([2][0][0][0] / 0x0002), 44100 Hz, stereo, s16p  
  
  Metadata:  
  
  encoder : Lavc58.64.101 libmp3lame  
  
  [flv @ 0x2abda90] Failed to update header with correct duration.  
  
  [flv @ 0x2abda90] Failed to update header with correct filesize.  
  
  Error writing trailer of pipe:: Broken pipe  
  
  frame= 0 fps=0.0 q=0.0 Lsize= 0kB time=00:00:00.00 bitrate=N/A speed= 0x
  
  video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown  
  
  Exiting normally, received signal 15.         

  This message repeats until script is terminated with the Circuit Express button. For length, many instances of this line were cut out.

  [video4linux2,v4l2 @ 0x2aac5e0] Dequeued v4l2 buffer contains 4147200
  
  bytes, but 3110400 were expected. Flags: 0x00012001.
  
  Last message repeated xx times

  Output from v4l2-ctl --list-formats-ext

  ioctl: VIDIOC_ENUM_FMT
  
      Type: Video Capture
  
  
  
      [0]: 'YUYV' (YUYV 4:2:2)
  
          Size: Discrete 1920x1080
  
              Interval: Discrete 0.017s (59.940 fps)
  
      [1]: 'NV12' (Y/CbCr 4:2:0)
  
          Size: Discrete 1920x1080
  
              Interval: Discrete 0.017s (59.940 fps)
  
      [2]: 'YU12' (Planar YUV 4:2:0)
  
          Size: Discrete 1920x1080
  
              Interval: Discrete 0.017s (59.940 fps)

  Log output after ffmpeg command modification.

  Starting ffmpeg
  
  ffmpeg version N-95970-gd5274f8 Copyright (c) 2000-2019 the FFmpeg developers
  
    built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  
    configuration: --prefix=/home/pi/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/home/pi/ffmpeg_build/include --extra-ldflags=-L/home/pi/ffmpeg_build/lib --extra-libs='-lpthread -lm' --bindir=/home/pi/bin --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-nonfree
  
    libavutil      56. 36.101 / 56. 36.101
  
    libavcodec     58. 64.101 / 58. 64.101
  
    libavformat    58. 35.101 / 58. 35.101
  
    libavdevice    58.  9.101 / 58.  9.101
  
    libavfilter     7. 67.100 /  7. 67.100
  
    libswscale      5.  6.100 /  5.  6.100
  
    libswresample   3.  6.100 /  3.  6.100
  
    libpostproc    55.  6.100 / 55.  6.100
  
  terminated script
  
  Input #0, video4linux2,v4l2, from '/dev/video0':
  
    Duration: N/A, bitrate: 1491503 kb/s
  
      Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1920x1080, 1491503 kb/s, 59.94 fps, 59.94 tbr, 1000k tbn, 1000k tbc
  
  Guessed Channel Layout for Input Stream #1.0 : stereo
  
  Input #1, alsa, from 'hw:1,0':
  
    Duration: N/A, bitrate: 1536 kb/s
  
      Stream #1:0: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s
  
  [rtmp @ 0x2605cd0] Cannot open connection tcp://live.twitch.tv:1935
  
  rtmp://live.twitch.tv/app/live: Immediate exit requested
  
  Exiting normally, received signal 15.