Recherche avancée

Sur d’autres sites (7868)

• Why does ffmpeg keep using more and more RAM and crash ?

  29 décembre 2022, par József Márton Kakas

  I am using the following command with ffmpeg to encode a video file using the libsvtav1 codec : ffmpeg -i hevc.mkv -map 0:v:0 -c:v:0 libsvtav1 -preset 8 -crf 22 -format matroska av1.mkv.

  


  However, ffmpeg is using up all of my available RAM and crashing. I have also tried using the libx264 codec, but the same issue occurs, although it happens more slowly. I have already allocated 12 GB of RAM to ffmpeg, but it still seems to be insufficient. How can I prevent ffmpeg from using all of my available RAM and crashing when using either the libsvtav1 or libx264 codecs ?

  


  I have tried it on another VM, but the same issue occurs. Here is the full output of the program.

  


  ffmpeg -i akira.mkv -map 0:v:0 -c:v:0 libsvtav1 -preset 8 -crf 22 -c:a copy -c:s copy -format matroska av1.mkv
ffmpeg version n5.1.2-7-ga6e26053c2-20221106 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 12.2.0 (crosstool-NG 1.25.0.90_cf9beb1)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --disable-chromaprint --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --disable-openal --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20221106
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, matroska,webm, from 'akira.mkv':
  Metadata:
    title           : Akira 4K
    encoder         : libebml v1.3.10 + libmatroska v1.5.2
    creation_time   : 2020-05-04T19:57:18.000000Z
  Duration: 02:04:46.50, start: 0.000000, bitrate: 10945 kb/s
  Stream #0:0: Video: hevc (Main 10), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x2074 [SAR 1:1 DAR 1920:1037], 23.98 fps, 23.98 tbr, 1k tbn (default)
    Metadata:
      BPS-eng         : 9531297
      DURATION-eng    : 02:04:46.479000000
      NUMBER_OF_FRAMES-eng: 179496
      NUMBER_OF_BYTES-eng: 8919482644
      _STATISTICS_WRITING_APP-eng: mkvmerge v43.0.0 ('The Quartermaster') 32-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-05-04 19:57:18
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:1(eng): Subtitle: subrip
    Metadata:
      BPS-eng         : 46
      DURATION-eng    : 01:59:00.755000000
      NUMBER_OF_FRAMES-eng: 1277
      NUMBER_OF_BYTES-eng: 41675
      _STATISTICS_WRITING_APP-eng: mkvmerge v43.0.0 ('The Quartermaster') 32-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-05-04 19:57:18
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:2(ita): Audio: ac3, 48000 Hz, 5.1(side), fltp, 448 kb/s (default)
    Metadata:
      title           : Nuovo doppiaggio
      BPS-eng         : 448000
      DURATION-eng    : 02:04:28.480000000
      NUMBER_OF_FRAMES-eng: 233390
      NUMBER_OF_BYTES-eng: 418234880
      _STATISTICS_WRITING_APP-eng: mkvmerge v43.0.0 ('The Quartermaster') 32-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-05-04 19:57:18
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:3(ita): Audio: ac3, 48000 Hz, stereo, fltp, 448 kb/s
    Metadata:
      title           : Doppiaggio Storico
      BPS-eng         : 448000
      DURATION-eng    : 02:04:28.480000000
      NUMBER_OF_FRAMES-eng: 233390
      NUMBER_OF_BYTES-eng: 418234880
      _STATISTICS_WRITING_APP-eng: mkvmerge v43.0.0 ('The Quartermaster') 32-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-05-04 19:57:18
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:4(ita): Subtitle: dvd_subtitle, 1920x1080
    Metadata:
      title           : Italiano
      BPS-eng         : 8426
      DURATION-eng    : 02:00:14.955000000
      NUMBER_OF_FRAMES-eng: 1201
      NUMBER_OF_BYTES-eng: 7599491
      _STATISTICS_WRITING_APP-eng: mkvmerge v43.0.0 ('The Quartermaster') 32-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-05-04 19:57:18
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:5(ita): Subtitle: subrip (default)
    Metadata:
      title           : Forced
      BPS-eng         : 0
      DURATION-eng    : 01:46:31.753000000
      NUMBER_OF_FRAMES-eng: 14
      NUMBER_OF_BYTES-eng: 462
      _STATISTICS_WRITING_APP-eng: mkvmerge v43.0.0 ('The Quartermaster') 32-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-05-04 19:57:18
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:6(jpn): Audio: eac3, 48000 Hz, 5.1(side), fltp, 256 kb/s
    Metadata:
      BPS-eng         : 256000
      DURATION-eng    : 02:04:46.496000000
      NUMBER_OF_FRAMES-eng: 233953
      NUMBER_OF_BYTES-eng: 239567872
      _STATISTICS_WRITING_APP-eng: mkvmerge v43.0.0 ('The Quartermaster') 32-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-05-04 19:57:18
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:7(eng): Audio: eac3, 48000 Hz, 5.1(side), fltp, 256 kb/s
    Metadata:
      BPS-eng         : 256000
      DURATION-eng    : 02:04:46.496000000
      NUMBER_OF_FRAMES-eng: 233953
      NUMBER_OF_BYTES-eng: 239567872
      _STATISTICS_WRITING_APP-eng: mkvmerge v43.0.0 ('The Quartermaster') 32-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-05-04 19:57:18
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:8: Video: mjpeg (Baseline), yuvj444p(pc, bt470bg/unknown/unknown), 1067x600, 90k tbr, 90k tbn (attached pic)
    Metadata:
      filename        : cover_land.jpg
      mimetype        : image/jpeg
  Stream #0:9: Video: mjpeg (Baseline), yuvj444p(pc, bt470bg/unknown/unknown), 120x176, 90k tbr, 90k tbn (attached pic)
    Metadata:
      filename        : small_cover.jpg
      mimetype        : image/jpeg
  Stream #0:10: Video: mjpeg (Baseline), yuvj444p(pc, bt470bg/unknown/unknown), 213x120, 90k tbr, 90k tbn (attached pic)
    Metadata:
      filename        : small_cover_land.jpg
      mimetype        : image/jpeg
  Stream #0:11: Video: mjpeg (Baseline), yuvj444p(pc, bt470bg/unknown/unknown), 600x882, 90k tbr, 90k tbn (attached pic)
    Metadata:
      filename        : cover.jpg
      mimetype        : image/jpeg
  Stream #0:12: Attachment: otf
    Metadata:
      filename        : StoneSansStd-Semibold.otf
      mimetype        : application/vnd.ms-opentype
  Stream #0:13: Attachment: otf
    Metadata:
      filename        : StoneSansStd-SemiboldItalic.otf
      mimetype        : application/vnd.ms-opentype
  Stream #0:14: Attachment: otf
    Metadata:
      filename        : ATPacella-Black.otf
      mimetype        : application/vnd.ms-opentype
  Stream #0:15: Attachment: ttf
    Metadata:
      filename        : ITC Franklin Gothic LT Medium Condensed.ttf
      mimetype        : application/x-truetype-font
File 'av1.mkv' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> av1 (libsvtav1))
Press [q] to stop, [?] for help
Svt[info]: -------------------------------------------
Svt[info]: SVT [version]:       SVT-AV1 Encoder Lib v1.3.0
Svt[info]: SVT [build]  :       GCC 12.2.0       64 bit
Svt[info]: LIB Build date: Nov  6 2022 00:41:54
Svt[info]: -------------------------------------------
Svt[info]: Number of logical cores available: 16
Svt[info]: Number of PPCS 71
Svt[info]: [asm level on system : up to avx]
Svt[info]: [asm level selected : up to avx]
Svt[info]: -------------------------------------------
Svt[info]: SVT [config]: main profile   tier (auto)     level (auto)
Svt[info]: SVT [config]: width / height / fps numerator / fps denominator               : 3840 / 2080 / 24000 / 1001
Svt[info]: SVT [config]: bit-depth / color format                      : 10 / YUV420
Svt[info]: SVT [config]: preset / tune / pred struct                   : 8 / PSNR / random access
Svt[info]: SVT [config]: gop size / mini-gop size / key-frame type     : 161 / 16 / key frame
Svt[info]: SVT [config]: BRC mode / rate factor                        : CRF / 22 
Svt[info]: -------------------------------------------
Output #0, matroska, to 'av1.mkv':
  Metadata:
    title           : Akira 4K
    encoder         : Lavf59.27.100
  Stream #0:0: Video: av1 (AV01 / 0x31305641), yuv420p10le(tv, bt2020nc/bt2020/smpte2084, progressive), 3840x2074 [SAR 1:1 DAR 1920:1037], q=2-31, 23.98 fps, 1k tbn (default)
    Metadata:
      BPS-eng         : 9531297
      DURATION-eng    : 02:04:46.479000000
      NUMBER_OF_FRAMES-eng: 179496
      NUMBER_OF_BYTES-eng: 8919482644
      _STATISTICS_WRITING_APP-eng: mkvmerge v43.0.0 ('The Quartermaster') 32-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2020-05-04 19:57:18
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      encoder         : Lavc59.37.100 libsvtav1
frame=    1 fps=0.0 q=0.0 size=       1kB time=00:00:00.00 bitrate=N/A sframe=    2 fps=1.3 q=0.0 size=       1kB time=00:00:00.00 bitrate=N/A sframe=   51 fps= 25 q=0.0 size=       1kB time=00:00:00.00 bitrate=N/A sframe=   73 fps= 16 q=16.0 size=       1kB time=00:00:00.00 bitrate=7480frame=   76 fps= 14 q=16.0 size=       1kB time=00:00:00.00 bitrate=7480frame=   81 fps= 14 q=16.0 size=       1kB time=00:00:00.00 bitrate=7480frame=   87 fps= 14 q=21.0 size=       1kB time=00:00:00.16 bitrate=  44frame=   95 fps= 14 q=21.0 size=       1kB time=00:00:00.50 bitrate=  14frame=  100 fps= 13 q=16.0 size=       1kB time=00:00:00.66 bitrate=  11frame=  110 fps= 13 q=23.0 size=       1kB time=00:00:00.96 bitrate=   7frame=  114 fps= 13 q=23.0 size=       1kB time=00:00:01.12 bitrate=   6frame=  122 fps= 13 q=23.0 size=       1kB time=00:00:01.46 bitrate=   5frame=  126 fps= 12 q=23.0 size=       1kB time=00:00:01.62 bitrate=   4frame=  133 fps= 12 q=22.0 size=       1kB time=00:00:01.92 bitrate=   3frame=  139 fps= 12 q=21.0 size=       1kB time=00:00:02.17 bitrate=   3frame=  141 fps= 11 q=22.0 size=       1kB time=00:00:02.25 bitrate=   3frame=  144 fps= 11 q=23.0 size=       1kB time=00:00:02.37 bitrate=   3frame=  151 fps= 11 q=13.0 size=       1kB time=00:00:02.67 bitrate=   2frame=  159 fps= 11 q=18.0 size=       1kB time=00:00:03.00 bitrate=   2frame=  164 fps= 11 q=22.0 size=       1kB time=00:00:03.21 bitrate=   2frame=  174 fps= 11 q=22.0 size=       1kB time=00:00:03.63 bitrate=   2frame=  179 fps= 11 q=20.0 size=       1kB time=00:00:03.83 bitrate=   1frame=  187 fps= 11 q=20.0 size=       1kB time=00:00:04.17 bitrate=   1frame=  189 fps= 10 q=21.0 size=       1kB time=00:00:04.25 bitrate=   1frame=  192 fps= 10 q=22.0 size=       1kB time=00:00:04.38 bitrate=   1frame=  195 fps=8.0 q=20.0 size=       1kB time=00:00:04.50 bitrate=   1frame=  201 fps=8.0 q=11.0 size=       1kB time=00:00:04.67 bitrate=   1Killeds/s speed=0.187x


  


• Manipulating one video into multi outputs with FFmpeg results in no audio in the last output

  27 octobre 2015, par my name is

  Using FFmpeg, I’m trying to do some filtering operations on one input video to scale it (out1), scale and trim it (out2).
  This is the command I’m using :

  ffmpeg -y \
  
  -i "Robotica_1080.mkv" \
  
  -filter_complex "[0:v]split=2[v1][v2]; \
  
  [v1]scale=640:360,setpts=PTS-STARTPTS[vout1]; \
  
  [v2]trim=10:15,scale=640:360,setpts=PTS-STARTPTS[vout2]; \
  
  [0:a]asplit=2[a1][a2]; \
  
  [a1]anull,asetpts=PTS-STARTPTS[aout1]; \
  
  [a2]atrim=10:15,asetpts=PTS-STARTPTS[aout2]; \
  
  [vout1][aout1]concat=n=1:v=1:a=1[out1]; \
  
  [vout2][aout2]concat=n=1:v=1:a=1[out2]" \
  
  -map "[out1]" "1.mp4" \
  
  -map "[out2]" "2.mp4" 

  1.mp4 is ok while 2.mp4 lasts 5 seconds as expected but without audio at all (the QuickTime inspector doesn’t write the audio codec)

  I tried to remove the trim/atrim filters, so the filter_complex parameter looked like this :

  -filter_complex "[0:v]split=2[v1][v2]; \
  
  [v1]scale=640:360[vout1]; \
  
  [v2]scale=640:360[vout2]; \
  
  [0:a]asplit=2[a1][a2]; \
  
  [a1]anull[aout1]; \
  
  [a2]anull[aout2]; \
  
  [vout1][aout1]concat=n=1:v=1:a=1[out1]; \
  
  [vout2][aout2]concat=n=1:v=1:a=1[out2]" \

  but still no audio on 2.mp4

  Can anyone give me a hint ?

  —EDIT—
  This is the output from first ffmpeg command :

  ---

  Mac-mini:~ Luca$ /Applications/XAMPP/xamppfiles/htdocs/MediaGallery/ffmpeg/ffmpeg -y \
  
  > -i "/Users/Luca/Desktop/_TEMP UPLOAD/Video/Robotica_1080.mkv" \
  
  > -filter_complex "[0:v]split=2[v1][v2]; \
  
  > [v1]scale=640:360,setpts=PTS-STARTPTS[vout1]; \
  
  > [v2]trim=10:15,scale=640:360,setpts=PTS-STARTPTS[vout2]; \
  
  > [0:a]asplit=2[a1][a2]; \
  
  > [a1]anull,asetpts=PTS-STARTPTS[aout1]; \
  
  > [a2]atrim=10:15,asetpts=PTS-STARTPTS[aout2]; \
  
  > [vout1][aout1]concat=n=1:v=1:a=1[out1]; \
  
  > [vout2][aout2]concat=n=1:v=1:a=1[out2]" \
  
  > -map "[out1]" "/Users/Luca/Downloads/1.mp4" \
  
  > -map "[out2]" "/Users/Luca/Downloads/2.mp4" 
  
  ffmpeg version N-72460-gc5a07f1-tessus Copyright (c) 2000-2015 the FFmpeg developers
  
    built with Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
  
    configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --as=yasm --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libass --enable-libbluray --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzmq --enable-version3 --disable-ffplay --disable-indev=qtkit --disable-indev=x11grab_xcb
  
    libavutil      54. 23.101 / 54. 23.101
  
    libavcodec     56. 40.100 / 56. 40.100
  
    libavformat    56. 33.101 / 56. 33.101
  
    libavdevice    56.  4.100 / 56.  4.100
  
    libavfilter     5. 16.101 /  5. 16.101
  
    libswscale      3.  1.101 /  3.  1.101
  
    libswresample   1.  1.100 /  1.  1.100
  
    libpostproc    53.  3.100 / 53.  3.100
  
  Input #0, matroska,webm, from '/Users/Luca/Desktop/_TEMP UPLOAD/Video/Robotica_1080.mkv':
  
    Metadata:
  
      encoder         : libDivXMediaFormat 4.0.0.0578
  
    Duration: 00:00:20.04, start: 0.000000, bitrate: 4282 kb/s
  
      Stream #0:0(eng): Video: hevc (Main), yuvj420p(pc), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 1k tbn, 25 tbc (default)
  
      Stream #0:1(en): Audio: aac (LC), 44100 Hz, stereo, fltp (default)
  
  [swscaler @ 0x7fb4d181c400] deprecated pixel format used, make sure you did set range correctly
  
  [swscaler @ 0x7fb4d185be00] deprecated pixel format used, make sure you did set range correctly
  
  No pixel format specified, yuvj420p for H.264 encoding chosen.
  
  Use -pix_fmt yuv420p for compatibility with outdated media players.
  
      Last message repeated 1 times
  
  [libx264 @ 0x7fb4d183f400] using SAR=1/1
  
  [libx264 @ 0x7fb4d183f400] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.1 Cache64
  
  [libx264 @ 0x7fb4d183f400] profile High, level 3.0
  
  [libx264 @ 0x7fb4d183f400] 264 - core 142 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=3 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
  
  [libx264 @ 0x7fb4d184e400] using SAR=1/1
  
  [libx264 @ 0x7fb4d184e400] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.1 Cache64
  
  [libx264 @ 0x7fb4d184e400] profile High, level 3.0
  
  [libx264 @ 0x7fb4d184e400] 264 - core 142 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=3 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
  
  Output #0, mp4, to '/Users/Luca/Downloads/1.mp4':
  
    Metadata:
  
      encoder         : Lavf56.33.101
  
      Stream #0:0: Audio: aac (libvo_aacenc) ([64][0][0][0] / 0x0040), 44100 Hz, stereo, s16, 128 kb/s (default)
  
      Metadata:
  
        encoder         : Lavc56.40.100 libvo_aacenc
  
      Stream #0:1: Audio: aac (libvo_aacenc) ([64][0][0][0] / 0x0040), 44100 Hz, stereo, s16, 128 kb/s
  
      Metadata:
  
        encoder         : Lavc56.40.100 libvo_aacenc
  
      Stream #0:2: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuvj420p(pc), 640x360 [SAR 1:1 DAR 16:9], q=-1--1, 25 fps, 12800 tbn, 25 tbc (default)
  
      Metadata:
  
        encoder         : Lavc56.40.100 libx264
  
  Output #1, mp4, to '/Users/Luca/Downloads/2.mp4':
  
    Metadata:
  
      encoder         : Lavf56.33.101
  
      Stream #1:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuvj420p(pc), 640x360 [SAR 1:1 DAR 16:9], q=-1--1, 25 fps, 12800 tbn, 25 tbc (default)
  
      Metadata:
  
        encoder         : Lavc56.40.100 libx264
  
  Stream mapping:
  
    Stream #0:0 (hevc) -> split
  
    Stream #0:1 (aac) -> asplit
  
    concat:out:a0 -> Stream #0:0 (libvo_aacenc)
  
    concat:out:a0 -> Stream #0:1 (libvo_aacenc)
  
    concat:out:v0 -> Stream #0:2 (libx264)
  
    concat:out:v0 -> Stream #1:0 (libx264)
  
  Press [q] to stop, [?] for help
  
  frame=  501 fps= 26 q=-1.0 Lq=-1.0 size=    1512kB time=00:00:19.97 bitrate= 620.1kbits/s    
  
  video:1385kB audio:392kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
  
  [libx264 @ 0x7fb4d183f400] frame I:3     Avg QP:19.30  size:  3879
  
  [libx264 @ 0x7fb4d183f400] frame P:321   Avg QP:24.53  size:  3024
  
  [libx264 @ 0x7fb4d183f400] frame B:177   Avg QP:26.20  size:   825
  
  [libx264 @ 0x7fb4d183f400] consecutive B-frames: 40.7% 34.7%  5.4% 19.2%
  
  [libx264 @ 0x7fb4d183f400] mb I  I16..4: 24.8% 65.9%  9.3%
  
  [libx264 @ 0x7fb4d183f400] mb P  I16..4:  7.5% 10.5%  2.5%  P16..4: 22.9%  7.4%  2.7%  0.0%  0.0%    skip:46.5%
  
  [libx264 @ 0x7fb4d183f400] mb B  I16..4:  0.4%  0.6%  0.3%  B16..8: 17.2%  2.9%  0.7%  direct: 0.9%  skip:77.1%  L0:37.2% L1:51.8% BI:11.0%
  
  [libx264 @ 0x7fb4d183f400] 8x8 transform intra:51.6% inter:69.2%
  
  [libx264 @ 0x7fb4d183f400] coded y,uvDC,uvAC intra: 40.2% 24.2% 1.1% inter: 8.5% 4.0% 0.0%
  
  [libx264 @ 0x7fb4d183f400] i16 v,h,dc,p:  9% 65%  1% 25%
  
  [libx264 @ 0x7fb4d183f400] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 29% 16%  4%  6%  6%  7%  6%  6%
  
  [libx264 @ 0x7fb4d183f400] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 22% 34% 12%  3%  7%  6%  7%  4%  4%
  
  [libx264 @ 0x7fb4d183f400] i8c dc,h,v,p: 63% 25% 11%  1%
  
  [libx264 @ 0x7fb4d183f400] Weighted P-Frames: Y:0.0% UV:0.0%
  
  [libx264 @ 0x7fb4d183f400] ref P L0: 69.3% 11.5% 12.5%  6.7%
  
  [libx264 @ 0x7fb4d183f400] ref B L0: 82.4% 16.7%  0.8%
  
  [libx264 @ 0x7fb4d183f400] ref B L1: 98.6%  1.4%
  
  [libx264 @ 0x7fb4d183f400] kb/s:450.44
  
  [libx264 @ 0x7fb4d184e400] frame I:1     Avg QP:22.23  size:  6699
  
  [libx264 @ 0x7fb4d184e400] frame P:78    Avg QP:24.94  size:  2998
  
  [libx264 @ 0x7fb4d184e400] frame B:46    Avg QP:27.93  size:  1036
  
  [libx264 @ 0x7fb4d184e400] consecutive B-frames: 32.0% 56.0%  2.4%  9.6%
  
  [libx264 @ 0x7fb4d184e400] mb I  I16..4: 53.5% 26.6% 19.9%
  
  [libx264 @ 0x7fb4d184e400] mb P  I16..4:  9.8%  7.6%  3.1%  P16..4: 25.0%  8.0%  2.8%  0.0%  0.0%    skip:43.8%
  
  [libx264 @ 0x7fb4d184e400] mb B  I16..4:  0.8%  0.5%  0.4%  B16..8: 22.4%  3.5%  0.8%  direct: 1.1%  skip:70.4%  L0:41.1% L1:48.0% BI:10.9%
  
  [libx264 @ 0x7fb4d184e400] 8x8 transform intra:36.1% inter:66.1%
  
  [libx264 @ 0x7fb4d184e400] coded y,uvDC,uvAC intra: 33.5% 24.6% 1.8% inter: 8.5% 3.8% 0.0%
  
  [libx264 @ 0x7fb4d184e400] i16 v,h,dc,p:  4% 82%  1% 13%
  
  [libx264 @ 0x7fb4d184e400] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17% 40% 18%  3%  4%  4%  7%  3%  5%
  
  [libx264 @ 0x7fb4d184e400] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 23% 40% 13%  3%  5%  5%  6%  3%  4%
  
  [libx264 @ 0x7fb4d184e400] i8c dc,h,v,p: 54% 36%  8%  2%
  
  [libx264 @ 0x7fb4d184e400] Weighted P-Frames: Y:0.0% UV:0.0%
  
  [libx264 @ 0x7fb4d184e400] ref P L0: 60.6% 10.5% 17.5% 11.4%
  
  [libx264 @ 0x7fb4d184e400] ref B L0: 77.6% 22.0%  0.4%
  
  [libx264 @ 0x7fb4d184e400] ref B L1: 99.1%  0.9%
  
  [libx264 @ 0x7fb4d184e400] kb/s:461.15

• Overthinking My Search Engine Problem

  31 décembre 2013, par Multimedia Mike — General

  I wrote a search engine for my Game Music Appreciation website, because the site would have been significantly less valuable without it (and I would eventually realize that the search feature is probably the most valuable part of this endeavor). I came up with a search solution that was a bit sketchy, but worked… until it didn’t. I thought of a fix but still searched for more robust and modern solutions (where ‘modern’ is defined as something that doesn’t require compiling a C program into a static CGI script and hoping that it works on a server I can’t debug on).

  Finally, I realized that I was overthinking the problem– did you know that a bunch of relational database management systems (RDBMSs) support full text search (FTS) ? Okay, maybe you did, but I didn’t know this.

  Problem Statement
  My goal is to enable users to search the metadata (title, composer, copyright, other tags) attached to various games. To do this, I want to index a series of contrived documents that describe the metadata. 2 examples of these contrived documents, interesting because both of these games have very different titles depending on region, something the search engine needs to account for :

  system : Nintendo NES
  game : Snoopy’s Silly Sports Spectacular
  author : None ; copyright : 1988 Kemco ; dumped by : None
  additional tags : Donald Duck.nsf Donald Duck
  system : Super Nintendo
  
  game : Arcana
  
  author : Jun Ishikawa, Hirokazu Ando ; copyright : 1992 HAL Laboratory ; dumped by : Datschge
  
  additional tags : card.rsn.gamemusic Card Master Cardmaster
  

  The index needs to map these documents to various pieces of game music and the search solution needs to efficiently search these documents and find the various game music entries that match a user’s request.

  Now that I’ve been looking at it for long enough, I’m able to express the problem surprisingly succinctly. If I had understood that much originally, this probably would have been simpler.

  First Solution & Breakage
  My original solution was based on SWISH-E. The CGI script was a C program that statically linked the SWISH-E library into a binary that miraculously ran on my web provider. At least, it ran until it decided to stop working a month ago when I added a new feature unrelated to search. It was a very bizarre problem, the details of which would probably bore you to tears. But if you care, the details are all there in the Stack Overflow question I asked on the matter.

  While no one could think of a direct answer to the problem, I eventually thought of a roundabout fix. The problem seemed to pertain to the static linking. Since I couldn’t count on the relevant SWISH-E library to be on my host’s system, I uploaded the shared library to the same directory as the CGI script and used dlopen()/dlsym() to fetch the functions I needed. It worked again, but I didn’t know for how long.

  Searching For A Hosted Solution
  I know that anything is possible in this day and age ; while my web host is fairly limited, there are lots of solutions for things like this and you can deploy any technology you want, and for reasonable prices. I figured that there must be a hosted solution out there.

  I have long wanted a compelling reason to really dive into Amazon Web Services (AWS) and this sounded like a good opportunity. After all, my script works well enough ; if I could just find a simple Linux box out there where I could install the SWISH-E library and compile the CGI script, I should be good to go. AWS has a free tier and I started investigating this approach. But it seems like a rabbit hole with a lot of moving pieces necessary for such a simple task.

  I had heard that AWS had something in this area. Sure enough, it’s called CloudSearch. However, I’m somewhat discouraged by the fact that it would cost me around $75 per month to run the smallest type of search instance which is at the core of the service.

  Finally, I came to another platform called Heroku. It’s supposed to be super-scalable while having a free tier for hobbyists. I started investigating FTS on Heroku and found this article which recommends using the FTS capabilities of their standard hosted PostgreSQL solution. However, the free tier of Postgres hosting only allows for 10,000 rows of data. Right now, my database has about 5400 rows. I expect it to easily overflow the 10,000 limit as soon as I incorporate the C64 SID music corpus.

  However, this Postgres approach planted a seed.

  RDBMS Revelation
  I have 2 RDBMSs available on my hosting plan– MySQL and SQLite (the former is a separate service while SQLite is built into PHP). I quickly learned that both have FTS capabilities. Since I like using SQLite so much, I elected to leverage its FTS functionality. And it’s just this simple :

  CREATE VIRTUAL TABLE gamemusic_metadata_fts USING fts3
  ( content TEXT, game_id INT, title TEXT ) ;
  SELECT id, title FROM gamemusic_metadata_fts WHERE content MATCH "arcana" ;
  
  479|Arcana
  

  The ‘content’ column gets the metadata pseudo-documents. The SQL gets wrapped up in a little PHP so that it queries this small database and turns the result into JSON. The script is then ready as a drop-in replacement for the previous script.