Recherche avancée

Sur d’autres sites (3586)

• Overthinking My Search Engine Problem

  31 décembre 2013, par Multimedia Mike — General

  I wrote a search engine for my Game Music Appreciation website, because the site would have been significantly less valuable without it (and I would eventually realize that the search feature is probably the most valuable part of this endeavor). I came up with a search solution that was a bit sketchy, but worked… until it didn’t. I thought of a fix but still searched for more robust and modern solutions (where ‘modern’ is defined as something that doesn’t require compiling a C program into a static CGI script and hoping that it works on a server I can’t debug on).

  Finally, I realized that I was overthinking the problem– did you know that a bunch of relational database management systems (RDBMSs) support full text search (FTS) ? Okay, maybe you did, but I didn’t know this.

  Problem Statement
  My goal is to enable users to search the metadata (title, composer, copyright, other tags) attached to various games. To do this, I want to index a series of contrived documents that describe the metadata. 2 examples of these contrived documents, interesting because both of these games have very different titles depending on region, something the search engine needs to account for :

  system : Nintendo NES
  game : Snoopy’s Silly Sports Spectacular
  author : None ; copyright : 1988 Kemco ; dumped by : None
  additional tags : Donald Duck.nsf Donald Duck
  system : Super Nintendo
  
  game : Arcana
  
  author : Jun Ishikawa, Hirokazu Ando ; copyright : 1992 HAL Laboratory ; dumped by : Datschge
  
  additional tags : card.rsn.gamemusic Card Master Cardmaster
  

  The index needs to map these documents to various pieces of game music and the search solution needs to efficiently search these documents and find the various game music entries that match a user’s request.

  Now that I’ve been looking at it for long enough, I’m able to express the problem surprisingly succinctly. If I had understood that much originally, this probably would have been simpler.

  First Solution & Breakage
  My original solution was based on SWISH-E. The CGI script was a C program that statically linked the SWISH-E library into a binary that miraculously ran on my web provider. At least, it ran until it decided to stop working a month ago when I added a new feature unrelated to search. It was a very bizarre problem, the details of which would probably bore you to tears. But if you care, the details are all there in the Stack Overflow question I asked on the matter.

  While no one could think of a direct answer to the problem, I eventually thought of a roundabout fix. The problem seemed to pertain to the static linking. Since I couldn’t count on the relevant SWISH-E library to be on my host’s system, I uploaded the shared library to the same directory as the CGI script and used dlopen()/dlsym() to fetch the functions I needed. It worked again, but I didn’t know for how long.

  Searching For A Hosted Solution
  I know that anything is possible in this day and age ; while my web host is fairly limited, there are lots of solutions for things like this and you can deploy any technology you want, and for reasonable prices. I figured that there must be a hosted solution out there.

  I have long wanted a compelling reason to really dive into Amazon Web Services (AWS) and this sounded like a good opportunity. After all, my script works well enough ; if I could just find a simple Linux box out there where I could install the SWISH-E library and compile the CGI script, I should be good to go. AWS has a free tier and I started investigating this approach. But it seems like a rabbit hole with a lot of moving pieces necessary for such a simple task.

  I had heard that AWS had something in this area. Sure enough, it’s called CloudSearch. However, I’m somewhat discouraged by the fact that it would cost me around $75 per month to run the smallest type of search instance which is at the core of the service.

  Finally, I came to another platform called Heroku. It’s supposed to be super-scalable while having a free tier for hobbyists. I started investigating FTS on Heroku and found this article which recommends using the FTS capabilities of their standard hosted PostgreSQL solution. However, the free tier of Postgres hosting only allows for 10,000 rows of data. Right now, my database has about 5400 rows. I expect it to easily overflow the 10,000 limit as soon as I incorporate the C64 SID music corpus.

  However, this Postgres approach planted a seed.

  RDBMS Revelation
  I have 2 RDBMSs available on my hosting plan– MySQL and SQLite (the former is a separate service while SQLite is built into PHP). I quickly learned that both have FTS capabilities. Since I like using SQLite so much, I elected to leverage its FTS functionality. And it’s just this simple :

  CREATE VIRTUAL TABLE gamemusic_metadata_fts USING fts3
  ( content TEXT, game_id INT, title TEXT ) ;
  SELECT id, title FROM gamemusic_metadata_fts WHERE content MATCH "arcana" ;
  
  479|Arcana
  

  The ‘content’ column gets the metadata pseudo-documents. The SQL gets wrapped up in a little PHP so that it queries this small database and turns the result into JSON. The script is then ready as a drop-in replacement for the previous script.

• "ffmpeg has text relocations" error in Android

  5 janvier 2015, par djzmo

  I’m building an Android video-sending application. I need the video to be compressed before being uploaded. In this case, I chose to use this Android wrapper of FFmpeg : https://github.com/hiteshsondhi88/ffmpeg-android-java

  FFmpeg error: WARNING: linker: /data/data/com.mycompany.myapplication/files/ffmpeg has text relocations. This is wasting memory and prevents security hardening. Please fix.
  
      ffmpeg version n2.4.2 Copyright (c) 2000-2014 the FFmpeg developers
  
      built on Oct  7 2014 15:08:46 with gcc 4.8 (GCC)
  
      configuration: --target-os=linux --cross-prefix=/home/sb/Source-Code/ffmpeg-android/toolchain-android/bin/arm-linux-androideabi- --arch=arm --cpu=cortex-a8 --enable-runtime-cpudetect --sysroot=/home/sb/Source-Code/ffmpeg-android/toolchain-android/sysroot --enable-pic --enable-libx264 --enable-libass --enable-libfreetype --enable-libfribidi --enable-fontconfig --enable-pthreads --disable-debug --disable-ffserver --enable-version3 --enable-hardcoded-tables --disable-ffplay --disable-ffprobe --enable-gpl --enable-yasm --disable-doc --disable-shared --enable-static --pkg-config=/home/sb/Source-Code/ffmpeg-android/ffmpeg-pkg-config --prefix=/home/sb/Source-Code/ffmpeg-android/build/armeabi-v7a-neon --extra-cflags='-I/home/sb/Source-Code/ffmpeg-android/toolchain-android/include -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fno-strict-overflow -fstack-protector-all -mfpu=neon' --extra-ldflags='-L/home/sb/Source-Code/ffmpeg-android/toolchain-android/lib -Wl,-z,relro -Wl,-z,now -pie' --extra-libs='-lpng -lexpat -lm' --extra-cxxflags=
  
      libavutil      54.  7.100 / 54.  7.100
  
      libavcodec     56.  1.100 / 56.  1.100
  
      libavformat    56.  4.101 / 56.  4.101
  
      libavdevice    56.  0.100 / 56.  0.100
  
      libavfilter     5.  1.100 /  5.  1.100
  
      libswscale      3.  0.100 /  3.  0.100
  
      libswresample   1.  1.100 /  1.  1.100
  
      libpostproc    53.  0.100 / 53.  0.100
  
      Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/storage/emulated/0/DCIM/Camera/VID_20150104_235926.mp4':
  
      Metadata:
  
      major_brand     : mp42
  
      minor_version   : 0
  
      compatible_brands: isommp42
  
      creation_time   : 2015-01-04 16:00:40
  
      Duration: 00:01:12.60, start: 0.000000, bitrate: 19590 kb/s
  
      Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 19364 kb/s, SAR 65536:65536 DAR 16:9, 26.73 fps, 26.92 tbr, 90k tbn, 180k tbc (default)
  
      Metadata:
  
      creation_time   : 2015-01-04 16:00:40
  
      handler_name    : VideoHandle
  
      Stream #0:1(eng): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 196 kb/s (default)
  
      Metadata:
  
      creation_time   : 2015-01-04 16:00:40
  
      handler_name    : SoundHandle
  
      [libx264 @ 0x40208800] using SAR=1/1
  
      [libx264 @ 0x40208800] frame MB size (120x68) > level limit (1620)
  
      [libx264 @ 0x40208800] DPB size (1 frames, 8160 mbs) > level limit (0 frames, 8100 mbs)
  
      [libx264 @ 0x40208800] MB rate (219640) > level limit (40500)
  
      [libx264 @ 0x40208800] using cpu capabilities: none!
  
      [libx264 @ 0x40208800] profile Constrained Baseline, level 3.0
  
      [libx264 @ 0x40208800] 264 - core 142 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc_lookahead=0 rc=crf mbtree=0 crf=28.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 vbv_maxrate=2500 vbv_bufsize=1835 crf_max=0.0 nal_hrd=none filler=0 ip_ratio=1.40 aq=0
  
      Output #0, mp4, to '/storage/emulated/0/Android/data/com.mycompany.myapplication/cache/VID_20150104_235926.mp4':
  
      Metadata:
  
      major_brand     : mp42
  
      minor_version   : 0
  
      compatible_brands: isommp42
  
      encoder         : Lavf56.4.101
  
      Stream #0:0(eng): Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 1920x1080 [SAR 65536:65536 DAR 16:9], q=-1--1, 2500 kb/s, 26.92 fps, 10336 tbn, 26.92 tbc (default)
  
      Metadata:
  
      creation_time   : 2015-01-04 16:00:40
  
      handler_name    : VideoHandle
  
      encoder         : Lavc56.1.100 libx264
  
      Stream #0:1(eng): Audio: aac ([64][0][0][0] / 0x0040), 4

  Everything seemed fine when I compress small files (below 20 seconds), but when I tried to compress larger files, I seldom get the above error in the middle of the compression process, and the process was also terminated immediately. I need the application to be able to compress one minute video to about 20-35 MB.

  Here is the FFmpeg command I used :

  ffmpeg -y -i -codec:v libx264 -profile:v baseline -level 3.0 -b:v 2500k -minrate 2500k -maxrate 2500k -bufsize 1835k -pix_fmt yuv420p -preset ultrafast -crf 28 -codec:a copy

  Any idea on what went wrong ?
  Any kind of help would be very appreciated.

• FFMPEG concat 2 files of different resolution hangs

  6 octobre 2023, par knagode

  I am trying to concat 2 videos of different size and resize it to 426x240 :

  


  ffmpeg -y -i video_1.mp4 -i video_2.mp4 -filter_complex '[0]scale=426:240:force_original_aspect_ratio=decrease,pad=426:240:(ow-iw)/2:(oh-ih)/2,setsar=1[v0];[1]scale=426:240:force_original_aspect_ratio=decrease,pad=426:240:(ow-iw)/2:(oh-ih)/2,setsar=1[v1];[v0][0:a:0][v1][1:a:0]concat=n=2:v=1:a=1[v][a]' -map '[v]' -map '[a]' concatenated_video.mp4


  


  In the output I see :

  


  ffmpeg version 6.0 Copyright (c) 2000-2023 the FFmpeg developers
  built with Apple clang version 14.0.3 (clang-1403.0.22.14.1)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/6.0_1 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'video_1.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf60.3.100
  Duration: 00:00:05.76, start: 0.000000, bitrate: 1582 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 640x360 [SAR 1:1 DAR 16:9], 1473 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
    Metadata:
      handler_name    : ISO Media file produced by Google Inc. Created on: 08/17/2020.
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.3.100 libx264
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 112 kb/s (default)
    Metadata:
      handler_name    : ISO Media file produced by Google Inc. Created on: 08/17/2020.
      vendor_id       : [0][0][0][0]
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'video_2.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf60.3.100
  Duration: 00:00:16.40, start: 0.000000, bitrate: 383 kb/s
  Stream #1:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 426x240 [SAR 640:639 DAR 16:9], 245 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
    Metadata:
      handler_name    : Core Media Video
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.3.100 libx264
  Stream #1:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : Core Media Audio
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 (h264) -> scale:default
  Stream #0:1 (aac) -> concat
  Stream #1:0 (h264) -> scale:default
  Stream #1:1 (aac) -> concat
  concat -> Stream #0:0 (libx264)
  concat -> Stream #0:1 (aac)
Press [q] to stop, [?] for help
[vost#0:0/libx264 @ 0x7fc777006280] Frame rate very high for a muxer not efficiently supporting it.
Please consider specifying a lower framerate, a different muxer or setting vsync/fps_mode to vfr
[libx264 @ 0x7fc777006580] using SAR=1/1
[libx264 @ 0x7fc777006580] MB rate (405000000) > level limit (16711680)
[libx264 @ 0x7fc777006580] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x7fc777006580] profile High, level 6.2, 4:2:0, 8-bit
[libx264 @ 0x7fc777006580] 264 - core 164 r3095 baee400 - H.264/MPEG-4 AVC codec - Copyleft 2003-2022 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=7 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'concatenated_video.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf60.3.100
  Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p(tv, progressive), 426x240 [SAR 1:1 DAR 71:40], q=2-31, 1000k tbn
    Metadata:
      encoder         : Lavc60.3.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
  Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s
    Metadata:
      encoder         : Lavc60.3.100 aac
[vost#0:0/libx264 @ 0x7fc777006280] More than 1000 frames duplicated  1.1kbits/s speed=4.94x


  


  Process hangs and I see that ffmpeg uses 500% of the CPU. Any idea how to fix (deal with) this ?

  


  I can open both videos on my computer and play them.