Recherche avancée

Sur d’autres sites (8191)

• Re-solving My Search Engine Problem

  28 juillet 2012, par Multimedia Mike — General, swish-e

  14 years ago, I created a web database of 8-bit Nintendo Entertainment System games. To make it useful, I developed a very primitive search feature.

  A few months ago, I decided to create a web database of video game music. To make it useful, I knew it would need to have a search feature. I realized I needed to solve the exact same problem again.

  Requirements
  The last time I solved this problem, I came up with an excruciatingly naïve idea. Hey, it worked. I really didn’t want to deploy the same solution again because it felt so silly the first time. Surely there are many better ways to solve it now ? Many different workable software solutions that do all the hard work for me ?

  The first time I attacked this, it was 1998 and hosting resources were scarce. On my primary web host I was able to put static HTML pages, perhaps with server side includes. The web host also offered dynamic scripting capabilities via something called htmlscript (a.k.a. MIVA Script). I had a secondary web host in my ISP which allowed me to host conventional CGI scripts on a Unix host, so that’s where I hosted the search function (Perl CGI script accessing a key/value data store file).

  Nowadays, sky’s the limit. Any type of technology you want to deploy should be tractable. Still, a key requirement was that I didn’t want to pay for additional hosting resources for this silly little side project. That leaves me with options that my current shared web hosting plan allows, which includes such advanced features as PHP, Perl and Python scripts. I can also access MySQL.

  Candidates
  There are a lot of mature software packages out there which can index and search data and be plugged into a website. But a lot of them would be unworkable on my web hosting plan due to language or library package limitations. Further, a lot of them feel like overkill. At the most basic level, all I really want to do is map a series of video game titles to URLs in a website.

  Based on my research, Lucene seems to hold a fair amount of mindshare as an open source indexing and search solution. But I was unsure of my ability to run it on my hosting plan. I think MySQL does some kind of full text search, so I could have probably made a solution around that. Again, it just feels like way more power than I need for this project.

  I used Swish-e once about 3 years ago for a little project. I wasn’t confident of my ability to run that on my server either. It has a Perl API but it requires custom modules.

  My quest for a search solution grew deep enough that I started perusing a textbook on information retrieval techniques in preparation for possibly writing my own solution from scratch. However, in doing so, I figured out how I might subvert an existing solution to do what I want.

  Back to Swish-e
  Again, all I wanted to do was pull data out of a database and map that data to a URL in a website. Reading the Swish-e documentation, I learned that the software supports a mode specifically tailored for this. Rather than asking Swish-e to index a series of document files living on disk, you can specify a script for Swish-e to run and the script will generate what appears to be a set of phantom documents for Swish-e to index.
  

  When I ’add’ a game music file to the game music website, I have a scripts that scrape the metadata (game title, system, song titles, composers, company, copyright, the original file name on disk, even the ripper/dumper who extracted the chiptune in the first place) and store it all in an SQLite database. When it’s time to update the database, another script systematically generates a series of pseudo-documents that spell out the metadata for each game and prefix each document with a path name. Searching for a term in the index returns a lists of paths that contain the search term. Thus, it makes sense for that path to be a site URL.

  But what about a web script which can search this Swish-e index ? That’s when I noticed Swish-e’s C API and came up with a crazy idea : Write the CGI script directly in C. It feels like sheer madness (or at least the height of software insecurity) to write a CGI script directly in C in this day and age. But it works (with the help of cgic for input processing), just as long as I statically link the search script with libswish-e.a (and libz.a). The web host is an x86 machine, after all.

  I’m not proud of what I did here— I’m proud of how little I had to do here. The searching CGI script is all of about 30 lines of C code. The one annoyance I experienced while writing it is that I had to consult the Swish-e source code to learn how to get my search results (the "swishdocpath" key — or any other key — for SwishResultPropertyStr() is not documented). Also, the C program just does the simplest job possible, only querying the term in the index and returning the results in plaintext, in order of relevance, to the client-side JavaScript code which requested them. JavaScript gets the job of sorting and grouping the results for presentation.

  Tuning the Search
  Almost immediately, I noticed that the search engine could not find one of my favorite SNES games, U.N. Squadron. That’s because all of its associated metadata names Area 88, the game’s original title. Thus, I had to modify the metadata database to allow attaching somewhat free-form tags to games in order to compensate. In this case, an alias title would show up in the game’s pseudo-document.

  Roman numerals are still a thorn in my side, just as they were 14 years ago in my original iteration. I dealt with it back then by converting all numbers to Roman numerals during the index and searching processes. I’m not willing to do that for this case and I’m still looking for a good solution.

  Another annoying problem deals with Mega Man, a popular franchise. The proper spelling is 2 words but it’s common for people to mash it into one word, Megaman (see also : Spider-Man, Spiderman, Spider Man). The index doesn’t gracefully deal with that and I have some hacks in place to cope for the time being.

  Positive Results
  I’m pleased with the results so far, and so are the users I have heard from. I know one user expressed amazement that a search for Castlevania turned up Akumajou Densetsu, the Japanese version of Castlevania III : Dracula’s Curse. This didn’t surprise me because I manually added a hint for that mapping. (BTW, if you are a fan of Castlevania III, definitely check out the Akumajou Densetsu soundtrack which has an upgraded version of the same soundtrack using special audio channels.)

  I was a little more surprised when a user announced that searching for ’probotector’ correctly turned up Contra : Hard Corps. I looked into why this was. It turns out that the original chiptune filename was extremely descriptive : "Contra - Hard Corps [Probotector] (1994-08-08)(Konami)". The filenames themselves often carry a bunch of useful metadata which is why it’s important to index those as well.

  And of course, many rippers, dumpers, and taggers have labored for over a decade to lovingly tag these songs with as much composer information as possible, which all gets indexed. The search engine gets a lot of compliments for its ability to find many songs written by favorite composers.

• Non-blocking realtime read from multiple shell subprocesses (Python)

  8 février 2018, par Norman Edance

  I’m building real time multiple videostream monitoring using ffmpeg and subrocess.
  I currently have the following code, inspired by "Async and await with subprocesses" post.

  The problem is that after a certain period of time the output stops printing and the processes go into zombie mode. I guess that this problem is related to the overload of PIPE or deadlock. Help needed.

  """Async and await example using subprocesses
  
  
  
  Note:
  
      Requires Python 3.6.
  
  """
  
  
  
  import os
  
  import sys
  
  import time
  
  import platform
  
  import asyncio
  
  
  
  async def run_command_shell(command):
  
      """Run command in subprocess (shell)
  
  
  
      Note:
  
          This can be used if you wish to execute e.g. "copy"
  
          on Windows, which can only be executed in the shell.
  
      """
  
      # Create subprocess
  
      process = await asyncio.create_subprocess_shell(
  
          command,
  
          stderr=asyncio.subprocess.PIPE)
  
  
  
      # Status
  
      print('Started:', command, '(pid = ' + str(process.pid) + ')')
  
  
  
      # Wait for the subprocess to finish
  
      stdout, stderr = await process.communicate()
  
  
  
      # Progress
  
      if process.returncode == 0:
  
          print('Done:', command, '(pid = ' + str(process.pid) + ')')
  
      else:
  
          print('Failed:', command, '(pid = ' + str(process.pid) + ')')
  
  
  
      # Result
  
      result = stderr.decode().strip()
  
  
  
      # Real time print
  
      print(result)
  
  
  
      # Return stdout
  
      return result
  
  
  
  
  
  def make_chunks(l, n):
  
      """Yield successive n-sized chunks from l.
  
  
  
      Note:
  
          Taken from https://stackoverflow.com/a/312464
  
      """
  
      if sys.version_info.major == 2:
  
          for i in xrange(0, len(l), n):
  
              yield l[i:i + n]
  
      else:
  
          # Assume Python 3
  
          for i in range(0, len(l), n):
  
              yield l[i:i + n]
  
  
  
  
  
  def run_asyncio_commands(tasks, max_concurrent_tasks=0):
  
      """Run tasks asynchronously using asyncio and return results
  
  
  
      If max_concurrent_tasks are set to 0, no limit is applied.
  
  
  
      Note:
  
          By default, Windows uses SelectorEventLoop, which does not support
  
          subprocesses. Therefore ProactorEventLoop is used on Windows.
  
          https://docs.python.org/3/library/asyncio-eventloops.html#windows
  
      """
  
  
  
      all_results = []
  
  
  
      if max_concurrent_tasks == 0:
  
          chunks = [tasks]
  
      else:
  
          chunks = make_chunks(l=tasks, n=max_concurrent_tasks)
  
  
  
      for tasks_in_chunk in chunks:
  
          if platform.system() == 'Windows':
  
              loop = asyncio.ProactorEventLoop()
  
              asyncio.set_event_loop(loop)
  
          else:
  
              loop = asyncio.get_event_loop()
  
  
  
          commands = asyncio.gather(*tasks_in_chunk)  # Unpack list using *
  
          results = loop.run_until_complete(commands)
  
          all_results += results
  
          loop.close()
  
      return all_results
  
  
  
  
  
  if __name__ == '__main__':
  
  
  
      start = time.time()
  
  
  
      if platform.system() == 'Windows':
  
          # Commands to be executed on Windows
  
          commands = [
  
              ['hostname']
  
          ]
  
      else:
  
          # Commands to be executed on Unix
  
          commands = [
  
              ['du', '-sh', '/var/tmp'],
  
              ['hostname'],
  
          ]
  
      cmds = [["ffmpeg -y -i udp://xxx.xx.xx.xxx:xxxx  -f null -"],
  
              ["ffmpeg -y -i udp://xxx.xx.xx.xxx:xxxx  -f null -"],
  
              ["ffmpeg -y -i udp://xxx.xx.xx.xxx:xxxx -f null -"],
  
              ["ffmpeg -y -i udp://xxx.xx.xx.xxx:xxxx  -f null -"],
  
              ["ffmpeg -y -i udp://xxx.xx.xx.xxx:xxxx -f null -"],
  
              ["ffmpeg -y -i udp://xxx.xx.xx.xxx:xxxx  -f null -"],
  
              ["ffmpeg -y -i udp://xxx.xx.xx.xxx:xxxx  -f null -"],
  
              ["ffmpeg -y -i udp://xxx.xx.xx.xxx:xxxx  -f null -"],
  
              ["ffmpeg -y -i udp://xxx.xx.xx.xxx:xxxx -f null -"],
  
              ["ffmpeg -y -i udp://xxx.xx.xx.xxx:xxxx -f null -"],
  
              ["ffmpeg -y -i udp://xxx.xx.xx.xxx:xxxx -f null -"],
  
              ["ffmpeg -y -i udp://xxx.xx.xx.xxx:xxxx -f null -"]]
  
  
  
      tasks = []
  
      for command in cmds:
  
          tasks.append(run_command_shell(*command))
  
  
  
  
  
      # # Shell execution example
  
      # tasks = [run_command_shell('copy c:/somefile d:/new_file')]
  
  
  
      # # List comprehension example
  
      # tasks = [
  
      #     run_command(*command, get_project_path(project))
  
      #     for project in accessible_projects(all_projects)
  
      # ]
  
  
  
      results = run_asyncio_commands(tasks, max_concurrent_tasks=20)  # At most 20 parallel tasks
  
      print('Results:', results)
  
  
  
      end = time.time()
  
      rounded_end = ('{0:.4f}'.format(round(end-start,4)))
  
      print('Script ran in about', str(rounded_end), 'seconds')

  Related : Non-blocking read from multiple subprocesses (Python)

• FFMPEG stream to youtube crashing

  29 septembre 2022, par jakeb

  We have a script that runs FFMPEG and broadcasts a tab recording from puppeteer, it will run fine for a period of time, but if the chromium tab from puppeteer ends up doing certain things (playing a youtube video, for example) FFMPEG crashes. It doesn't matter if the video is 4k 60 FPS, or if it's a 480i video from 2009, the result seems to be the same.

  


  This is the FFMPEG command we're running :

  


  ffmpeg -loglevel info -i - -c:v libx264 -b:v 15000k -preset ultrafast -tune animation -force_key_frames "expr:gte(t,n_forced*2)" -sc_threshold 0 -c:a aac -minrate 15000k -s 1920x1080 -r 60 -vsync 1 -max_muxing_queue_size 1024 -max_interleave_delta 0 -f flv -flvflags no_duration_filesize "rtmps://a.rtmps.youtube.com/live2/*****"


  


  This is the output when it crashes :

  


  frame= 6554 fps= 57 q=0.0 size=   17033kB time=00:02:00.02 bitrate=1162. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 6564 fps= 56 q=1.0 size=   17189kB time=00:02:01.02 bitrate=1163. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 6566 fps= 56 q=0.0 size=   17224kB time=00:02:02.05 bitrate=1156. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 6570 fps= 56 q=1.0 size=   17276kB time=00:02:03.07 bitrate=1149. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 6580 fps= 55 q=2.0 size=   17431kB time=00:02:04.10 bitrate=1150. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 6602 fps= 55 q=0.0 size=   17730kB time=00:02:05.12 bitrate=1160. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 6648 fps= 55 q=1.0 size=   18802kB time=00:02:06.12 bitrate=1221. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 6710 fps= 55 q=0.0 size=   18824kB time=00:02:07.15 bitrate=1212. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 6770 fps= 55 q=0.0 size=   19539kB time=00:02:08.17 bitrate=1248. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 6831 fps= 55 q=1.0 size=   19560kB time=00:02:09.19 bitrate=1240. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 7013 fps= 56 q=0.0 size=   21011kB time=00:02:10.39 bitrate=1320. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 7256 fps= 58 q=0.0 size=   22483kB time=00:02:10.52 bitrate=1411. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 7625 fps= 60 q=0.0 size=   24692kB time=00:02:10.58 bitrate=1549. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 7828 fps= 62 q=2.0 size=   25456kB time=00:02:10.82 bitrate=1594. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 7895 fps= 62 q=0.0 size=   35079kB time=00:02:12.03 bitrate=2176. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 7936 fps= 62 q=2.0 size=   42929kB time=00:02:12.83 bitrate=2647. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 7981 fps= 62 q=2.0 size=   53139kB time=00:02:13.53 bitrate=3259. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 8030 fps= 62 q=0.0 size=   63944kB time=00:02:14.32 bitrate=3899. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 8081 fps= 62 q=0.0 size=   74523kB time=00:02:15.16 bitrate=4516. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 8128 fps= 62 q=2.0 size=   84385kB time=00:02:15.94 bitrate=5084. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 8145 fps= 62 q=1.0 size=   87711kB time=00:02:16.24 bitrate=5273. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 8193 fps= 62 q=1.0 size=   98046kB time=00:02:17.01 bitrate=5862. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 8204 fps= 62 q=0.0 size=  100564kB time=00:02:17.27 bitrate=6001. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 8252 fps= 62 q=0.0 size=  110255kB time=00:02:18.04 bitrate=6543. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
frame= 8266 fps= 62 q=2.0 size=  113024kB time=00:02:18.29 bitrate=6695. 5d7095f4-8035-4dc5-85e1-535f90d1935b   
[tls @ 0x563ee0299440] Error in the push function.
av_interleaved_write_frame(): Input/output error
    Last message repeated 2 times
Error writing trailer of rtmps://a.rtmps.youtube.com/live2/****: Input/output error
frame= 8311 fps= 62 q=2.0 Lsize=  123071kB time=00:02:19.02 bitrate=7252.1kbits/s dup=7716 drop=1 speed=1.04x    
video:123035kB audio:202kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[tls @ 0x563ee0299440] The specified session has been invalidated for some reason.
    Last message repeated 1 times
[libx264 @ 0x563ee026fe80] frame I:70    Avg QP: 0.31  size:221022
[libx264 @ 0x563ee026fe80] frame P:2770  Avg QP: 0.05  size: 29868
[libx264 @ 0x563ee026fe80] frame B:5471  Avg QP: 1.56  size:  6271
[libx264 @ 0x563ee026fe80] consecutive B-frames:  0.8%  1.7% 97.5%
[libx264 @ 0x563ee026fe80] mb I  I16..4: 100.0%  0.0%  0.0%
[libx264 @ 0x563ee026fe80] 
mb P  I16..4:  1.8%  0.0%  0.0%  P16..4:  5.2%  0.0%  0.0%  0.0%  0.0%    skip:93.0%
[libx264 @ 0x563ee026fe80] mb B  I16..4:  0.1%  0.0%  0.0%  B16..8:  0.6%  0.0%  0.0%  direct: 2.8%  skip:96.5%  L0:46.8% L1:38.5% BI:14.7%
[libx264 @ 0x563ee026fe80] final ratefactor: -5.03
[libx264 @ 0x563ee026fe80] coded y,uvDC,uvAC intra: 50.9% 43.8% 40.5% inter: 2.3% 1.8% 1.7%
[libx264 @ 0x563ee026fe80] i16 v,h,dc,p: 53% 30%  4% 12%
[libx264 @ 0x563ee026fe80] i8c dc,h,v,p: 58% 24%  8% 10%
[libx264 @ 0x563ee026fe80] kb/s:7653.50
[aac @ 0x563ee0282500] Qavg: 61059.410
Conversion failed!
[ERROR] 16:33:59 Error: write EPIPE