Recherche avancée

Sur d’autres sites (5469)

• Zipping Conda Environment Breaks Audioread's Backend (Python/Pyspark)

  25 octobre 2017, par Tim

  I have previously build pyspark environments using conda to package all dependancies and ship them to all the nodes at runtime. Here’s how I create the environment :

  `conda/bin/conda create -p conda_env --copy -y python=2  \
  
  numpy scipy ffmpeg gcc libsndfile gstreamer pygobject audioread librosa`
  
  
  
  `zip -r conda_env.zip conda_env`

  Then sourcing conda_env and running pyspark shell I can successfully execute :

  `import librosa
  
  y, sr = librosa.load("test.m4a")`

  Note without the environment sourced this script results in an error as ffmpeg/gstreamer are NOT installed on my locally.

  Submitting a script to the cluster results in a librosa.load error which traces back to audioread indicating the backend (either gstreamer or ffmpeg) can no longer be found in the zipped archive environment. The stacktrace is below :

  Submit :

  `PYSPARK_PYTHON=./NODE/conda_env/bin/python spark-submit --verbose \
  
          --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./NODE/conda_env/bin/python \
  
          --conf spark.yarn.appMasterEnv.PYTHON_EGG_CACHE=/tmp \
  
          --conf spark.executorEnv.PYTHON_EGG_CACHE=/tmp \
  
          --conf spark.yarn.executor.memoryOverhead=1024 \
  
          --conf spark.hadoop.validateOutputSpecs=false \
  
          --conf spark.driver.cores=5 \
  
          --conf spark.driver.maxResultSize=0 \
  
          --master yarn --deploy-mode cluster --queue production \
  
          --num-executors 20 --executor-cores 5 --executor-memory 40G \
  
          --driver-memory 20G --archives conda_env.zip#NODE \
  
          --jars /data/environments/sqljdbc41.jar \
  
          script.py`

  Trace :

  `Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/pyspark.zip/pyspark/worker.py", line 172, in main
  
      process()
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/pyspark.zip/pyspark/worker.py", line 167, in process
  
      serializer.dump_stream(func(split_index, iterator), outfile)
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/pyspark.zip/pyspark/serializers.py", line 263, in dump_stream
  
      vs = list(itertools.islice(iterator, batch))
  
    File "script.py", line 245, in <lambda>
  
    File "script.py", line 119, in download_audio
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/NODE/conda_env/lib/python2.7/site-packages/librosa/core/audio.py", line 107, in load
  
      with audioread.audio_open(os.path.realpath(path)) as input_file:
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/NODE/conda_env/lib/python2.7/site-packages/audioread/__init__.py", line 114, in audio_open
  
      raise NoBackendError()
  
  NoBackendError`
  
  </lambda>

  My question is : How can I package this archive so that librosa (really audioread) is able to find the backend and load .m4a files ?

• Autogenerate HTML5 tags from YouTube-DL

  13 novembre 2015, par Terence Eden

  I’m using youtube-dl to download videos I’ve stored on YouTube. It gives me the ability to download various formats, thumbnails, and subtitles.

  Is there any way I can automatically generate an HTML5 <video></video> snippet from the downloaded files ?

  For example, I’d like the end result to be .txt file containing :

  <video poster="file-160.jpg">
  <source src="file-135.mp4" type="video/mp4; codecs=mp4a.40.2, avc1.42001E">
  <source src="file-43.webm" type="video/webm; codecs=vorbis, vp8.0">
  ...
  <track kind="subtitles" src="file-160.en.srt">
</track></source></source></video>

  At the moment, I have a file list like :

  • file-135.mp4
  • file-18.mp4
  • file-134.mp4
  • file-160.mp4
  • file-43.webm
  • file-5.flv
  • file-36.3gp
  • file-17.3gp
  • file-160.jpg
  • file-160.en.srt

  I can run some Python/Ruby/bash over them to generate the <source src="..."></source>code>, but the problem is, I don't know what codecs each of those videos are, so I can't generate the <code>; codecs= portion.

  Using avconv or ffmpeg I can get the codecs, but not in a suitable format for embedding into HTML.

  I’m wary of asking "what tool should I use" - but is there any way to get avconv/ffmpeg/youtube-dl to spit out the codec information in a format I can put into an HTML5 tag ?

  Or, is there a way to get YouTube-DL only to spit out HTML5 compatible files with known codecs ?

• Revision 6723e34224 : Merge "fix permissions on cpplint.py (0644->0755)" into experimental

  4 mai 2013, par James Zern

  Merge "fix permissions on cpplint.py (0644->0755)" into experimental