Recherche avancée

Sur d’autres sites (9546)

• Zipping Conda Environment Breaks Audioread's Backend (Python/Pyspark)

  25 octobre 2017, par Tim

  I have previously build pyspark environments using conda to package all dependancies and ship them to all the nodes at runtime. Here’s how I create the environment :

  `conda/bin/conda create -p conda_env --copy -y python=2  \
  
  numpy scipy ffmpeg gcc libsndfile gstreamer pygobject audioread librosa`
  
  
  
  `zip -r conda_env.zip conda_env`

  Then sourcing conda_env and running pyspark shell I can successfully execute :

  `import librosa
  
  y, sr = librosa.load("test.m4a")`

  Note without the environment sourced this script results in an error as ffmpeg/gstreamer are NOT installed on my locally.

  Submitting a script to the cluster results in a librosa.load error which traces back to audioread indicating the backend (either gstreamer or ffmpeg) can no longer be found in the zipped archive environment. The stacktrace is below :

  Submit :

  `PYSPARK_PYTHON=./NODE/conda_env/bin/python spark-submit --verbose \
  
          --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./NODE/conda_env/bin/python \
  
          --conf spark.yarn.appMasterEnv.PYTHON_EGG_CACHE=/tmp \
  
          --conf spark.executorEnv.PYTHON_EGG_CACHE=/tmp \
  
          --conf spark.yarn.executor.memoryOverhead=1024 \
  
          --conf spark.hadoop.validateOutputSpecs=false \
  
          --conf spark.driver.cores=5 \
  
          --conf spark.driver.maxResultSize=0 \
  
          --master yarn --deploy-mode cluster --queue production \
  
          --num-executors 20 --executor-cores 5 --executor-memory 40G \
  
          --driver-memory 20G --archives conda_env.zip#NODE \
  
          --jars /data/environments/sqljdbc41.jar \
  
          script.py`

  Trace :

  `Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/pyspark.zip/pyspark/worker.py", line 172, in main
  
      process()
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/pyspark.zip/pyspark/worker.py", line 167, in process
  
      serializer.dump_stream(func(split_index, iterator), outfile)
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/pyspark.zip/pyspark/serializers.py", line 263, in dump_stream
  
      vs = list(itertools.islice(iterator, batch))
  
    File "script.py", line 245, in <lambda>
  
    File "script.py", line 119, in download_audio
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/NODE/conda_env/lib/python2.7/site-packages/librosa/core/audio.py", line 107, in load
  
      with audioread.audio_open(os.path.realpath(path)) as input_file:
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/NODE/conda_env/lib/python2.7/site-packages/audioread/__init__.py", line 114, in audio_open
  
      raise NoBackendError()
  
  NoBackendError`
  
  </lambda>

  My question is : How can I package this archive so that librosa (really audioread) is able to find the backend and load .m4a files ?

• FFMPEG fails with only two input frames

  3 mars 2019, par JeffThompson

  I’d like to use ffmpeg’s great frame interpolation to blend two images. I get great results when testing with about a dozen frames, but when only using two it finishes immediately and I get a video file that can’t be opened.

  My command :

  ffmpeg -y -r 24 -pattern_type glob -i "TestFrames/*[0-1].png" -pix_fmt yuv420p -filter:v "minterpolate='mi_mode=mci:mc_mode=aobmc:me_mode=bidir:vsbmc=1:fps=1024'" -vsync 2 8.mp4

  Output from ffmpeg :

  ffmpeg version 4.1.1 Copyright (c) 2000-2019 the FFmpeg developers
  
    built with Apple LLVM version 10.0.0 (clang-1000.11.45.5)
  
    configuration: --prefix=/usr/local/Cellar/ffmpeg/4.1.1 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags='-I/Library/Java/JavaVirtualMachines/openjdk-11.0.2.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/openjdk-11.0.2.jdk/Contents/Home/include/darwin' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-libaom --enable-libsoxr
  
    libavutil      56. 22.100 / 56. 22.100
  
    libavcodec     58. 35.100 / 58. 35.100
  
    libavformat    58. 20.100 / 58. 20.100
  
    libavdevice    58.  5.100 / 58.  5.100
  
    libavfilter     7. 40.101 /  7. 40.101
  
    libavresample   4.  0.  0 /  4.  0.  0
  
    libswscale      5.  3.100 /  5.  3.100
  
    libswresample   3.  3.100 /  3.  3.100
  
    libpostproc    55.  3.100 / 55.  3.100
  
  Input #0, image2, from 'TestFrames/*.png':
  
    Duration: 00:00:00.08, start: 0.000000, bitrate: N/A
  
      Stream #0:0: Video: png, rgba(pc), 1080x1080, 25 tbr, 25 tbn, 25 tbc
  
  Stream mapping:
  
    Stream #0:0 -> #0:0 (png (native) -> h264 (libx264))
  
  Press [q] to stop, [?] for help
  
  [libx264 @ 0x7fa5c2813200] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
  
  [libx264 @ 0x7fa5c2813200] profile High, level 6.1
  
  [libx264 @ 0x7fa5c2813200] 264 - core 155 r2917 0a84d98 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
  
  Output #0, mp4, to '8.mp4':
  
    Metadata:
  
      encoder         : Lavf58.20.100
  
      Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 1080x1080, q=-1--1, 1024 fps, 16384 tbn, 1024 tbc
  
      Metadata:
  
        encoder         : Lavc58.35.100 libx264
  
      Side data:
  
        cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
  
  frame=    0 fps=0.0 q=0.0 Lsize=       0kB time=00:00:00.00 bitrate=N/A speed=   0x    
  
  video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

  (Why fps=1024 ? I wanted to generate a a bunch of frames between the two images, so I plan to later separate the resulting video into separate images.)

• Evolution #4300 (Nouveau) : Rendre facultatif les plugins-dist

  1er mars 2019, par Franck D

  Hello :-)
  Le commit de toutati https://zone.spip.net/trac/spip-zone/changeset/114194/spip-zone m’a fait repenser à ce vieux sujet (2009) https://blog.spip.net/SPIP-Core-tout-change-rien-ne-change.html
  Jluc avait fait un sujet à ce propos https://contrib.spip.net/plugins-dist-indispensables-ou-non
  Je me doute que cela représente un boulot de malade... mais ne faudrait’il pas dire au moins dans la F.A.Q. https://www.spip.net/fr_rubrique170.html les plugins-dist qui sont encore obligatoire ?

  J’ai fait le choix de "evolution" et non de "documentation" concernant le ticket, car sans ticket personne n’y pense :-D
  Pour spip 3.3, c’est sans doute un peu tard, mais à chaque sortie majeure, ne serait’il pas possible de rendre au moins "un" plugins-dist optionnel ?

  Pour le moment, en spip 3.3 [24240], il y a comme plugins-dist https://zone.spip.net/trac/spip-zone/browser/spip-zone/_core_/plugins
  Mais tous ne sont pas obligatoirement installer avec la dist (grenier, ?)
  Faudrait rayer les plugs qui sont en options

  Aide
  Archiviste
  Breves
  Compagnon
  Compresseur
  dev
  dist
  dump
  filtre_images
  forum
  grenier
  jquery_ui
  mediabox
  medias
  mots
  msie_compat
  organiseur
  petitions
  plan
  porte_plume
  revisions
  safehtml
  sites
  squelettes_par-rubrique
  statistique
  svp
  textwhell
  url_etendues
  vertèbres

  Franck