Recherche avancée

Sur d’autres sites (8008)

• Zipping Conda Environment Breaks Audioread's Backend (Python/Pyspark)

  25 octobre 2017, par Tim

  I have previously build pyspark environments using conda to package all dependancies and ship them to all the nodes at runtime. Here’s how I create the environment :

  `conda/bin/conda create -p conda_env --copy -y python=2  \
  
  numpy scipy ffmpeg gcc libsndfile gstreamer pygobject audioread librosa`
  
  
  
  `zip -r conda_env.zip conda_env`

  Then sourcing conda_env and running pyspark shell I can successfully execute :

  `import librosa
  
  y, sr = librosa.load("test.m4a")`

  Note without the environment sourced this script results in an error as ffmpeg/gstreamer are NOT installed on my locally.

  Submitting a script to the cluster results in a librosa.load error which traces back to audioread indicating the backend (either gstreamer or ffmpeg) can no longer be found in the zipped archive environment. The stacktrace is below :

  Submit :

  `PYSPARK_PYTHON=./NODE/conda_env/bin/python spark-submit --verbose \
  
          --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./NODE/conda_env/bin/python \
  
          --conf spark.yarn.appMasterEnv.PYTHON_EGG_CACHE=/tmp \
  
          --conf spark.executorEnv.PYTHON_EGG_CACHE=/tmp \
  
          --conf spark.yarn.executor.memoryOverhead=1024 \
  
          --conf spark.hadoop.validateOutputSpecs=false \
  
          --conf spark.driver.cores=5 \
  
          --conf spark.driver.maxResultSize=0 \
  
          --master yarn --deploy-mode cluster --queue production \
  
          --num-executors 20 --executor-cores 5 --executor-memory 40G \
  
          --driver-memory 20G --archives conda_env.zip#NODE \
  
          --jars /data/environments/sqljdbc41.jar \
  
          script.py`

  Trace :

  `Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/pyspark.zip/pyspark/worker.py", line 172, in main
  
      process()
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/pyspark.zip/pyspark/worker.py", line 167, in process
  
      serializer.dump_stream(func(split_index, iterator), outfile)
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/pyspark.zip/pyspark/serializers.py", line 263, in dump_stream
  
      vs = list(itertools.islice(iterator, batch))
  
    File "script.py", line 245, in <lambda>
  
    File "script.py", line 119, in download_audio
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/NODE/conda_env/lib/python2.7/site-packages/librosa/core/audio.py", line 107, in load
  
      with audioread.audio_open(os.path.realpath(path)) as input_file:
  
    File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/NODE/conda_env/lib/python2.7/site-packages/audioread/__init__.py", line 114, in audio_open
  
      raise NoBackendError()
  
  NoBackendError`
  
  </lambda>

  My question is : How can I package this archive so that librosa (really audioread) is able to find the backend and load .m4a files ?

• Evolution #4749 (Nouveau) : [UX] Comportement des labels : quoi par défaut, quoi ponctuel ?

  27 avril 2021, par RastaPopoulos ♥

  Ce ticket sert à réfléchir et possiblement reconcevoir les choix par défaut pour les labels des formulaires.

  État des lieux¶

  On le sait, l’ergo c’est normalement beaucoup d’objectif : certains placements, certaines tailles, épaisseurs, etc fonctionnent mieux que d’autres, et ceci est prouvable par tests utilisateurs.

  Or cela fait des années que les tests par eye-tracking montrent que les formulaires sont
  1) lu plus rapidement
  2) avec une meilleure compréhension
  lorsque les labels sont au-dessus des champs.

  Ça ne veut pas dire qu’il faut totalement interdire autrement mais : ça veut clairement dire que ça devrait être le comportement par défaut. Et seulement ponctuellement, par choix explicite, pouvoir mettre les labels sur le côté.

  Par ailleurs les pros de l’ergo (sur base des mêmes tests) préconisent tou⋅tes : dans les rares cas où on met les labels sur le côté, ça devrait être calé à droite sur le champ, pour les mêmes raisons de compréhension.

  Les avantages des labels au-dessus :
  - prouvé que c’est bien mieux compris par tout le monde
  - lecture plus rapide
  - fonctionne de base sur tous les écrans, pas d’adaptation à faire
  - polyvalent et générique sur le contenu des labels : marche mieux quelque soit la longueur, et donc à prioriser dans un contexte multilingue
  => cela correspond bien au maximum de notre utilisation : un CMS multi-lingue, allant enfin vers le responsive, se souciant d’accessibilité.

  Le seul désavantage : allonge la hauteur des formulaires, mais ça n’a un impact surtout que pour les formulaires ayant vraiment vraiment beaucoup de champs, ce qui est rare !
  Quand un formulaire est extrêmement long, il y a même plusieurs méthodes qui peuvent être utilisées sans pour autant passer les labels sur le côté :
  1) placer certains champs sur le même ligne (prénom + nom, etc)
  2) découper le formulaire en plusieurs étapes.

  Proposition pour le futur¶

  - tous les labels doivent être au-dessus comme comportement par défaut
  - pour certains cas, une classe permet de mettre sur le côté : valable uniquement en grand écran, ça reste au-dessus en mobile first
  - si sur le côté : c’est mieux si aligné sur le champ (donc à droite en LTR)
  - ex de rare formulaire candidat : changement des dates

  Quelques sources¶

  Tests utilisateurs
  https://www.uxmatters.com/mt/archives/2006/07/label-placement-in-forms.php

  Placing a label above an input field works better in most cases
  Placing labels above input fields is preferable
  In most cases, when placing labels to the left of input fields, using left-aligned labels imposes a heavy cognitive workload on users
  if you choose to place them to the left of input fields, at least make them right aligned

  Chez le très connu cabinet d’ergo Nielsen Group
  https://www.nngroup.com/articles/form-design-white-space/

  We recommend placing field labels above the corresponding text fields [en gras chez eux !]
  it makes the form easier to scan, because users can see the text field in the same fixation as the label. Top placement also allows for longer field labels
  If the labels are too far to the left, it can be difficult to associate the correct label with its corresponding field

  Chez Adobe, ils préconisent de suivre les recommandations de la première source
  https://xd.adobe.com/ideas/principles/web-design/best-practices-form-design/

  Matteo Penzo’s 2006 article on label placement suggests that forms are completed faster if labels are on top of the fields. Top-aligned labels are good if you want users to scan the form as quickly as possible.
  The biggest advantage of top-aligned labels is that different-sized labels and localized versions can more easily fit the UI.
  Takeaway : If you want users to scan a form quickly, put labels above the fields. The layout will be easier to scan because the eye will move straight down the page. However, if you want users to read carefully, put labels to the left of the fields. This layout will slow down the reader and make them scan in a Z-shaped motion.

  Chez une appli de conception d’interface
  https://phase.com/magazine/usability-of-forms/

  from a cognitive point of view, the association is powerful
  Also, the eyes move only in one direction since the scanning is top down as compared to Z shape (left-right and top-bottom) for inline labels
  Design is space efficient and hence adaptable to all resolutions ; in short, responsive in nature
  We also get flexibility regarding the length of labels. This proves useful while working with variable label lengths like multilingual support for applications
  One drawback of this approach is the increased height of the form. However, it can be solved with alternate designs like a grouping of fields or stepper forms

• Creating a shared library that statically includes ffmpeg

  10 octobre 2017, par El Sampsa

  I’m having hard time trying to create a shared library that has ffmpeg libraries "baked in" as static ones.

  Consider the following directory schema :

  include/
  
    my own .h files
  
    ext/
  
      ffmpeg .h files
  
  lib/
  
    libav*.a archive files (softlinks to the actual .a files)
  
    libValkka.so (my shared library)
  
  test/
  
    mytest.cpp
  
  bin/
  
    (binaries appear here)

  I’ve come a long way (see Including objects to a shared library from a C++ archive (.a) ) and the library compiles ok with this : ([STUFF] has been omitted for brevity)

  /usr/bin/c++ -fPIC -std=c++14 -pthread -Iinclude/ext -I/usr/include/libdrm -g -shared -Wl,-soname,libValkka.so -o lib/libValkka.so CMakeFiles/Valkka.dir/src/avthread.cpp.o CMakeFiles/Valkka.dir/src/opengl.cpp.o CMakeFiles/Valkka.dir/src/openglthread.cpp.o [STUFF] CMakeFiles/Valkka.dir/src/filters.cpp.o -lX11 -lGLEW -lGLU -lGL -Wl,—allow-multiple-definition -Wl,-Bsymbolic -Wl,—whole-archive -Wreorder lib/libavdevice.a lib/libavfilter.a lib/libavformat.a lib/libavcodec.a lib/libavutil.a lib/libswscale.a lib/libswresample.a -Wl,—no-whole-archive

  However, when creating executables - their source code does not use any ffmpeg api (just my own api) - with :

  c++ -std=c++14 -pthread -Iinclude -Iinclude/ext -Llib test/mytest.cpp -lValkka -g -o bin/mytest

  I get a hoard of errors about missing ffmpeg dependencies. Not everything is missing, just some weird stuff :

  lib/libValkka.so: undefined reference to `pa_stream_get_index'
  
  lib/libValkka.so: undefined reference to `deflateInit_'
  
  lib/libValkka.so: undefined reference to `pa_stream_get_state'
  
  lib/libValkka.so: undefined reference to `lzma_stream_decoder'
  
  lib/libValkka.so: undefined reference to `BZ2_bzDecompress'
  
  lib/libValkka.so: undefined reference to `vaInitialize'
  
  lib/libValkka.so: undefined reference to `pa_stream_unref'
  
  lib/libValkka.so: undefined reference to `deflateInit2_'
  
  lib/libValkka.so: undefined reference to `snd_pcm_close'
  
  ...
  
  lib/libValkka.so: undefined reference to `vaGetDisplayDRM'
  
  lib/libValkka.so: undefined reference to `vaMaxNumEntrypoints'
  
  lib/libValkka.so: undefined reference to `uncompress'
  
  lib/libValkka.so: undefined reference to `pa_stream_drop'
  
  lib/libValkka.so: undefined reference to `pa_context_connect'
  
  lib/libValkka.so: undefined reference to `FT_Get_Kerning'
  
  lib/libValkka.so: undefined reference to `ass_free_track'
  
  lib/libValkka.so: undefined reference to `pa_operation_unref'
  
  lib/libValkka.so: undefined reference to `FT_Stroker_Done'
  
  lib/libValkka.so: undefined reference to `vaTerminate'
  
  lib/libValkka.so: undefined reference to `ass_new_track'
  
  lib/libValkka.so: undefined reference to `jack_client_close'
  
  ...
  
  lib/libValkka.so: undefined reference to `xcb_xfixes_query_version'
  
  lib/libValkka.so: undefined reference to `xcb_shape_rectangles'
  
  lib/libValkka.so: undefined reference to `pa_mainloop_free'
  
  lib/libValkka.so: undefined reference to `snd_device_name_hint'
  
  lib/libValkka.so: undefined reference to `vaCreateImage'
  
  lib/libValkka.so: undefined reference to `vaBeginPicture'
  
  lib/libValkka.so: undefined reference to `DtsSetColorSpace'
  
  lib/libValkka.so: undefined reference to `vaDestroyConfig'
  
  lib/libValkka.so: undefined reference to `pa_stream_writable_size'
  
  lib/libValkka.so: undefined reference to `snd_pcm_hw_params_get_buffer_size_max'
  
  lib/libValkka.so: undefined reference to `ass_read_file'

  This is pretty frustrating, especially when I can see that those names are included in the shared library..!

  nm lib/libValkka.so | grep "vaBeginPicture"

  gives

  U vaBeginPicture

  etc. I thought it might be a problem regarding the dependency order the archive .a files, and also tried with :

  ..... -Wl,—allow-multiple-definition -Wl,-Bsymbolic -Wl,—start-group -Wl,—whole-archive -Wreorder lib/libavdevice.a lib/libavfilter.a lib/libavformat.a lib/libavcodec.a lib/libavutil.a lib/libswscale.a lib/libswresample.a -Wl,—no-whole-archive -Wl,—end-group

  But the problem persists.

  I have succesfully created a shared library that does not "bake in" those .a archives, i.e. that just depends dynamically on ffmpeg libraries, and there are no such problems.

  I am baffled.. Have I misunderstood something fundamental, forgot some annoying linked option, or both ? Help appreciated !