Recherche avancée

Sur d’autres sites (11355)

• Installing ffmpeg, librosa nad pydub in Apache Spark container

  17 avril 2023, par yaviens

  I'm working on a Python spark streaming project, that needs to use pydub and librosa to process audio, this libraries require having installed ffmpeg library. I'm having trubles to build the spark containers with this libraries, I don't know how to solve this problem.

  


  I use a docker-compose.yml to buil the images, define ports, etc of the spark master and workers.
docker-compose :

  


  version: "3.3"
services:
  spark-master:
    build:
      context: ./
      dockerfile: Dockerfile
    #image: docker.io/bitnami/spark:3.3
    ports:
      - "9090:8080"
      - "7077:7077"
    volumes:
       - ./apps:/opt/spark-apps
       - ./data:/opt/spark-data
       - ./data:/data
       - ./src:/src
       - ./output:/output
    environment:
      - SPARK_LOCAL_IP=spark-master
      - SPARK_WORKLOAD=master
  spark-worker-a:
    build:
      context: ./
      dockerfile: Dockerfile
    #image: docker.io/bitnami/spark:3.3
    ports:
      - "9091:8080"
      - "7000:7000"
    depends_on:
      - spark-master
    environment:
      - SPARK_MASTER=spark://spark-master:7077
      - SPARK_WORKER_CORES=1
      - SPARK_WORKER_MEMORY=1G
      - SPARK_DRIVER_MEMORY=1G
      - SPARK_EXECUTOR_MEMORY=1G
      - SPARK_WORKLOAD=worker
      - SPARK_LOCAL_IP=spark-worker-a
    volumes:
       - ./apps:/opt/spark-apps
       - ./data:/opt/spark-data
       - ./data:/data
       - ./src:/src
       - ./output:/output       
  spark-worker-b:
    build:
      context: ./
      dockerfile: Dockerfile
    #image: docker.io/bitnami/spark:3.3
    ports:
      - "9092:8080"
      - "7001:7000"
    depends_on:
      - spark-master
    environment:
      - SPARK_MASTER=spark://spark-master:7077
      - SPARK_WORKER_CORES=1
      - SPARK_WORKER_MEMORY=1G
      - SPARK_DRIVER_MEMORY=1G
      - SPARK_EXECUTOR_MEMORY=1G
      - SPARK_WORKLOAD=worker
      - SPARK_LOCAL_IP=spark-worker-b
    volumes:
        - ./apps:/opt/spark-apps
        - ./data:/opt/spark-data
        - ./data:/data
        - ./src:/src
        - ./output:/output 


  


  In the same path of the docker-compose.yml is the Dockerfile I'm using to build the image :

  


  # builder step used to download and configure spark environment
FROM openjdk:11.0.11-jre-slim-buster as builder

# Add Dependencies for PySpark
RUN apt-get update && apt-get install -y curl vim wget software-properties-common ssh net-tools ca-certificates python3 python3-pip python3-numpy python3-matplotlib python3-scipy python3-pandas python3-simpy

RUN update-alternatives --install "/usr/bin/python" "python" "$(which python3)" 1

# Fix the value of PYTHONHASHSEED
# Note: this is needed when you use Python 3.3 or greater
ENV SPARK_VERSION=3.0.2 \
HADOOP_VERSION=3.2 \
SPARK_HOME=/opt/spark \
PYTHONHASHSEED=1

# Download and uncompress spark from the apache archive
RUN wget --no-verbose -O apache-spark.tgz "https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" \
&& mkdir -p /opt/spark \
&& tar -xf apache-spark.tgz -C /opt/spark --strip-components=1 \
&& rm apache-spark.tgz


# Apache spark environment
FROM builder as apache-spark

WORKDIR /opt/spark

ENV SPARK_MASTER_PORT=7077 \
SPARK_MASTER_WEBUI_PORT=8080 \
SPARK_LOG_DIR=/opt/spark/logs \
SPARK_MASTER_LOG=/opt/spark/logs/spark-master.out \
SPARK_WORKER_LOG=/opt/spark/logs/spark-worker.out \
SPARK_WORKER_WEBUI_PORT=8080 \
SPARK_WORKER_PORT=7000 \
SPARK_MASTER="spark://spark-master:7077" \
SPARK_WORKLOAD="master"

EXPOSE 8080 7077 6066

RUN mkdir -p $SPARK_LOG_DIR && \
touch $SPARK_MASTER_LOG && \
touch $SPARK_WORKER_LOG && \
ln -sf /dev/stdout $SPARK_MASTER_LOG && \
ln -sf /dev/stdout $SPARK_WORKER_LOG

# Install ffmpeg lib
RUN apt-get -y update
RUN apt-get -y upgrade
RUN apt-get install -y ffmpeg
RUN apt-get -y install apt-utils gcc libpq-dev libsndfile-dev

#RUN apt-get update \
#&& apt-get upgrade -y \
#&& apt-get install -y \
#&& apt-get -y install apt-utils gcc libpq-dev libsndfile-dev

# Install required python libs
COPY requirements.txt .
RUN pip3 install -r requirements.txt

COPY start-spark.sh /

CMD ["/bin/bash", "https://net.cloudinfrastructureservices.co.uk/start-spark.sh"]


  


  The start-spark.sh :

  


  #!/bin/bash
. "https://net.cloudinfrastructureservices.co.uk/opt/spark/bin/load-spark-env.sh"
# When the spark work_load is master run class org.apache.spark.deploy.master.Master
if [ "$SPARK_WORKLOAD" == "master" ];
then

export SPARK_MASTER_HOST=`hostname`

cd /opt/spark/bin && ./spark-class org.apache.spark.deploy.master.Master --ip $SPARK_MASTER_HOST --port $SPARK_MASTER_PORT --webui-port $SPARK_MASTER_WEBUI_PORT >> $SPARK_MASTER_LOG

elif [ "$SPARK_WORKLOAD" == "worker" ];
then
# When the spark work_load is worker run class org.apache.spark.deploy.master.Worker
cd /opt/spark/bin && ./spark-class org.apache.spark.deploy.worker.Worker --webui-port $SPARK_WORKER_WEBUI_PORT $SPARK_MASTER >> $SPARK_WORKER_LOG

elif [ "$SPARK_WORKLOAD" == "submit" ];
then
    echo "SPARK SUBMIT"
else
    echo "Undefined Workload Type $SPARK_WORKLOAD, must specify: master, worker, submit"
fi


  


  When I execute : "docker-compose up" I have the next error message :

  


  [+] Running 4/4
 - Network spark_container_default             Created                                                                                0.9s 
 - Container spark_container-spark-master-1    Created                                                                                0.3s 
 - Container spark_container-spark-worker-a-1  Created                                                                                0.4s 
 - Container spark_container-spark-worker-b-1  Created                                                                                0.4s 
Attaching to spark_container-spark-master-1, spark_container-spark-worker-a-1, spark_container-spark-worker-b-1
spark_container-spark-master-1    | /bin/bash: https://net.cloudinfrastructureservices.co.uk/start-spark.sh: No such file or directory     
spark_container-spark-master-1 exited with code 127
spark_container-spark-worker-a-1  | /bin/bash: https://net.cloudinfrastructureservices.co.uk/start-spark.sh: No such file or directory     
spark_container-spark-worker-b-1  | /bin/bash: https://net.cloudinfrastructureservices.co.uk/start-spark.sh: No such file or directory     
spark_container-spark-worker-a-1 exited with code 127
spark_container-spark-worker-b-1 exited with code 127


  


• Need help to compile FFmpeg with MSVC tools

  30 janvier 2018, par Antwane

  I need to compile FFmpeg on Windows 10 using MSVC 2017 compiler. I followed some guides from :

  • https://trac.ffmpeg.org/wiki/CompilationGuide/MSVC
  • https://www.ffmpeg.org/platform.html#Microsoft-Visual-C_002b_002b-or-Intel-C_002b_002b-Compiler-for-Windows
  • https://pracucci.com/compile-ffmpeg-on-windows-with-visual-studio-compiler.html

  I installed current stable version of MSYS, fixed link.exe and use of PATH environment, installed make diffutils gcc pkg-config and downloaded c99-to-c89, nasm, and inttypes.h

  I think my environment is now correctly set up, I ran MSYS shell from MSVC Command prompt and have everything reachable.

  $ which cl
  
  /c/Program Files (x86)/Microsoft Visual Studio/2017/Professional/VC/Tools/MSVC/14.12.25827/bin/HostX64/x64/cl
  
  
  
  $ which link
  
  /c/Program Files (x86)/Microsoft Visual Studio/2017/Professional/VC/Tools/MSVC/14.12.25827/bin/HostX64/x64/link
  
  
  
  $ which yasm
  
  /home/Antoine/bin/yasm
  
  
  
  $ which c99conv
  
  /home/Antoine/bin/c99conv
  
  
  
  $ which c99wrap
  
  /home/Antoine/bin/c99wrap

  Now I configure the build

  $ ./configure --toolchain=msvc
  
  install prefix            /usr/local
  
  source path               .
  
  C compiler                cl
  
  C library                 msvcrt
  
  ARCH                      x86 (generic)
  
  big-endian                no
  
  runtime cpu detection     yes
  
  standalone assembly       yes
  
  x86 assembler             nasm
  
  MMX enabled               yes
  
  MMXEXT enabled            yes
  
  3DNow! enabled            yes
  
  3DNow! extended enabled   yes
  
  SSE enabled               yes
  
  SSSE3 enabled             yes
  
  AESNI enabled             yes
  
  AVX enabled               yes
  
  AVX2 enabled              yes
  
  XOP enabled               yes
  
  FMA3 enabled              yes
  
  FMA4 enabled              yes
  
  i686 features enabled     yes
  
  CMOV is fast              yes
  
  EBX available             no
  
  EBP available             no
  
  debug symbols             yes
  
  strip symbols             no
  
  optimize for size         no
  
  optimizations             yes
  
  static                    yes
  
  shared                    no
  
  postprocessing support    no
  
  network support           yes
  
  threading support         w32threads
  
  safe bitstream reader     yes
  
  texi2html enabled         no
  
  perl enabled              no
  
  pod2man enabled           no
  
  makeinfo enabled          no
  
  makeinfo supports HTML    no
  
  
  
  External libraries:
  
  schannel                 xlib
  
  
  
  External libraries providing hardware acceleration:
  
  cuda                     cuvid                    d3d11va                  dxva2                    nvenc
  
  
  
  Libraries:
  
  avcodec                  avdevice                 avfilter                 avformat                 avutil                   swresample               swscale
  
  
  
  Programs:
  
  ffmpeg                   ffprobe
  
  
  
  [...]
  
  Hundreds of modules, library, etc.
  
  [...]
  
  
  
  License: LGPL version 2.1 or later
  
  Creating configuration files ...
  
  config.h is unchanged
  
  config.asm is unchanged
  
  libavutil/avconfig.h is unchanged
  
  libavcodec/bsf_list.c is unchanged
  
  libavformat/protocol_list.c is unchanged

  But when I run make it stops very quickly

  $ make
  
  Makefile:47: la cible « qt-faststart » ne correspond pas au motif de cible
  
  Makefile:47: la cible « trasher » ne correspond pas au motif de cible
  
  Makefile:47: la cible « uncoded_frame » ne correspond pas au motif de cible
  
  Makefile:91: ffbuild/library.mak: No such file or directory
  
  make: *** Aucune règle pour fabriquer la cible « ffbuild/library.mak ». Arrêt.

  Sorry for the french wording, I don’t know how to tell MSVC print its output in english. Basically, the error means :

  Makefile:47: target « qt-faststart » doesn't match the target pattern
  
  make: *** No rule to make target « ffbuild/library.mak ». Stopped.

  Side notes :

  $ tree ffbuild
  
  ffbuild
  
  ├── config.fate
  
  ├── config.log
  
  ├── config.mak
  
  └── config.sh

  That’s right, library.mak were not generated into ffbuild, but I don’t know why, and configure script didn’t output any error message...

  Does anybody can help me to find out what am I doing wrong ?

  Edit :

  make distclean doesn’t help

  $ make distclean
  
  Makefile:47: la cible « qt-faststart » ne correspond pas au motif de cible
  
  Makefile:47: la cible « trasher » ne correspond pas au motif de cible
  
  Makefile:47: la cible « uncoded_frame » ne correspond pas au motif de cible
  
  Makefile:91: ffbuild/library.mak: No such file or directory
  
  make: *** Aucune règle pour fabriquer la cible « ffbuild/library.mak ». Arrêt.

• python [WinError 2] the System Cannot Find the File Specified

  15 août 2024, par user26831166

  Code cant create a certain file
The thing is code isn't mine a took it from a friend and my friend get it from another person
and this 2 person can run code without any problem
but i have.

  


  import os
import random
import shutil
import subprocess

# Путь к папке с видео
video_folder = r'D:\bots\ttvidads\VID\Videorez'

# Путь к папке для сохранения результатов
output_folder = r'D:\bots\ttvidads\VID\ZAGOTOVKI\Videopod1'

# Очищаем содержимое конечной папки перед сохранением
for file in os.listdir(output_folder):
    file_path = os.path.join(output_folder, file)
    try:
        if os.path.isfile(file_path):
            os.unlink(file_path)
    except Exception as e:
        print(f"Failed to delete {file_path}. Reason: {e}")

# Получаем список видеофайлов
video_files = [os.path.join(video_folder, file) for file in os.listdir(video_folder) if file.endswith(('.mp4', '.avi'))]

# Выбираем случайное видео
random_video = random.choice(video_files)

# Получаем длительность видео в секундах
video_duration_command = f'ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "{random_video}"'
video_duration_process = subprocess.Popen(video_duration_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
video_duration_output, _ = video_duration_process.communicate()
video_duration = float(video_duration_output)

# Выбираем случайное начальное время для вырезания
random_start = random.randrange(0, int(video_duration) - 19, 8)

# Получаем ширину и высоту исходного видео
video_info_command = f'ffprobe -v error -select_streams v:0 -show_entries stream=width,height -of csv=s=x:p=0 "{random_video}"'
video_info_process = subprocess.Popen(video_info_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
video_info_output, _ = video_info_process.communicate()
video_width, video_height = map(int, video_info_output.strip().split(b'x'))

# Вычисляем новые координаты x1 и x2 для обрезки
max_x1 = video_width - int(video_height * 9 / 16)
random_x1 = random.randint(0, max_x1)
random_x2 = random_x1 + int(video_height * 9 / 16)

# Формируем команду для FFmpeg для выборки случайного отрезка видео с соотношением 9:16
ffmpeg_command = f'ffmpeg -hwaccel cuda -ss {random_start} -i "{random_video}" -t 19 -vf "crop={random_x2-random_x1}:{video_height}:{random_x1}:0" -c:v h264_nvenc -preset default -an -c:a aac -b:a 128k "{output_folder}\\temp.mp4"'

# Выполняем команду с помощью subprocess
subprocess.run(ffmpeg_command, shell=True)

# Изменяем яркость, контрастность и размываем видео
brightness_factor = random.uniform(-0.18, -0.12)  # Случайный коэффициент яркости
contrast_factor = random.uniform(0.95, 1.05)  # Случайный коэффициент контрастности
blur_factor = random.uniform(4, 5)  # Случайный коэффициент размытия

# Формируем команду для FFmpeg для изменения яркости, контрастности и размытия видео
ffmpeg_modify_command = f'ffmpeg -hwaccel cuda -i "{output_folder}\\temp.mp4" -vf "eq=brightness={brightness_factor}:contrast={contrast_factor},boxblur={blur_factor}:{blur_factor}" -c:v h264_nvenc -preset default -an -c:a aac -b:a 128k "{output_folder}\\temp_modify.mp4"'

# Выполняем команду с помощью subprocess
subprocess.run(ffmpeg_modify_command, shell=True)

# Растягиваем видео до нужного разрешения (1080x1920)
ffmpeg_stretch_command = f'ffmpeg -hwaccel cuda -i "{output_folder}\\temp_modify.mp4" -vf "scale=1080:1920" -c:v h264_nvenc -preset default -an -c:a aac -b:a 128k -r 30 "{output_folder}\\final_output.mp4"'

# Выполняем команду с помощью subprocess
subprocess.run(ffmpeg_stretch_command, shell=True)

# Удаляем временные файлы
os.remove(os.path.join(output_folder, 'temp.mp4'))
os.remove(os.path.join(output_folder, 'temp_modify.mp4'))

print("Видеофайл успешно обработан и сохранен.")


  


  Error i got after run the code

  


  = RESTART: D:\Bots\2vidpod.py&#xA;Traceback (most recent call last):&#xA;  File "D:\Bots\2vidpod.py", line 71, in <module>&#xA;    os.remove(os.path.join(output_folder, &#x27;temp.mp4&#x27;))&#xA;FileNotFoundError: [WinError 2] Не удается найти указанный файл: &#x27;D:\\bots\\ttvidads\\VID\\ZAGOTOVKI\\Videopod1\\temp.mp4&#x27;&#xA;</module>

  


  so things i checked is
path is right
programs is installed FFMPEG and PYTHON all additional libraries downloaded
i pretty sure error caused by regular path and i wanna know if absolute path can do the thing