Recherche avancée

Sur d’autres sites (3367)

• Unable to install ffmpeg on CircleCi 2.0

  16 novembre 2017, par Mateusz Urbański

  I have Ruby on Rails project which uses CircleCi to run tests. In the past I was using CircleCi 1.0 but now I migrated to CircleCi 2.0. I have problem with installing ffmpeg. CircleCi 2.0 uses Ubuntu 14.04. I install ffmpeg like this :

  # ffmpeg installation
  
  sudo add-apt-repository ppa:mc3man/trusty-media -y
  
  sudo apt-get update
  
  sudo apt-get install ffmpeg

  and my circle.yml config file looks like this :

  version: 2
  
  environment:
  
    TZ: "/usr/share/zoneinfo/America/Los_Angeles"
  
  
  
  jobs:
  
    build:
  
      parallelism: 2
  
      working_directory: ~/circleci-survey-builder
  
      docker:
  
        - image: circleci/ruby:2.4.1-node
  
          environment:
  
            PGHOST: 127.0.0.1
  
            PGUSER: ubuntu
  
            RAILS_ENV: test
  
        - image: circleci/postgres:9.6-alpine
  
          environment:
  
            POSTGRES_USER: ubuntu
  
            POSTGRES_DB: circle_test
  
            POSTGRES_PASSWORD: ''
  
      steps:
  
        - checkout
  
  
  
        - run:
  
            name: 'Install CircleCI dependencies'
  
            command: bash deploy/circle-dependencies.sh
  
  
  
        - type: cache-restore
  
          key: dashboard-{{ checksum "Gemfile.lock" }}
  
  
  
        - run:
  
            name: 'Install gems'
  
            command: bundle install --path vendor/bundle
  
  
  
        - type: cache-save
  
          key: dashboard-{{ checksum "Gemfile.lock" }}
  
          paths:
  
            - vendor/bundle
  
  
  
        - run:
  
            name: 'Install postgresql-client'
  
            command: sudo apt install postgresql-client
  
  
  
        - run:
  
            name: 'Create database.yml'
  
            command: mv config/database.ci.yml config/database.yml
  
  
  
        - run:
  
            name: Set up SurveyBuilder database
  
            command: bundle exec rake db:structure:load --trace
  
  
  
        - run:
  
            name: 'Run tests'
  
            command: |
  
              bundle exec rspec spec

  It returns following error when I run build on CircleCi :

  #!/bin/bash -eo pipefail
  
  bash deploy/circle-dependencies.sh
  
  
  
  Fetching: bundler-1.16.0.gem (100%)
  
  Successfully installed bundler-1.16.0
  
  1 gem installed
  
  sudo: add-apt-repository: command not found
  
  
  
  
  
  Ign http://deb.debian.org jessie InRelease
  
  
  
  Get:1 http://deb.debian.org jessie-updates InRelease [145 kB]
  
  
  
  10% [1 InRelease 13.8 kB/145 kB 10%] [Waiting for headers] [Connecting to securHit http://deb.debian.org jessie Release.gpg
  
  
  
  Hit http://deb.debian.org jessie Release
  
  
  
  Get:2 http://deb.debian.org jessie-updates/main amd64 Packages [23.2 kB]
  
  
  
  Get:3 http://deb.debian.org jessie/main amd64 Packages [9063 kB]
  
  
  
  Get:4 http://security.debian.org jessie/updates InRelease [63.1 kB]
  
  
  
  Get:5 http://security.debian.org jessie/updates/main amd64 Packages [610 kB]
  
  
  
  100% [3 Packages 9063 kB]100% [3 Packages 9063 kB]Fetched 9904 kB in 1s (8178 kB/s)
  
  
  
  Reading package lists... 1%
  
  Reading package lists... 61%
  
  Reading package lists... Done
  
  
  
  
  
  Reading package lists... 1%
  
  Reading package lists... Done
  
  
  
  
  
  Building dependency tree... 0%
  
  Building dependency tree       
  
  
  
  
  
  Reading state information... Done
  
  
  
  Package ffmpeg is not available, but is referred to by another package.
  
  This may mean that the package is missing, has been obsoleted, or
  
  is only available from another source
  
  
  
  E: Package 'ffmpeg' has no installation candidate
  
  Exited with code 100

  How can I fix that ?

• Unable to instal ffmpeg on CircleCi 2.0

  15 novembre 2017, par Mateusz Urbański

  I have Ruby on Rails project which uses CircleCi to run tests. In the past I was using CircleCi 1.0 but now I migrated to CircleCi 2.0. I have problem with installing ffmpeg. CircleCi 2.0 uses Ubuntu 14.04. I install ffmpeg like this :

  # ffmpeg installation
  
  sudo add-apt-repository ppa:mc3man/trusty-media -y
  
  sudo apt-get update
  
  sudo apt-get install ffmpeg

  and my circle.yml config file looks like this :

  version: 2
  
  environment:
  
    TZ: "/usr/share/zoneinfo/America/Los_Angeles"
  
  
  
  jobs:
  
    build:
  
      parallelism: 2
  
      working_directory: ~/circleci-survey-builder
  
      docker:
  
        - image: circleci/ruby:2.4.1-node
  
          environment:
  
            PGHOST: 127.0.0.1
  
            PGUSER: ubuntu
  
            RAILS_ENV: test
  
        - image: circleci/postgres:9.6-alpine
  
          environment:
  
            POSTGRES_USER: ubuntu
  
            POSTGRES_DB: circle_test
  
            POSTGRES_PASSWORD: ''
  
      steps:
  
        - checkout
  
  
  
        - run:
  
            name: 'Install CircleCI dependencies'
  
            command: bash deploy/circle-dependencies.sh
  
  
  
        - type: cache-restore
  
          key: dashboard-{{ checksum "Gemfile.lock" }}
  
  
  
        - run:
  
            name: 'Install gems'
  
            command: bundle install --path vendor/bundle
  
  
  
        - type: cache-save
  
          key: dashboard-{{ checksum "Gemfile.lock" }}
  
          paths:
  
            - vendor/bundle
  
  
  
        - run:
  
            name: 'Install postgresql-client'
  
            command: sudo apt install postgresql-client
  
  
  
        - run:
  
            name: 'Create database.yml'
  
            command: mv config/database.ci.yml config/database.yml
  
  
  
        - run:
  
            name: Set up SurveyBuilder database
  
            command: bundle exec rake db:structure:load --trace
  
  
  
        - run:
  
            name: 'Run tests'
  
            command: |
  
              bundle exec rspec spec

  It returns following error when I run build on CircleCi :

  #!/bin/bash -eo pipefail
  
  bash deploy/circle-dependencies.sh
  
  
  
  Fetching: bundler-1.16.0.gem (100%)
  
  Successfully installed bundler-1.16.0
  
  1 gem installed
  
  sudo: add-apt-repository: command not found
  
  
  
  
  
  Ign http://deb.debian.org jessie InRelease
  
  
  
  Get:1 http://deb.debian.org jessie-updates InRelease [145 kB]
  
  
  
  10% [1 InRelease 13.8 kB/145 kB 10%] [Waiting for headers] [Connecting to securHit http://deb.debian.org jessie Release.gpg
  
  
  
  Hit http://deb.debian.org jessie Release
  
  
  
  Get:2 http://deb.debian.org jessie-updates/main amd64 Packages [23.2 kB]
  
  
  
  Get:3 http://deb.debian.org jessie/main amd64 Packages [9063 kB]
  
  
  
  Get:4 http://security.debian.org jessie/updates InRelease [63.1 kB]
  
  
  
  Get:5 http://security.debian.org jessie/updates/main amd64 Packages [610 kB]
  
  
  
  100% [3 Packages 9063 kB]100% [3 Packages 9063 kB]Fetched 9904 kB in 1s (8178 kB/s)
  
  
  
  Reading package lists... 1%
  
  Reading package lists... 61%
  
  Reading package lists... Done
  
  
  
  
  
  Reading package lists... 1%
  
  Reading package lists... Done
  
  
  
  
  
  Building dependency tree... 0%
  
  Building dependency tree       
  
  
  
  
  
  Reading state information... Done
  
  
  
  Package ffmpeg is not available, but is referred to by another package.
  
  This may mean that the package is missing, has been obsoleted, or
  
  is only available from another source
  
  
  
  E: Package 'ffmpeg' has no installation candidate
  
  Exited with code 100

  How can I fix that ?

• Video encoding task not working with Django Celery Redis FFMPEG and GraphQL

  18 juin 2023, par phanio

  I'm having a hard time trying to understand how is this FFMPEG encoding works while using Django, Celery, Redis, GraphQL and Docker too.

  


  I have this video / courses platform project and want I'm trying to do using FFMPEG, Celery and Redis is to create different video resolutions so I can display them the way Youtube does inside the videoplayer ( the videoplayer is handled in frontend by Nextjs and Apollo Client ), now on the backend I've just learned that in order to use properly the FFMPEG to resize the oridinal video size, I need to use Celery and Redis to perform asyncronus tasks. I've found a few older posts here on stackoverflow and google, but is not quite enough info for someone who is using the ffmpeg and clery and redis for the first time ( I've started already step by step and created that example that adds two numbers together with celery, that works well ). Now I'm not sure what is wrong with my code, because first of all I'm not really sure where should I trigger the task from, I mean from which file, because at the end of the task I want to send the data through api using GrapQL Strawberry.

  


  This is what I've tried by now :

  


  So first things first my project structure looks like this

  


  - backend #root directory
 --- backend
    -- __init__.py
    -- celery.py
    -- settings.py
    -- urls.py
      etc..

 --- static
   -- videos

 --- video
   -- models.py
   -- schema.py
   -- tasks.py
   -- types.py
   etc..

 --- .env

 --- db.sqlite3

 --- docker-compose.yml

 --- Dockerfile

 --- manage.py

 --- requirements.txt


  


  here is my settings.py file :

  


  from pathlib import Path
import os

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent

DEBUG = True

ALLOWED_HOSTS=["localhost", "0.0.0.0", "127.0.0.1"]

DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'


# Application definition

INSTALLED_APPS = [
    "corsheaders",
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    "strawberry.django",
    "video"
]

etc...

STATIC_URL = '/static/'
MEDIA_URL = '/videos/'

STATICFILES_DIRS = [
    BASE_DIR / 'static',
    # BASE_DIR / 'frontend/build/static',
]

MEDIA_ROOT = BASE_DIR / 'static/videos'

STATIC_ROOT = BASE_DIR / 'staticfiles'

STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'

CORS_ALLOW_ALL_ORIGINS = True


CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'

# REDIS CACHE
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": f"redis://127.0.0.1:6379/1",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
        },
    }
}

# Docker
CELERY_BROKER_URL = os.environ.get("CELERY_BROKER", "redis://redis:6379/0")
CELERY_RESULT_BACKEND = os.environ.get("CELERY_BROKER", "redis://redis:6379/0")


  


  This is my main urls.py file :

  


  from django.contrib import admin
from django.conf import settings
from django.conf.urls.static import static
from django.urls import path
from django.urls.conf import include
from strawberry.django.views import GraphQLView

from video.schema import schema

urlpatterns = [
    path('admin/', admin.site.urls),
    path("graphql", GraphQLView.as_view(schema=schema)),
]

if settings.DEBUG:
    urlpatterns += static(settings.MEDIA_URL,
                          document_root=settings.MEDIA_ROOT)
    urlpatterns += static(settings.STATIC_URL,
                          document_root=settings.STATIC_ROOT)


  


  This is my celery.py file :

  


  from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'backend.settings')

backend = Celery('backend')

backend.config_from_object('django.conf:settings', namespace="CELERY")

backend.autodiscover_tasks()

@backend.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))


  


  This is my init.py file :

  


  from .celery import backend as celery_backend

__all__ = ('celery_backend',)


  


  This is my Dockerfile :

  


  FROM python:3
ENV PYTHONUNBUFFERED=1

WORKDIR /usr/src/backend

RUN apt-get -y update
RUN apt-get -y upgrade
RUN apt-get install -y ffmpeg

COPY requirements.txt ./
RUN pip install -r requirements.txt


  


  This is my docker-compose.yml file :

  


  version: "3.8"

services:
  django:
    build: .
    container_name: django
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/usr/src/backend/
    ports:
      - "8000:8000"
    environment:
      - DEBUG=1
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
      - CELERY_BROKER=redis://redis:6379/0
      - CELERY_BACKEND=redis://redis:6379/0
    depends_on:
      - pgdb
      - redis

  celery:
    build: .
    command: celery -A backend worker -l INFO
    volumes:
      - .:/usr/src/backend
    depends_on:
      - django
      - redis

  pgdb:
    image: postgres
    container_name: pgdb
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    volumes:
      - pgdata:/var/lib/postgresql/data/

  redis:
    image: "redis:alpine"

volumes:
  pgdata:


  


  And now inside my video app folder :

  


  My models.py file :

  


  

  • here I've created separated fields for all resolution sizes, from video_file_2k to video_file_144, I was thinking that maybe after the process of the encoding this will populate those fields..

  


  


  from django.db import models
from django.urls import reverse


class Video(models.Model):
    video_id = models.AutoField(primary_key=True, editable=False)
    slug = models.SlugField(max_length=255)
    title = models.CharField(max_length=150, blank=True, null=True)
    description = models.TextField(blank=True, null=True)
    video_file = models.FileField(null=False, blank=False)
    video_file_2k = models.FileField(null=True, blank=True)
    video_file_fullhd = models.FileField(null=True, blank=True)
    video_file_hd = models.FileField(null=True, blank=True)
    video_file_480 = models.FileField(null=True, blank=True)
    video_file_360 = models.FileField(null=True, blank=True)
    video_file_240 = models.FileField(null=True, blank=True)
    video_file_144 = models.FileField(null=True, blank=True)
    category = models.CharField(max_length=64, blank=False, null=False)
    created_at = models.DateTimeField(
        ("Created at"), auto_now_add=True, editable=False)
    updated_at = models.DateTimeField(("Updated at"), auto_now=True)

    class Meta:
        ordering = ("-created_at",)
        verbose_name = ("Video")
        verbose_name_plural = ("Videos")

    def get_absolute_url(self):
        return reverse("store:video_detail", args=[self.slug])

    def __str__(self):
        return self.title


  


  This is my schema.py file :

  


  import strawberry
from strawberry.file_uploads import Upload
from typing import List
from .types import VideoType
from .models import Video
from .tasks import task_video_encoding_1080p, task_video_encoding_720p


@strawberry.type
class Query:
    @strawberry.field
    def videos(self, category: str = None) -> List[VideoType]:
        if category:
            videos = Video.objects.filter(category=category)
            return videos
        return Video.objects.all()

    @strawberry.field
    def video(self, slug: str) -> VideoType:
        if slug == slug:
            video = Video.objects.get(slug=slug)
            return video

    @strawberry.field
    def video_by_id(self, video_id: int) -> VideoType:
        if video_id == video_id:
            video = Video.objects.get(pk=video_id)

          # Here I've tried to trigger my tasks, when I visited 0.0.0.0:8000/graphql url
          # and I was querying for a video by it's id , then I've got the error from celery 
            task_video_encoding_1080p.delay(video_id)
            task_video_encoding_720p.delay(video_id)

            return video


@strawberry.type
class Mutation:
    @strawberry.field
    def create_video(self, slug: str, title: str, description: str, video_file: Upload, video_file_2k: str, video_file_fullhd: str, video_file_hd: str, video_file_480: str, video_file_360: str, video_file_240: str, video_file_144: str, category: str) -> VideoType:

        video = Video(slug=slug, title=title, description=description,
                      video_file=video_file, video_file_2k=video_file_2k, video_file_fullhd=video_file_fullhd, video_file_hd=video_file_hd, video_file_480=video_file_480, video_file_360=video_file_360, video_file_240=video_file_240, video_file_144=video_file_144,category=category)
        
        video.save()
        return video

    @strawberry.field
    def update_video(self, video_id: int, slug: str, title: str, description: str, video_file: str, category: str) -> VideoType:
        video = Video.objects.get(video_id=video_id)
        video.slug = slug
        video.title = title
        video.description = description
        video.video_file = video_file
        video.category = category
        video.save()
        return video

    @strawberry.field
    def delete_video(self, video_id: int) -> bool:
        video = Video.objects.get(video_id=video_id)
        video.delete
        return True


schema = strawberry.Schema(query=Query, mutation=Mutation)


  


  This is my types.py file ( strawberry graphql related ) :

  


  import strawberry

from .models import Video


@strawberry.django.type(Video)
class VideoType:
    video_id: int
    slug: str
    title: str
    description: str
    video_file: str
    video_file_2k: str
    video_file_fullhd: str
    video_file_hd: str
    video_file_480: str
    video_file_360: str
    video_file_240: str
    video_file_144: str
    category: str


  


  And this is my tasks.py file :

  


  from __future__ import absolute_import, unicode_literals
import os, subprocess
from django.conf import settings
from django.core.exceptions import ValidationError
from celery import shared_task
from celery.utils.log import get_task_logger
from .models import Video
FFMPEG_PATH = os.environ["IMAGEIO_FFMPEG_EXE"] = "/opt/homebrew/Cellar/ffmpeg/6.0/bin/ffmpeg"

logger = get_task_logger(__name__)


# CELERY TASKS
@shared_task
def add(x,y):
    return x + y


@shared_task
def task_video_encoding_720p(video_id):
    logger.info('Video Processing started')
    try:
        video = Video.objects.get(video_id=video_id)
        input_file_path = video.video_file.path
        input_file_url = video.video_file.url
        input_file_name = video.video_file.name

        # get the filename (without extension)
        filename = os.path.basename(input_file_url)

        # path to the new file, change it according to where you want to put it
        output_file_name = os.path.join('videos', 'mp4', '{}.mp4'.format(filename))
        output_file_path = os.path.join(settings.MEDIA_ROOT, output_file_name)

        # 2-pass encoding
        for i in range(1):
           new_video_720p = subprocess.call([FFMPEG_PATH, '-i', input_file_path, '-s', '1280x720', '-vcodec', 'mpeg4', '-acodec', 'libvo_aacenc', '-b', '10000k', '-pass', i, '-r', '30', output_file_path])
        #    new_video_720p = subprocess.call([FFMPEG_PATH, '-i', input_file_path, '-s', '{}x{}'.format(height * 16/9, height), '-vcodec', 'mpeg4', '-acodec', 'libvo_aacenc', '-b', '10000k', '-pass', i, '-r', '30', output_file_path])

        if new_video_720p == 0:
            # save the new file in the database
            # video.video_file_hd.name = output_file_name
            video.save(update_fields=['video_file_hd'])
            logger.info('Video Processing Finished')
            return video

        else:
            logger.info('Proceesing Failed.') # Just for now

    except:
        raise ValidationError('Something went wrong')


@shared_task
# def task_video_encoding_1080p(video_id, height):
def task_video_encoding_1080p(video_id):
    logger.info('Video Processing started')
    try:
        video = Video.objects.get(video_id=video_id)
        input_file_path = video.video_file.url
        input_file_name = video.video_file.name

        # get the filename (without extension)
        filename = os.path.basename(input_file_path)

        # path to the new file, change it according to where you want to put it
        output_file_name = os.path.join('videos', 'mp4', '{}.mp4'.format(filename))
        output_file_path = os.path.join(settings.MEDIA_ROOT, output_file_name)

        for i in range(1):
            new_video_1080p = subprocess.call([FFMPEG_PATH, '-i', input_file_path, '-s', '1920x1080', '-vcodec', 'mpeg4', '-acodec', 'libvo_aacenc', '-b', '10000k', '-pass', i, '-r', '30', output_file_path])

        if new_video_1080p == 0:
            # save the new file in the database
            # video.video_file_hd.name = output_file_name
            video.save(update_fields=['video_file_fullhd'])
            logger.info('Video Processing Finished')
            return video
        else:
            logger.info('Proceesing Failed.') # Just for now

    except:
        raise ValidationError('Something went wrong')


  


  In my first attempt I wasn't triggering the tasks no where, then I've tried to trigger the task from the schema.py file from graphql inside the video_by_id, but there I've got this error :

  


  backend-celery-1  | django.core.exceptions.ValidationError: ['Something went wrong']
backend-celery-1  | [2023-06-18 16:38:52,859: ERROR/ForkPoolWorker-4] Task video.tasks.task_video_encoding_1080p[d33b1a42-5914-467c-ad5c-00565bc8be6f] raised unexpected: ValidationError(['Something went wrong'])
backend-celery-1  | Traceback (most recent call last):
backend-celery-1  |   File "/usr/src/backend/video/tasks.py", line 81, in task_video_encoding_1080p
backend-celery-1  |     new_video_1080p = subprocess.call([FFMPEG_PATH, '-i', input_file_path, '-s', '1920x1080', '-vcodec', 'mpeg4', '-acodec', 'libvo_aacenc', '-b', '10000k', '-pass', i, '-r', '30', output_file_path])
backend-celery-1  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-celery-1  |   File "/usr/local/lib/python3.11/subprocess.py", line 389, in call
backend-celery-1  |     with Popen(*popenargs, **kwargs) as p:
backend-celery-1  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-celery-1  |   File "/usr/local/lib/python3.11/subprocess.py", line 1026, in __init__
backend-celery-1  |     self._execute_child(args, executable, preexec_fn, close_fds,
backend-celery-1  |   File "/usr/local/lib/python3.11/subprocess.py", line 1883, in _execute_child
backend-celery-1  |     self.pid = _fork_exec(
backend-celery-1  |                ^^^^^^^^^^^
backend-celery-1  | TypeError: expected str, bytes or os.PathLike object, not int
backend-celery-1  | 
backend-celery-1  | During handling of the above exception, another exception occurred:
backend-celery-1  | 
backend-celery-1  | Traceback (most recent call last):
backend-celery-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
backend-celery-1  |     R = retval = fun(*args, **kwargs)
backend-celery-1  |                  ^^^^^^^^^^^^^^^^^^^^
backend-celery-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
backend-celery-1  |     return self.run(*args, **kwargs)
backend-celery-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
backend-celery-1  |   File "/usr/src/backend/video/tasks.py", line 93, in task_video_encoding_1080p
backend-celery-1  |     raise ValidationError('Something went wrong')
backend-celery-1  | django.core.exceptions.ValidationError: ['Something went wrong']


  


  If anyone has done this kind of project or something like this please any suggestion or help is much appreciated.

  


  Thank you in advance !