Recherche avancée

Sur d’autres sites (5786)

• Revision 33976 : bugs html

  27 décembre 2009, par cedric@… — Log

  bugs html

• AWS Lambda subprocess OSError : [Errno 2] No such file or directory

  11 septembre 2016, par Lev

  I’m trying to create a lambda function that makes collection of thumbnails from a video on amazon s3 using ffmpeg. ffmpeg binary is included into fuction package.

  function code :

  # -*- coding: utf-8 -*-
  
  
  
  import stat
  
  import shutil
  
  import boto3
  
  import logging
  
  import subprocess as sp
  
  import os
  
  import threading
  
  
  
  thumbnail_prefix = 'thumb_'
  
  thumbnail_ext = '.jpg'
  
  time_delta = 1
  
  video_frames_path = 'media/videos/frames'
  
  
  
  print('Loading function')
  
  logger = logging.getLogger()
  
  logger.setLevel(logging.INFO)
  
  
  
  lambda_tmp_dir = '/tmp'  # Lambda fuction can use this directory.
  
  
  
  # ffmpeg is stored with this script.
  
  # When executing ffmpeg, execute permission is requierd.
  
  # But Lambda source directory do not have permission to change it.
  
  # So move ffmpeg binary to `/tmp` and add permission.
  
  ffmpeg_bin = "{0}/ffmpeg.linux64".format(lambda_tmp_dir)
  
  shutil.copyfile('/var/task/ffmpeg.linux64', ffmpeg_bin)
  
  
  
  os.chmod(ffmpeg_bin, 777)
  
  
  
  # tried also:
  
  # os.chmod(ffmpeg_bin, os.stat(ffmpeg_bin).st_mode | stat.S_IEXEC)
  
  
  
  s3 = boto3.client('s3')
  
  
  
  
  
  def get_thumb_filename(num):
  
      return '{prefix}{num:03d}{ext}'.format(prefix=thumbnail_prefix, num=num, ext=thumbnail_ext)
  
  
  
  
  
  def create_thumbnails(video_url):
  
      i = 1
  
      filenames_list = []
  
      filename = None
  
      while i == 1 or os.path.isfile(os.path.join(os.getcwd(), get_thumb_filename(i-1))):
  
          if filename:
  
              filenames_list.append(filename)
  
          time = time_delta * (i - 1)
  
          filename = get_thumb_filename(i)
  
          print(ffmpeg_bin)
  
          if os.path.isfile(ffmpeg_bin):
  
              print('ok')
  
          sp.call(['sudo',
  
                   ffmpeg_bin,
  
                   '-ss',
  
                   str(time),
  
                   '-i',
  
                   video_url,
  
                   '-frames:v',
  
                   '1',
  
                   get_thumb_filename(i)])
  
          i += 1
  
      print(filenames_list)
  
      return filenames_list
  
  
  
  
  
  def s3_upload_file(file_path, key, bucket, acl, content_type):
  
      file = open(file_path, 'r')
  
      s3.put_object(
  
          Bucket=bucket,
  
          ACL=acl,
  
          Body=file,
  
          Key=key,
  
          ContentType=content_type
  
      )
  
      logger.info("file {0} moved to {1}/{2}".format(file_path, bucket, key))
  
  
  
  
  
  def s3_upload_files_in_threads(filenames_list, dir_path, bucket, s3path, acl, content_type):
  
      for filename in filenames_list:
  
          if os.path.isfile(os.path.join(dir_path, filename)):
  
              print(os.path.join(dir_path, filename))
  
          t = threading.Thread(target=s3_upload_file,
  
                               args=(os.path.join(dir_path, filename),
  
                                     '{0}/{1}'.format(s3path, filename),
  
                                     bucket,
  
                                     acl,
  
                                     content_type)).start()
  
  
  
  
  
  def lambda_handler(event, context):
  
      bucket = event['Records'][0]['s3']['bucket']['name']
  
      video_key = event['Records'][0]['s3']['object']['key']
  
      video_name = video_key.split('/')[-1].split('.')[0]
  
      video_url = 'http://{0}/{1}'.format(bucket, video_key)
  
      filenames_list = create_thumbnails(video_url)
  
      s3_upload_files_in_threads(filenames_list,
  
                                 os.getcwd(),
  
                                 bucket,
  
                                 '{0}/{1}'.format(video_frames_path, video_name),
  
                                 'public-read',
  
                                 'image/jpeg')
  
      return

  during the execution I get following logs :

  Loading function
  
  
  
  /tmp/ffmpeg.linux64
  
  
  
  ok
  
  
  
  [Errno 2] No such file or directory: OSError
  
  Traceback (most recent call last):
  
   File "/var/task/lambda_function.py", line 112, in lambda_handler
  
   filenames_list = create_thumbnails(video_url)
  
   File "/var/task/lambda_function.py", line 77, in create_thumbnails
  
   get_thumb_filename(i)])
  
   File "/usr/lib64/python2.7/subprocess.py", line 522, in call
  
   return Popen(*popenargs, **kwargs).wait()
  
   File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__
  
   errread, errwrite)
  
   File "/usr/lib64/python2.7/subprocess.py", line 1335, in _execute_child
  
   raise child_exception
  
  OSError: [Errno 2] No such file or directory

  When I use the same sp.call() with the same ffmpeg binary on my ec2 instance it works fine.

• How to use (django-celery,RQ) worker to execute a video filetype conversion (ffmpeg) in django on heroku (My code works locally)

  15 janvier 2013, par GetItDone

  One part of my website includes a form that allows users to upload video. I use ffmpeg to convert the video to flv. My media and static files are stored on Amazon S3. I can get everything to work perfectly locally, however I can't seem to figure out how to use a worker to run the video conversion subprocess in production. I have dj-celery and rq installed in my app. The code in my view that I was able to get to work locally is :

  #views.py
  
  def upload_broadcast(request):
  
      if request.method == 'POST':
  
          form = VideoUploadForm(request.POST, request.FILES)
  
          if form.is_valid():
  
              new_video=form.save()
  
              def convert_to_flv(video):
  
                  filename = video.video_upload
  
                  sourcefile = "%s%s" % (settings.MEDIA_ROOT, filename)
  
                  flvfilename = "%s.flv" % video.id
  
                  imagefilename = "%s.png" % video.id
  
                  thumbnailfilename = "%svideos/flv/%s" % (settings.MEDIA_ROOT, imagefilename)
  
                  targetfile = "%svideos/flv/%s" % (settings.MEDIA_ROOT, flvfilename)
  
                  ffmpeg = "ffmpeg -i %s -acodec mp3 -ar 22050 -f flv -s 320x240 %s" % (sourcefile, targetfile)
  
                  grabimage = "ffmpeg -y -i %s -vframes 1 -ss 00:00:02 -an -vcodec png -f rawvideo -s 320x240 %s" % (sourcefile, thumbnailfilename)
  
                  print ("SOURCE: %s" % sourcefile)
  
                  print ("TARGET: %s" % targetfile)
  
                  print ("TARGET IMAGE: %s" % thumbnailfilename)
  
                  print ("FFMPEG TASK CODE: %s" % ffmpeg)
  
                  print ("IMAGE TASK CODE: %s" % grabimage)
  
                  try:
  
                      ffmpegresult = subprocess.call(ffmpeg)
  
                      print "---------------FFMPEG---------------"
  
                      print ffmpegresult
  
                  except:
  
                      print "Not working."
  
                  try:
  
                      videothumbnail = subprocess.call(grabimage)
  
                      print "---------------IMAGE---------------"
  
                      print videothumbnail
  
                  except:
  
                      print "Not working."
  
                  video.flvfilename = flvfilename
  
                  video.videothumbnail = imagefilename
  
                  video.save()
  
  
  
              convert_to_flv(new_video)
  
  
  
              return HttpResponseRedirect('/video_list/')
  
      else:
  
  ...

  This is my first time trying to use a worker (or ever pushing a project to production), so even with the documentation it is still unclear to me what I need to do. I have tried several different things but nothing seems to work. Is there just a simple way to tell celery to run the ffmpegresult = subprocess.call(ffmpeg) ? Thanks in advance for any help or insight.

  EDIT- Added heroku logs

  2013-01-10T20:58:57+00:00 app[web.1]: TARGET: /media/videos/flv/8.flv
  
  2013-01-10T20:58:57+00:00 app[web.1]: IMAGE TASK CODE: ffmpeg -y -i /media/videos/practice.wmv -vframes 1 -ss 00:00:02 - an -vcodec png -f rawvideo -s 320x240 /media/videos/flv/8.png
  
  2013-01-10T20:58:57+00:00 app[web.1]: SOURCE: /media/videos/practice.wmv
  
  2013-01-10T20:58:57+00:00 app[web.1]: FFMPEG TASK CODE: ffmpeg -i /media/videos/practice.wmv -acodec mp3 -ar 22050 -f fl v -s 320x240 /media/videos/flv/8.flv
  
  2013-01-10T20:58:57+00:00 app[web.1]: TARGET IMAGE: /media/videos/flv/8.png
  
  2013-01-10T20:58:57+00:00 app[web.1]: Not working.
  
  2013-01-10T20:58:57+00:00 app[web.1]: Not working.

  NEWER EDIT

  I tried adding a tasks.py and added the task :

  celery = Celery('tasks', broker='redis://guest@localhost//')
  
  
  
  @celery.task
  
  def ffmpeg_task(video):
  
      converted_file = subprocess.call(video)
  
      return converted_file

  then I changed the relevant section of my view to :

  ...
  
  try:
  
      ffmpeg_task.delay(ffmpeg)
  
      print "---------------FFMPEG---------------"
  
      print ffmpegresult
  
  except:
  
      print "Not working."
  
  ...

  My new logs are :

  2013-01-15T13:19:52+00:00 app[web.1]: TARGET IMAGE: /media/videos/flv/12.png
  
  2013-01-15T13:19:52+00:00 app[web.1]: SOURCE: /media/videos/practice.wmv
  
  2013-01-15T13:19:52+00:00 app[web.1]: FFMPEG TASK CODE: ffmpeg -i /media/videos/practice.wmv -acodec mp3 -ar 22050 -f fl v -s 320x240 /media/videos/flv/12.flv
  
  2013-01-15T13:19:52+00:00 app[web.1]: IMAGE TASK CODE: ffmpeg -y -i /media/videos/practice.wmv -vframes 1 -ss 00:00:02 -an -vcodec png -f rawvideo -s 320x240 /media/videos/flv/12.png
  
  2013-01-15T13:19:52+00:00 app[web.1]: TARGET: /media/videos/flv/12.flv
  
  2013-01-15T13:20:17+00:00 app[web.1]: 2013-01-15 13:20:17 [2] [CRITICAL] WORKER TIMEOUT (pid:12)
  
  2013-01-15T13:20:17+00:00 app[web.1]: 2013-01-15 13:20:17 [2] [CRITICAL] WORKER TIMEOUT (pid:12)
  
  2013-01-15T13:20:17+00:00 app[web.1]: 2013-01-15 13:20:17 [19] [INFO] Booting worker with pid: 19

  Am I completely missing something ? I'll keep trying, but will be very appreciative of any direction or assistance.