Recherche avancée

Sur d’autres sites (14530)

• AWS Lambda subprocess OSError : [Errno 2] No such file or directory

  11 septembre 2016, par Lev

  I’m trying to create a lambda function that makes collection of thumbnails from a video on amazon s3 using ffmpeg. ffmpeg binary is included into fuction package.

  function code :

  # -*- coding: utf-8 -*-
  
  
  
  import stat
  
  import shutil
  
  import boto3
  
  import logging
  
  import subprocess as sp
  
  import os
  
  import threading
  
  
  
  thumbnail_prefix = 'thumb_'
  
  thumbnail_ext = '.jpg'
  
  time_delta = 1
  
  video_frames_path = 'media/videos/frames'
  
  
  
  print('Loading function')
  
  logger = logging.getLogger()
  
  logger.setLevel(logging.INFO)
  
  
  
  lambda_tmp_dir = '/tmp'  # Lambda fuction can use this directory.
  
  
  
  # ffmpeg is stored with this script.
  
  # When executing ffmpeg, execute permission is requierd.
  
  # But Lambda source directory do not have permission to change it.
  
  # So move ffmpeg binary to `/tmp` and add permission.
  
  ffmpeg_bin = "{0}/ffmpeg.linux64".format(lambda_tmp_dir)
  
  shutil.copyfile('/var/task/ffmpeg.linux64', ffmpeg_bin)
  
  
  
  os.chmod(ffmpeg_bin, 777)
  
  
  
  # tried also:
  
  # os.chmod(ffmpeg_bin, os.stat(ffmpeg_bin).st_mode | stat.S_IEXEC)
  
  
  
  s3 = boto3.client('s3')
  
  
  
  
  
  def get_thumb_filename(num):
  
      return '{prefix}{num:03d}{ext}'.format(prefix=thumbnail_prefix, num=num, ext=thumbnail_ext)
  
  
  
  
  
  def create_thumbnails(video_url):
  
      i = 1
  
      filenames_list = []
  
      filename = None
  
      while i == 1 or os.path.isfile(os.path.join(os.getcwd(), get_thumb_filename(i-1))):
  
          if filename:
  
              filenames_list.append(filename)
  
          time = time_delta * (i - 1)
  
          filename = get_thumb_filename(i)
  
          print(ffmpeg_bin)
  
          if os.path.isfile(ffmpeg_bin):
  
              print('ok')
  
          sp.call(['sudo',
  
                   ffmpeg_bin,
  
                   '-ss',
  
                   str(time),
  
                   '-i',
  
                   video_url,
  
                   '-frames:v',
  
                   '1',
  
                   get_thumb_filename(i)])
  
          i += 1
  
      print(filenames_list)
  
      return filenames_list
  
  
  
  
  
  def s3_upload_file(file_path, key, bucket, acl, content_type):
  
      file = open(file_path, 'r')
  
      s3.put_object(
  
          Bucket=bucket,
  
          ACL=acl,
  
          Body=file,
  
          Key=key,
  
          ContentType=content_type
  
      )
  
      logger.info("file {0} moved to {1}/{2}".format(file_path, bucket, key))
  
  
  
  
  
  def s3_upload_files_in_threads(filenames_list, dir_path, bucket, s3path, acl, content_type):
  
      for filename in filenames_list:
  
          if os.path.isfile(os.path.join(dir_path, filename)):
  
              print(os.path.join(dir_path, filename))
  
          t = threading.Thread(target=s3_upload_file,
  
                               args=(os.path.join(dir_path, filename),
  
                                     '{0}/{1}'.format(s3path, filename),
  
                                     bucket,
  
                                     acl,
  
                                     content_type)).start()
  
  
  
  
  
  def lambda_handler(event, context):
  
      bucket = event['Records'][0]['s3']['bucket']['name']
  
      video_key = event['Records'][0]['s3']['object']['key']
  
      video_name = video_key.split('/')[-1].split('.')[0]
  
      video_url = 'http://{0}/{1}'.format(bucket, video_key)
  
      filenames_list = create_thumbnails(video_url)
  
      s3_upload_files_in_threads(filenames_list,
  
                                 os.getcwd(),
  
                                 bucket,
  
                                 '{0}/{1}'.format(video_frames_path, video_name),
  
                                 'public-read',
  
                                 'image/jpeg')
  
      return

  during the execution I get following logs :

  Loading function
  
  
  
  /tmp/ffmpeg.linux64
  
  
  
  ok
  
  
  
  [Errno 2] No such file or directory: OSError
  
  Traceback (most recent call last):
  
   File "/var/task/lambda_function.py", line 112, in lambda_handler
  
   filenames_list = create_thumbnails(video_url)
  
   File "/var/task/lambda_function.py", line 77, in create_thumbnails
  
   get_thumb_filename(i)])
  
   File "/usr/lib64/python2.7/subprocess.py", line 522, in call
  
   return Popen(*popenargs, **kwargs).wait()
  
   File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__
  
   errread, errwrite)
  
   File "/usr/lib64/python2.7/subprocess.py", line 1335, in _execute_child
  
   raise child_exception
  
  OSError: [Errno 2] No such file or directory

  When I use the same sp.call() with the same ffmpeg binary on my ec2 instance it works fine.

• How to use (django-celery,RQ) worker to execute a video filetype conversion (ffmpeg) in django on heroku (My code works locally)

  15 janvier 2013, par GetItDone

  One part of my website includes a form that allows users to upload video. I use ffmpeg to convert the video to flv. My media and static files are stored on Amazon S3. I can get everything to work perfectly locally, however I can't seem to figure out how to use a worker to run the video conversion subprocess in production. I have dj-celery and rq installed in my app. The code in my view that I was able to get to work locally is :

  #views.py
  
  def upload_broadcast(request):
  
      if request.method == 'POST':
  
          form = VideoUploadForm(request.POST, request.FILES)
  
          if form.is_valid():
  
              new_video=form.save()
  
              def convert_to_flv(video):
  
                  filename = video.video_upload
  
                  sourcefile = "%s%s" % (settings.MEDIA_ROOT, filename)
  
                  flvfilename = "%s.flv" % video.id
  
                  imagefilename = "%s.png" % video.id
  
                  thumbnailfilename = "%svideos/flv/%s" % (settings.MEDIA_ROOT, imagefilename)
  
                  targetfile = "%svideos/flv/%s" % (settings.MEDIA_ROOT, flvfilename)
  
                  ffmpeg = "ffmpeg -i %s -acodec mp3 -ar 22050 -f flv -s 320x240 %s" % (sourcefile, targetfile)
  
                  grabimage = "ffmpeg -y -i %s -vframes 1 -ss 00:00:02 -an -vcodec png -f rawvideo -s 320x240 %s" % (sourcefile, thumbnailfilename)
  
                  print ("SOURCE: %s" % sourcefile)
  
                  print ("TARGET: %s" % targetfile)
  
                  print ("TARGET IMAGE: %s" % thumbnailfilename)
  
                  print ("FFMPEG TASK CODE: %s" % ffmpeg)
  
                  print ("IMAGE TASK CODE: %s" % grabimage)
  
                  try:
  
                      ffmpegresult = subprocess.call(ffmpeg)
  
                      print "---------------FFMPEG---------------"
  
                      print ffmpegresult
  
                  except:
  
                      print "Not working."
  
                  try:
  
                      videothumbnail = subprocess.call(grabimage)
  
                      print "---------------IMAGE---------------"
  
                      print videothumbnail
  
                  except:
  
                      print "Not working."
  
                  video.flvfilename = flvfilename
  
                  video.videothumbnail = imagefilename
  
                  video.save()
  
  
  
              convert_to_flv(new_video)
  
  
  
              return HttpResponseRedirect('/video_list/')
  
      else:
  
  ...

  This is my first time trying to use a worker (or ever pushing a project to production), so even with the documentation it is still unclear to me what I need to do. I have tried several different things but nothing seems to work. Is there just a simple way to tell celery to run the ffmpegresult = subprocess.call(ffmpeg) ? Thanks in advance for any help or insight.

  EDIT- Added heroku logs

  2013-01-10T20:58:57+00:00 app[web.1]: TARGET: /media/videos/flv/8.flv
  
  2013-01-10T20:58:57+00:00 app[web.1]: IMAGE TASK CODE: ffmpeg -y -i /media/videos/practice.wmv -vframes 1 -ss 00:00:02 - an -vcodec png -f rawvideo -s 320x240 /media/videos/flv/8.png
  
  2013-01-10T20:58:57+00:00 app[web.1]: SOURCE: /media/videos/practice.wmv
  
  2013-01-10T20:58:57+00:00 app[web.1]: FFMPEG TASK CODE: ffmpeg -i /media/videos/practice.wmv -acodec mp3 -ar 22050 -f fl v -s 320x240 /media/videos/flv/8.flv
  
  2013-01-10T20:58:57+00:00 app[web.1]: TARGET IMAGE: /media/videos/flv/8.png
  
  2013-01-10T20:58:57+00:00 app[web.1]: Not working.
  
  2013-01-10T20:58:57+00:00 app[web.1]: Not working.

  NEWER EDIT

  I tried adding a tasks.py and added the task :

  celery = Celery('tasks', broker='redis://guest@localhost//')
  
  
  
  @celery.task
  
  def ffmpeg_task(video):
  
      converted_file = subprocess.call(video)
  
      return converted_file

  then I changed the relevant section of my view to :

  ...
  
  try:
  
      ffmpeg_task.delay(ffmpeg)
  
      print "---------------FFMPEG---------------"
  
      print ffmpegresult
  
  except:
  
      print "Not working."
  
  ...

  My new logs are :

  2013-01-15T13:19:52+00:00 app[web.1]: TARGET IMAGE: /media/videos/flv/12.png
  
  2013-01-15T13:19:52+00:00 app[web.1]: SOURCE: /media/videos/practice.wmv
  
  2013-01-15T13:19:52+00:00 app[web.1]: FFMPEG TASK CODE: ffmpeg -i /media/videos/practice.wmv -acodec mp3 -ar 22050 -f fl v -s 320x240 /media/videos/flv/12.flv
  
  2013-01-15T13:19:52+00:00 app[web.1]: IMAGE TASK CODE: ffmpeg -y -i /media/videos/practice.wmv -vframes 1 -ss 00:00:02 -an -vcodec png -f rawvideo -s 320x240 /media/videos/flv/12.png
  
  2013-01-15T13:19:52+00:00 app[web.1]: TARGET: /media/videos/flv/12.flv
  
  2013-01-15T13:20:17+00:00 app[web.1]: 2013-01-15 13:20:17 [2] [CRITICAL] WORKER TIMEOUT (pid:12)
  
  2013-01-15T13:20:17+00:00 app[web.1]: 2013-01-15 13:20:17 [2] [CRITICAL] WORKER TIMEOUT (pid:12)
  
  2013-01-15T13:20:17+00:00 app[web.1]: 2013-01-15 13:20:17 [19] [INFO] Booting worker with pid: 19

  Am I completely missing something ? I'll keep trying, but will be very appreciative of any direction or assistance.

• Inside WebM Technology : The VP8 Alternate Reference Frame

  15 juin 2010, par noreply@blogger.com (John Luther) — inside webm, vp8

  Since the WebM project was open-sourced just a week ago, we’ve seen blog posts and articles about its capabilities. As an open project, we welcome technical scrutiny and contributions that improve the codec. We know from our extensive testing that VP8 can match or exceed other leading codecs, but to get the best results, it helps to understand more about how the codec works. In this first of a series of blog posts, I’ll explain some of the fundamental techniques in VP8, along with examples and metrics.

  The alternative reference frame is one of the most exciting quality innovations in VP8. Let’s delve into how VP8 uses these frames to improve prediction and thereby overall video quality.

  Alternate Reference Frames in VP8

  VP8 uses three types of reference frames for inter prediction : the last frame, a "golden" frame (one frame worth of decompressed data from the arbitrarily distant past) and an alternate reference frame. Overall, this design has a much smaller memory footprint on both encoders and decoders than designs with many more reference frames. In video compression, it is very rare for more than three reference frames to provide significant quality benefit, but the undesirable increase in memory footprint from the extra frames is substantial.

  Unlike other types of reference frames used in video compression, which are displayed to the user by the decoder, the VP8 alternate reference frame is decoded normally but is never shown to the user. It is used solely as a reference to improve inter prediction for other coded frames. Because alternate reference frames are not displayed, VP8 encoders can use them to transmit any data that are helpful to compression. For example, a VP8 encoder can construct one alternate reference frame from multiple source frames, or it can create an alternate reference frame using different macroblocks from hundreds of different video frames.

  The current VP8 implementation enables two different types of usage for the alternate reference frame : noise-reduced prediction and past/future directional prediction.

  Noise-Reduced Prediction

  The alternate reference frame is transmitted and decoded similar to other frames, hence its usage does not add extra computation in decoding. The VP8 encoder however is free to use more sophisticated processing to create them in off-line encoding. One application of the alternate reference frame is for noise-reduced prediction. In this application, the VP8 encoder uses multiple input source frames to construct one reference frame through temporal or spatial noise filtering. This "noise-free" alternate reference frame is then used to improve prediction for encoding subsequent frames.

  You can make use of this feature by setting ARNR parameters in VP8 encoding, where ARNR stands for "Alternate Reference Noise Reduction." A sample two-pass encoding setting with the parameters :

  --arnr-maxframes=5 --arnr-strength=3

  enables the encoder to use "5" consecutive input source frames to produce one alternate reference frame using a filtering strength of "3". Here is an example showing the quality benefit of using this experimental "ARNR" feature on the standard test clip "Hall Monitor." (Each line on the graph represents the quality of an encoded stream on a given clip at multiple datarates. The higher points on the Y axis (PSNR) indicates the stream with the better quality.)

  
  

  The only difference between the two curves in the graph is that VP8_ARNR was produced by encodings with ARNR parameters and VP8_NO_ARNR was not. As we can see from the graph, noise reduced prediction is very helpful to compression quality when encoding noisy sources. We’ve just started to explore this idea but have already seen strong improvements on noisy input clips similar to this "Hall Monitor." We feel there’s a lot more we can do in this area.

  Improving Prediction without B Frames

  The lack of B frames in VP8 has sparked some discussion about its ability to achieve competitive compression efficiency. VP8 encoders, however, can make intelligent use of the golden reference and the alternate reference frames to compensate for this. The VP8 encoder can choose to transmit an alternate reference frame similar to a "future" frame, and encoding of subsequent frames can make use of information from the past (last frame and golden frame) and from the future (alternate reference frame). Effectively, this helps the encoder to achieve results similar to bidirectional (B frame) prediction without requiring frame reordering in the decoder. Running in two-pass encoding mode, compression can be improved in the VP8 encoder by using encoding parameters that enable lagged encoding and automatic placement of alternate reference frames :

  --auto-alt-ref=1 --lag-in-frames=16

  Used this way, the VP8 encoder can achieve improved prediction and compression efficiency without increasing the decoder’s complexity :

  
  

  In the video compression community, "Mobile and calendar" is known as a clip that benefits significantly from the usage of B frames. The graph above illustrates that the use of alternate reference frame benefits VP8 significantly without using B frames.

  Keep an eye on this blog for more posts about VP8 encoding. You can find more information on above encoding parameters or other detailed instructions to use with our VP8 encoders on our site, or join our discussion list.

  Yaowu Xu, Ph.D. is a codec engineer at Google.