Sur d’autres sites (5654)

• AWS Lambda subprocess OSError : [Errno 2] No such file or directory

  11 septembre 2016, par Lev

  I’m trying to create a lambda function that makes collection of thumbnails from a video on amazon s3 using ffmpeg. ffmpeg binary is included into fuction package.

  function code :

  # -*- coding: utf-8 -*-
  
  
  
  import stat
  
  import shutil
  
  import boto3
  
  import logging
  
  import subprocess as sp
  
  import os
  
  import threading
  
  
  
  thumbnail_prefix = 'thumb_'
  
  thumbnail_ext = '.jpg'
  
  time_delta = 1
  
  video_frames_path = 'media/videos/frames'
  
  
  
  print('Loading function')
  
  logger = logging.getLogger()
  
  logger.setLevel(logging.INFO)
  
  
  
  lambda_tmp_dir = '/tmp'  # Lambda fuction can use this directory.
  
  
  
  # ffmpeg is stored with this script.
  
  # When executing ffmpeg, execute permission is requierd.
  
  # But Lambda source directory do not have permission to change it.
  
  # So move ffmpeg binary to `/tmp` and add permission.
  
  ffmpeg_bin = "{0}/ffmpeg.linux64".format(lambda_tmp_dir)
  
  shutil.copyfile('/var/task/ffmpeg.linux64', ffmpeg_bin)
  
  
  
  os.chmod(ffmpeg_bin, 777)
  
  
  
  # tried also:
  
  # os.chmod(ffmpeg_bin, os.stat(ffmpeg_bin).st_mode | stat.S_IEXEC)
  
  
  
  s3 = boto3.client('s3')
  
  
  
  
  
  def get_thumb_filename(num):
  
      return '{prefix}{num:03d}{ext}'.format(prefix=thumbnail_prefix, num=num, ext=thumbnail_ext)
  
  
  
  
  
  def create_thumbnails(video_url):
  
      i = 1
  
      filenames_list = []
  
      filename = None
  
      while i == 1 or os.path.isfile(os.path.join(os.getcwd(), get_thumb_filename(i-1))):
  
          if filename:
  
              filenames_list.append(filename)
  
          time = time_delta * (i - 1)
  
          filename = get_thumb_filename(i)
  
          print(ffmpeg_bin)
  
          if os.path.isfile(ffmpeg_bin):
  
              print('ok')
  
          sp.call(['sudo',
  
                   ffmpeg_bin,
  
                   '-ss',
  
                   str(time),
  
                   '-i',
  
                   video_url,
  
                   '-frames:v',
  
                   '1',
  
                   get_thumb_filename(i)])
  
          i += 1
  
      print(filenames_list)
  
      return filenames_list
  
  
  
  
  
  def s3_upload_file(file_path, key, bucket, acl, content_type):
  
      file = open(file_path, 'r')
  
      s3.put_object(
  
          Bucket=bucket,
  
          ACL=acl,
  
          Body=file,
  
          Key=key,
  
          ContentType=content_type
  
      )
  
      logger.info("file {0} moved to {1}/{2}".format(file_path, bucket, key))
  
  
  
  
  
  def s3_upload_files_in_threads(filenames_list, dir_path, bucket, s3path, acl, content_type):
  
      for filename in filenames_list:
  
          if os.path.isfile(os.path.join(dir_path, filename)):
  
              print(os.path.join(dir_path, filename))
  
          t = threading.Thread(target=s3_upload_file,
  
                               args=(os.path.join(dir_path, filename),
  
                                     '{0}/{1}'.format(s3path, filename),
  
                                     bucket,
  
                                     acl,
  
                                     content_type)).start()
  
  
  
  
  
  def lambda_handler(event, context):
  
      bucket = event['Records'][0]['s3']['bucket']['name']
  
      video_key = event['Records'][0]['s3']['object']['key']
  
      video_name = video_key.split('/')[-1].split('.')[0]
  
      video_url = 'http://{0}/{1}'.format(bucket, video_key)
  
      filenames_list = create_thumbnails(video_url)
  
      s3_upload_files_in_threads(filenames_list,
  
                                 os.getcwd(),
  
                                 bucket,
  
                                 '{0}/{1}'.format(video_frames_path, video_name),
  
                                 'public-read',
  
                                 'image/jpeg')
  
      return

  during the execution I get following logs :

  Loading function
  
  
  
  /tmp/ffmpeg.linux64
  
  
  
  ok
  
  
  
  [Errno 2] No such file or directory: OSError
  
  Traceback (most recent call last):
  
   File "/var/task/lambda_function.py", line 112, in lambda_handler
  
   filenames_list = create_thumbnails(video_url)
  
   File "/var/task/lambda_function.py", line 77, in create_thumbnails
  
   get_thumb_filename(i)])
  
   File "/usr/lib64/python2.7/subprocess.py", line 522, in call
  
   return Popen(*popenargs, **kwargs).wait()
  
   File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__
  
   errread, errwrite)
  
   File "/usr/lib64/python2.7/subprocess.py", line 1335, in _execute_child
  
   raise child_exception
  
  OSError: [Errno 2] No such file or directory

  When I use the same sp.call() with the same ffmpeg binary on my ec2 instance it works fine.

• How to use (django-celery,RQ) worker to execute a video filetype conversion (ffmpeg) in django on heroku (My code works locally)

  15 janvier 2013, par GetItDone

  One part of my website includes a form that allows users to upload video. I use ffmpeg to convert the video to flv. My media and static files are stored on Amazon S3. I can get everything to work perfectly locally, however I can't seem to figure out how to use a worker to run the video conversion subprocess in production. I have dj-celery and rq installed in my app. The code in my view that I was able to get to work locally is :

  #views.py
  
  def upload_broadcast(request):
  
      if request.method == 'POST':
  
          form = VideoUploadForm(request.POST, request.FILES)
  
          if form.is_valid():
  
              new_video=form.save()
  
              def convert_to_flv(video):
  
                  filename = video.video_upload
  
                  sourcefile = "%s%s" % (settings.MEDIA_ROOT, filename)
  
                  flvfilename = "%s.flv" % video.id
  
                  imagefilename = "%s.png" % video.id
  
                  thumbnailfilename = "%svideos/flv/%s" % (settings.MEDIA_ROOT, imagefilename)
  
                  targetfile = "%svideos/flv/%s" % (settings.MEDIA_ROOT, flvfilename)
  
                  ffmpeg = "ffmpeg -i %s -acodec mp3 -ar 22050 -f flv -s 320x240 %s" % (sourcefile, targetfile)
  
                  grabimage = "ffmpeg -y -i %s -vframes 1 -ss 00:00:02 -an -vcodec png -f rawvideo -s 320x240 %s" % (sourcefile, thumbnailfilename)
  
                  print ("SOURCE: %s" % sourcefile)
  
                  print ("TARGET: %s" % targetfile)
  
                  print ("TARGET IMAGE: %s" % thumbnailfilename)
  
                  print ("FFMPEG TASK CODE: %s" % ffmpeg)
  
                  print ("IMAGE TASK CODE: %s" % grabimage)
  
                  try:
  
                      ffmpegresult = subprocess.call(ffmpeg)
  
                      print "---------------FFMPEG---------------"
  
                      print ffmpegresult
  
                  except:
  
                      print "Not working."
  
                  try:
  
                      videothumbnail = subprocess.call(grabimage)
  
                      print "---------------IMAGE---------------"
  
                      print videothumbnail
  
                  except:
  
                      print "Not working."
  
                  video.flvfilename = flvfilename
  
                  video.videothumbnail = imagefilename
  
                  video.save()
  
  
  
              convert_to_flv(new_video)
  
  
  
              return HttpResponseRedirect('/video_list/')
  
      else:
  
  ...

  This is my first time trying to use a worker (or ever pushing a project to production), so even with the documentation it is still unclear to me what I need to do. I have tried several different things but nothing seems to work. Is there just a simple way to tell celery to run the ffmpegresult = subprocess.call(ffmpeg) ? Thanks in advance for any help or insight.

  EDIT- Added heroku logs

  2013-01-10T20:58:57+00:00 app[web.1]: TARGET: /media/videos/flv/8.flv
  
  2013-01-10T20:58:57+00:00 app[web.1]: IMAGE TASK CODE: ffmpeg -y -i /media/videos/practice.wmv -vframes 1 -ss 00:00:02 - an -vcodec png -f rawvideo -s 320x240 /media/videos/flv/8.png
  
  2013-01-10T20:58:57+00:00 app[web.1]: SOURCE: /media/videos/practice.wmv
  
  2013-01-10T20:58:57+00:00 app[web.1]: FFMPEG TASK CODE: ffmpeg -i /media/videos/practice.wmv -acodec mp3 -ar 22050 -f fl v -s 320x240 /media/videos/flv/8.flv
  
  2013-01-10T20:58:57+00:00 app[web.1]: TARGET IMAGE: /media/videos/flv/8.png
  
  2013-01-10T20:58:57+00:00 app[web.1]: Not working.
  
  2013-01-10T20:58:57+00:00 app[web.1]: Not working.

  NEWER EDIT

  I tried adding a tasks.py and added the task :

  celery = Celery('tasks', broker='redis://guest@localhost//')
  
  
  
  @celery.task
  
  def ffmpeg_task(video):
  
      converted_file = subprocess.call(video)
  
      return converted_file

  then I changed the relevant section of my view to :

  ...
  
  try:
  
      ffmpeg_task.delay(ffmpeg)
  
      print "---------------FFMPEG---------------"
  
      print ffmpegresult
  
  except:
  
      print "Not working."
  
  ...

  My new logs are :

  2013-01-15T13:19:52+00:00 app[web.1]: TARGET IMAGE: /media/videos/flv/12.png
  
  2013-01-15T13:19:52+00:00 app[web.1]: SOURCE: /media/videos/practice.wmv
  
  2013-01-15T13:19:52+00:00 app[web.1]: FFMPEG TASK CODE: ffmpeg -i /media/videos/practice.wmv -acodec mp3 -ar 22050 -f fl v -s 320x240 /media/videos/flv/12.flv
  
  2013-01-15T13:19:52+00:00 app[web.1]: IMAGE TASK CODE: ffmpeg -y -i /media/videos/practice.wmv -vframes 1 -ss 00:00:02 -an -vcodec png -f rawvideo -s 320x240 /media/videos/flv/12.png
  
  2013-01-15T13:19:52+00:00 app[web.1]: TARGET: /media/videos/flv/12.flv
  
  2013-01-15T13:20:17+00:00 app[web.1]: 2013-01-15 13:20:17 [2] [CRITICAL] WORKER TIMEOUT (pid:12)
  
  2013-01-15T13:20:17+00:00 app[web.1]: 2013-01-15 13:20:17 [2] [CRITICAL] WORKER TIMEOUT (pid:12)
  
  2013-01-15T13:20:17+00:00 app[web.1]: 2013-01-15 13:20:17 [19] [INFO] Booting worker with pid: 19

  Am I completely missing something ? I'll keep trying, but will be very appreciative of any direction or assistance.

• Berlin website owners need consent for using Google Analytics

  19 novembre 2019, par Joselyn Khor — Uncategorized

  Website owners in Berlin need consent for using Google Analytics

  According to the Berlin Data Protection Office, if you’re a website owner collecting and sending data to third-party services (like Google Analytics) who are also using that data “for own purpose uses” in Berlin, you are now required to ask for specific consent from visitors in order to collect that information. 

  This means you can only use Google Analytics or similar services once you’ve gotten consent from visitors. In contrast, Matomo does not use information from Cloud or On-Premise users for “own purpose uses”.

  Consent is also needed when keystrokes or mouse movements are recorded. That means you need users to consent to your usage of features like Heatmaps and Session Recordings.

  It’s advised that website owners in Berlin should check their websites for third-party content and tracking mechanisms. If you do use these third-party functions that require consent, you must either get consent or remove the functions. Consent is only effective if the visitor (whose data you’re collecting) is informed and agrees to their data being processed.

  Currently this applies to website owners in Berlin, and should likely be applied for the whole of Germany to be on the safe side.

  Further reading :

  • Learn more about Matomo’s Cloud Data Processing Agreement
  • Read about asking for consent in Matomo
  • To use Matomo Analytics and be compliant with the regulation we recommend you learn how to make Matomo GDPR compliant in 12 steps.

  And learn more about all the features in our GDPR user guide and privacy user guide. You can also learn more about GDPR through our blog posts.