Recherche avancée

Sur d’autres sites (7342)

• Running a py script in the Cloud

  12 janvier 2018, par Anay Bose

  I’m new to Google’s cloud & Virtual Machine(VM) instances, and I need some clarifications on a couple of points. I have a python script ; it imports a long range of functions. I need to run those functions in parallel. I’m using multiprocessing and Process, not threads. These functions are basically image and media processors, and they use many other tools like FFMPEG, imagemagick and Avisynth in addition to a wide range of python modules, including moviepy. Now, I would like to run some 50 functions in parallel assigning a CPU for each process. Images, media and avi files are stored in seperate folders. I’m on Windows7 Core-i7 machine. So, need cloud computing power.

  Now, my question can I run such a python script/app in the cloud that requires a very complicated file system and non-python tools i.e. ffmpeg, avisynth and avi files ?

  Can Google VMs emulate my local machine and empower me with more cores and memory to run such a program ? if not, then what are my options ? Is their any tutorials that I can follow ? I need your suggestions. I have given below an example script and some codes to help facilitate your understanding about my situation.

  from __future__ import unicode_literals
  
  import youtube_dl
  
  import os
  
  import time
  
  import sys
  
  reload(sys)  
  
  sys.setdefaultencoding('utf-8')
  
  from multiprocessing import Process
  
  from utils import *
  
  
  
  from clip31 import VIDEO31
  
  from clip32 import VIDEO32
  
  from clip189 import VIDEO189
  
  from clip16 import VIDEO16
  
  from clip39 import VIDEO39
  
  
  
  
  
  if __name__== '__main__':
  
  
  
      # 1. CALLING A FUNCTION
  
      folder = "bodyforce3\\16"
  
      serial = "16"
  
      images = get_filepaths("../16")
  
      videos = get_filepaths("12__media")
  
      pngs = get_filepaths("../pngs")
  
  
  
      Process(target=VIDEO192, args=(folder, serial, color1, color2, color3, images, videos)).start()
  
  
  
  
  
      # 2. CALLING A FUNCTION
  
      folder = "bodyforce3\\20"
  
      serial = "20"
  
      images = get_filepaths("../20")
  
      videos = get_filepaths("18__media")
  
  
  
      Process(target=VIDEO32, args=(folder, serial, color1, color2, color3, images, videos)).start()
  
  
  
  
  
      # 3. CALLING A FUNCTION
  
      folder = "bodyforce3\\14"
  
      serial = "14"
  
      images = get_filepaths("../14")
  
      videos = get_filepaths("16__media")
  
  
  
      Process(target=VIDEO91, args=(folder, serial, color1, color2, color3, images, videos)).start()

  I copy avi files in functions like this :

  src = "clip50_files"
  
  src_files = os.listdir(src)
  
  for file_name in src_files:
  
      full_file_name = os.path.join(src, file_name)
  
      if (os.path.isfile(full_file_name)):
  
          shutil.copy(full_file_name, folder)

  I call ffmpeg commands like this, and they are included within py functions.

  ###########################
  
  #### FFMPEG OPERATIONS ####
  
  ###########################
  
  
  
  print "Starting FFMPEG operations ..."
  
  
  
  if os.path.isfile(os.path.join(folder, "bounce-(3).avi")):
  
      os.remove(os.path.join(folder, "bounce-(3).avi"))
  
  
  
  
  
  infile = folder + "/bounce-(3).avs"
  
  outfile = folder + "/bounce-(3).avi"
  
  codec = "rawvideo"
  
  pix_fmt = "bgra"
  
  
  
  try:
  
      subprocess.call(["ffmpeg",
  
                       "-i" ,infile, 
  
                       "-c:v" ,codec, 
  
                       "-pix_fmt", pix_fmt, 
  
                       outfile],
  
                      stdout=open(os.devnull, 'w'), 
  
                      stderr=subprocess.STDOUT)
  
  except subprocess.CalledProcessError as e:   
  
      #except subprocess.CalledProcessError as e:
  
      sys.exit(e.output)
  
  except OSError as e:
  
      sys.exit(e.strerror)
  
  
  
  
  
  print "FFMPEG operations ended"

• java.io.IOException : Error running exec() Working Directory : null Environment : null

  16 janvier 2018, par Muhammad Hamza Shahid

  I am using WritingMinds/ffmpeg-android-java in application.

  here is my code

  loadFFmpeg();
  
  String cmd="ffmpeg -i /storage/emulated/0/media/audio/a.mp3 -i  /storage/emulated/0/recording.3gp -filter_complex \"[0:a][1:a]amerge=inputs=2[aout]\" -map \"[aout]\" " + outputFile;
  
  executeFFmpeg(cmd.split(" "));

  and

  private void loadFFmpeg() {
  
      FFmpeg ffmpeg = FFmpeg.getInstance(MainActivity.this.getApplicationContext());
  
      try {
  
          ffmpeg.loadBinary(new LoadBinaryResponseHandler() {
  
  
  
              @Override
  
              public void onStart() {}
  
  
  
              @Override
  
              public void onFailure() {}
  
  
  
              @Override
  
              public void onSuccess() {}
  
  
  
              @Override
  
              public void onFinish() {}
  
          });
  
      } catch (FFmpegNotSupportedException e) {
  
          // Handle if FFmpeg is not supported by device
  
      }
  
  }
  
  
  
  private void executeFFmpeg(String[] cmd)
  
  {
  
      /*String workFolder = getApplicationContext().getFilesDir() + "/ffmpeg";
  
      String environment = Environment.getExternalStorageDirectory().getAbsolutePath();
  
      Map map = new HashMap();
  
      map.put("Working Directory", workFolder);
  
      map.put("Environment",environment);*/
  
      FFmpeg ffmpeg = FFmpeg.getInstance(MainActivity.this.getApplicationContext());
  
      try {
  
          // to execute "ffmpeg -version" command you just need to pass "-version"
  
          ffmpeg.execute(cmd, new ExecuteBinaryResponseHandler() {
  
  
  
              @Override
  
              public void onStart() {}
  
  
  
              @Override
  
              public void onProgress(String message) {}
  
  
  
              @Override
  
              public void onFailure(String message) {}
  
  
  
              @Override
  
              public void onSuccess(String message) {}
  
  
  
              @Override
  
              public void onFinish() {
  
                  stop.setEnabled(false);
  
                  play.setEnabled(true);
  
              }
  
          });
  
      } catch (FFmpegCommandAlreadyRunningException e) {
  
          // Handle if FFmpeg is already running
  
      }
  
  }

  but I am getting following error

  6784-6962/com.flipartstudio.playandrecord E/FFmpeg : Exception while trying to run : [Ljava.lang.String ;@41803270
  java.io.IOException : Error running exec(). Command : [/data/data/com.flipartstudio.playandrecord/files/ffmpeg, /system/bin/ls, -l, /data/data/com.example.foo/files/ffmpeg] Working Directory : null Environment : null
  at java.lang.ProcessManager.exec(ProcessManager.java:211)
  at java.lang.Runtime.exec(Runtime.java:168)
  at java.lang.Runtime.exec(Runtime.java:123)
  at com.github.hiteshsondhi88.libffmpeg.ShellCommand.run(ShellCommand.java:10)
  at com.github.hiteshsondhi88.libffmpeg.FFmpegExecuteAsyncTask.doInBackground(FFmpegExecuteAsyncTask.java:38)
  at com.github.hiteshsondhi88.libffmpeg.FFmpegExecuteAsyncTask.doInBackground(FFmpegExecuteAsyncTask.java:10)
  at android.os.AsyncTask$2.call(AsyncTask.java:287)
  at java.util.concurrent.FutureTask.run(FutureTask.java:234)
  at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:230)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
  at java.lang.Thread.run(Thread.java:856)
  Caused by : java.io.IOException : No such file or directory
  at java.lang.ProcessManager.exec(Native Method)
  at java.lang.ProcessManager.exec(ProcessManager.java:209)
  at java.lang.Runtime.exec(Runtime.java:168) 
  at java.lang.Runtime.exec(Runtime.java:123) 
  at com.github.hiteshsondhi88.libffmpeg.ShellCommand.run(ShellCommand.java:10) 
  at com.github.hiteshsondhi88.libffmpeg.FFmpegExecuteAsyncTask.doInBackground(FFmpegExecuteAsyncTask.java:38) 
  at com.github.hiteshsondhi88.libffmpeg.FFmpegExecuteAsyncTask.doInBackground(FFmpegExecuteAsyncTask.java:10) 
  at android.os.AsyncTask$2.call(AsyncTask.java:287) 
  at java.util.concurrent.FutureTask.run(FutureTask.java:234) 
  at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:230) 
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080) 
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573) 
  at java.lang.Thread.run(Thread.java:856) 

  I have also added

• Limit number of Start-Process running in poweshell

  7 mars 2018, par Ahhhhhhhhhhhhhdfgbv

  I have tried to limit the number of Start-Process running from a Powershell, but I can’t seem to get it to work.

  I tried to follow this process : https://exchange12rocks.org/2015/05/24/how-to-limit-a-number-of-powershell-jobs-running-simultaneously/ and Run N parallel jobs in powershell

  But these are for Jobs not Processes, and I would like to remove the -Wait from the Start-Process

  My concern with the script is that if there are 1000 audio files in the folder, then FFMpeg would crash the system.

  ---

  # get the folder for conversion
  
  function mbAudioConvert {
  
      [Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms") | Out-Null
  
      [System.Windows.Forms.Application]::EnableVisualStyles()
  
  
  
      $fileBrowser = New-Object System.Windows.Forms.FolderBrowserDialog
  
      $fileBrowser.SelectedPath = "B:\"
  
      $fileBrowser.ShowNewFolderButton = $false
  
      $fileBrowser.Description = "Select the folder with the audio which you wish to convert to Avid DNxHD 120 25P 48kHz"
  
  
  
      $mbLoop     = $true
  
      $mbCount    = 0001
  
      $mbMaxJob   = 4
  
  
  
      while( $mbLoop ) {
  
          if( $fileBrowser.ShowDialog() -eq "OK" ) {
  
              $mbLoop     = $false
  
  
  
  
  
              $mbImage    = ( Get-Item -Path "C:\Users\user\Desktop\lib\AudioOnly.jpg" )
  
              $mbff32     = ( Get-Item -Path "C:\Users\user\Desktop\lib\ffmpeg32.exe" )
  
              $mbff64     = ( Get-Item -Path "C:\Users\user\Desktop\lib\ffmpeg64.exe" )
  
  
  
              $mbFolder   = $fileBrowser.SelectedPath
  
              $mbItemInc  = ( ls $mbFolder\* -Include *.mp3, *.MP3, *.wav*, *.WAV*, *.ogg, *.OGG, *.wma, *.WMA, *.flac, *.FLAC, *.m4a, *.M4a )
  
              $mbProgress = ( Get-ChildItem -Path $mbItemInc )
  
  
  
              $mbHasRaw   = ( $mbFolder + "\RAW" )
  
  
  
              if( !( Test-Path -Path $mbHasRaw ) ) {
  
                  # force create a RAW folder if it does not exist
  
                  New-Item -ItemType Directory -Force -Path "$mbHasRaw"
  
              }
  
  
  
  
  
              foreach( $mbItem in $mbItemInc ) {
  
  
  
                  $mbCheck    = $false
  
  
  
                  # output the progress
  
                  # Suggestion: You might want to consider updating this after starting the job and do the final update after running ex. Get-Job | Wait-Job to make the progress-bar stay until all processes are finished
  
                  #Write-Progress -Activity "Counting files for conversion" -status "Currently processing: $mbCount" -percentComplete ($mbCount / $mbItemInc.count*100)
  
  
  
                  # limit the run number
  
                  while ($mbCheck -eq $false) {
  
  
  
                      if( (Get-Job -State 'Running').count -lt $mbMaxJob) {
  
  
  
                          $mbScriptBlock = {
  
                              $mbItemName = $using:mbItem.BaseName
  
  
  
                              $mbNewItem  = ( $using:mbFolder + "\RAW\" + $mbItemName + ".mov" )
  
                              $mbArgs     = " -loop 1 -i $using:mbImage -i $using:mbItem -shortest -c:v dnxhd -b:v 120M -s 1920x1080 -pix_fmt yuv422p -r 25 -c:a pcm_s16le -ar 48k -af loudnorm=I=-12 $mbNewItem"
  
  
  
                              Start-Process -FilePath $using:mbff32 -ArgumentList $mbArgs -NoNewWindow -Wait
  
                          }
  
  
  
                          Start-Job -ScriptBlock $mbScriptBlock
  
  
  
                          #The job-thread doesn't know about $mbCount, better to increment it after starting the job
  
                          $mbCount++
  
                          $mbCheck  = $true          
  
                      }
  
  
  
                  }
  
              }
  
  
  
          } else {
  
  
  
              $mbResponse = [System.Windows.Forms.MessageBox]::Show("You have exited out of the automation process!", "User has cancelled")
  
              if( $mbResponse -eq "OK" ) {
  
                  return
  
              }
  
          }
  
      }
  
  
  
      $fileBrowser.SelectedPath
  
      $fileBrowser.Dispose()
  
  }
  
  
  
  # call to function
  
  mbAudioConvert