Recherche avancée

Sur d’autres sites (7540)

• AWS Lambda : "Unzipped size must be smaller than 106534017 bytes" after adding single file

  17 septembre 2023, par leon

  When trying to deploy my lambdas using AWS through the serverless framework I had no problems until I tried adding the ffmpeg binary.

  


  Now the ffmpeg binaries I have tried to add have ranged from 26 mb to 50 mb. Whichever I add, I get the following error :

  


  UPDATE_FAILED: WhatsappDocumentHandlerLambdaFunction (AWS::Lambda::Function)
Resource handler returned message: "Unzipped size must be smaller than 106534017 bytes (Service: Lambda, Status Code: 400, Request ID: ...)" (RequestToken: ..., HandlerErrorCode: InvalidRequest)


  


  The problem is that I did not add the file to this function. I added it to a completely different one.

  


  I have tried the following things :

  


  

  • Creating an "empty" function that only contains the ffmpeg binary and a function handler

  


  • Creating a layer that only contains the ffmpeg binary

  


  • Deleting the ffmpeg binary (the error goes away and deployment succeeds

  


  • Varying sizes of ffmpeg binaries between 26 and 50mb

  


  • Getting the ffmpeg-lambda-layer (https://github.com/serverlesspub/ffmpeg-aws-lambda-layer ; https://serverlessrepo.aws.amazon.com/applications/us-east-1/145266761615/ffmpeg-lambda-layer) and deploying it myself

  


  


  When trying every single one of these options I get the UPDATE_FAILED error in a different function that surely is not too big.

  


  I know I can deploy using a docker image but why complicate things with docker images when it should work ?

  


  I am very thankful for any ideas.

  


• Using ffmpeg to assemble images from S3 into a video

  10 juillet 2020, par Mass Dot Net

  I can easily assemble images from local disk into a video using ffmpeg and passing a %06d filespec. Here's what a typical (pseudocode) command would look like :

  


  ffmpeg.exe -hide_banner -y -r 60 -t 12 -i /JpgsToCombine/%06d.JPG <..etc..>


  


  However, I'm struggling to do the same with images stored in AWS S3, without using some third party software to mount a virtual drive (e.g. TNTDrive). The S3 folder containing our images is too large to download to the 20GB ephemeral storage provided for AWS containers, and we're trying to avoid EFS because we'd have to provision expensive bandwidth.

  


  Here's what the HTTP and S3 URLs to each of our JPGs looks like :

  


  # HTTP URL
https://massdotnet.s3.amazonaws.com/jpgs-to-combine/000000.JPG # frame 0
https://massdotnet.s3.amazonaws.com/jpgs-to-combine/000012.JPG # frame 12
https://massdotnet.s3.amazonaws.com/jpgs-to-combine/000123.JPG # frame 123
https://massdotnet.s3.amazonaws.com/jpgs-to-combine/456789.JPG # frame 456789

# S3 URL
s3://massdotnet/jpgs-to-combine/000000.JPG # frame 0
s3://massdotnet/jpgs-to-combine/000012.JPG # frame 12
s3://massdotnet/jpgs-to-combine/000123.JPG # frame 123
s3://massdotnet/jpgs-to-combine/456789.JPG # frame 456789


  


  Is there any way to get ffmpeg to assemble these ? We could generate a signed URL for each S3 file, and put several thousand of those URLs onto a command line with an FFMPEG concat filter. However, we'd run up into the command line input limit in Linux at some point using this approach. I'm hoping there's a better way...

  


• ffmpeg exit code 429496724 when opening file with {} in name

  16 février, par Holly Wilson

  I'm trying to write a program that will read the names of files in a directory, and parse them for info which it will then write into that file's metadata (seems pointless, I know, but it's more of a trial run for a larger project. If I can figure this out, then I'll be ready to move on to what I actually want to do).
All of my filenames are formatted :

  


  Title, Article* - Subtitle* Cut* [Year]
*if present

  


  The four test videos I'm using are titled :

  


  Test Video 1 [2000]

  


  Test Video 2, A [2000]

  


  Test Video 3 Test Cut [2000]

  


  Test Video 4 - The Testening [2000]

  


  The code seems to be working fine on videos 1, 2, & 4 ; but video 3 is causing me a lot of headache.

  


  //node C:\Users\User\Documents\Coding\Tools\testMDG.js
const fs = require('fs');
const path = require('path');
const ffmpeg = require('fluent-ffmpeg');
const async = require('async');
const directory = path.normalize('F:\\Movies & TV\\Movies\\testDir');
let x = 0;
const fileArray = [];
const succArray = [];
const failArray = [];
// Set the path to the ffmpeg executable
ffmpeg.setFfmpegPath(path.normalize('C:\\ffmpeg\\bin\\ffmpeg.exe'));

// Add this near the start of your script
const tempDir = path.join(directory, 'temp');
if (!fs.existsSync(tempDir)) {
    fs.mkdirSync(tempDir, { recursive: true });
}

const progresscsv = path.normalize('C:\\Users\\User\\Documents\\Coding\\Tools\\progress.csv');
if (fs.existsSync(progresscsv)) {
    fs.unlinkSync(progresscsv);
};

const csvStream = fs.createWriteStream(progresscsv);
csvStream.write('File Name,Status\n');

const CONCURRENCY_LIMIT = 3; // Adjust based on your system capabilities

// Add at the start of your script
const processedFiles = new Set();

function sanitizeFilePath(filePath) {
    return filePath.replace(/([{}])/g, '\\$1');
}

// Create a queue
const queue = async.queue(async (task, callback) => {
    const { file, filePath, tempFilePath, movieMetadata } = task;
    try {
        await new Promise((resolve, reject) => {
            console.log(`ffmpeg reading: ${sanitizeFilePath(filePath)}`);
            ffmpeg(sanitizeFilePath(filePath))
                .outputOptions([
                    '-y',
                    '-c', 'copy',
                    '-map', '0',
                    '-metadata', `title=${movieMetadata.title}`,
                    '-metadata', `subtitle=${movieMetadata.subtitle || ''}`,
                    '-metadata', `comment=${movieMetadata.cut || ''}`,
                    '-metadata', `year=${movieMetadata.year}`
                ])
                .on('error', (err) => reject(err))
                .on('end', () => resolve())
                .saveToFile(tempFilePath);
        });
        
        // Handle success
        console.log(`Successfully processed: ${file}`);
        succArray.push(file);
        // Only call callback once
        if (callback && typeof callback === 'function') {
            callback();
        }
    } catch (err) {
        // Handle error
        console.error(`Error processing ${file}: ${err.message}`);
        failArray.push(file);
        // Only call callback once with error
        if (callback && typeof callback === 'function') {
            callback(err);
        }
    }
}, CONCURRENCY_LIMIT);

fs.readdir(directory, (err, files) => {
    if (err) {
        console.error(`Error reading directory: ${err.message}`);
        return;
    }
    console.log(directory);

    // Filter for files only
    files = files.filter(file => fs.statSync(path.join(directory, file)).isFile());
    console.log(files);

    for (const file of files) {
        x++;
        const filePath = path.join(directory, file);
        let desort = file.replace(/(.*),\s(the\s|an\s|a\s)/i, '$2'+'$1 ') || file;
        
        // Create task object for queue
        const task = {
            file,
            filePath: filePath,
            tempFilePath: path.join(directory, 'temp', `temp_${x}_${path.parse(file).name
                .replace(/[^a-zA-Z0-9]/g, '_')}${path.extname(file)}`),
            movieMetadata: {
                title: desort.replace(/(\s[\-\{\[].*)/gi, ``),
                subtitle: desort.includes('-') ? desort.replace(/(.*)\-\s(.*?)[\{\[].*/gi, '$2') : null,
                cut: desort.includes('{') ? desort.replace(/.*\{(.*)\}.*/gi, '$1') : null,
                year: desort.replace(/.*\[(.*)\].*/gi, '$1')
            }
        };
        
        // Add to processing queue
        queue.push(task, (err) => {
            if (!processedFiles.has(task.file)) {
                processedFiles.add(task.file);
                if (err) {
                    csvStream.write(`${task.file},Failed\n`);
                } else {
                    csvStream.write(`${task.file},Processed\n`);
                }
            }
        });
    }
});

// Add queue completion handler
queue.drain(() => {
    console.log('All files have been processed');
    console.log(`Success: ${succArray.length} files: ${succArray}`);
    console.log(`Failed: ${failArray.length} files: ${failArray}`);
});

//node C:\Users\User\Documents\Coding\Tools\testMDG.js


  


  And the console is logging :

  


  PS C:\Users\User> node C:\Users\User\Documents\Coding\Tools\testMDG.js
F:\Movies & TV\Movies\testDir
[
  'Test Video 1 [2020].mp4',
  'Test Video 2, A [2020].mp4',
  'Test Video 3 {Test Cut} [2020].mp4',
  'Test Video 4 - The Testening [2020].mp4'
]
ffmpeg reading: F:\Movies & TV\Movies\testDir\Test Video 1 [2020].mp4
ffmpeg reading: F:\Movies & TV\Movies\testDir\Test Video 2, A [2020].mp4
ffmpeg reading: F:\Movies & TV\Movies\testDir\Test Video 3 \{Test Cut\} [2020].mp4
Error processing Test Video 3 {Test Cut} [2020].mp4: ffmpeg exited with code 4294967294: Error opening input file F:\Movies & TV\Movies\testDir\Test Video 3 \{Test Cut\} [2020].mp4.
Error opening input files: No such file or directory

ffmpeg reading: F:\Movies & TV\Movies\testDir\Test Video 4 - The Testening [2020].mp4
Successfully processed: Test Video 1 [2020].mp4
Successfully processed: Test Video 2, A [2020].mp4
Successfully processed: Test Video 4 - The Testening [2020].mp4
All files have been processed
Success: 3 files: Test Video 1 [2020].mp4,Test Video 2, A [2020].mp4,Test Video 4 - The Testening [2020].mp4
Failed: 1 files: Test Video 3 {Test Cut} [2020].mp4


  


  I've tried so many different solutions in the sanitizeFilePath function. Escaping the {} characters (as included below), escaping all non-alphanumeric characters, putting the filepath in quotes, etc. VSCode's CoPilot is just pulling me round in circles, suggesting solutions I've already tried.