Recherche avancée

Sur d’autres sites (12380)

• Detect volume via mic, start recording, end on silence, transcribe and sent to endpoint

  15 juin 2023, par alphadmon

  I have been attempting to get this to work in many ways but I can't seem to get it right. Most of the time I get a part of it to work and then when I try to make other parts work, I generally break other things.

  


  I am intercepting the volume coming from the mic and if it is louder than 50, I start a recording. I then keep recording until there is a silence, if the silence is equal to 5 seconds I then stop the recording.

  


  I then send the recording to be transcribed by whisper using OpenAI API.

  


  Once that is returned, I then want to send it to the open ai chat end point and get the response.

  


  After that, I would like to start listening again.

  


  Here is what I have that is sort of working so far, but the recording is an empty file always :

  


  // DETECT SPEECH
const recorder = require('node-record-lpcm16');

// TRANSCRIBE
const fs = require("fs");
const ffmpeg = require("fluent-ffmpeg");
const mic = require("mic");
const { Readable } = require("stream");
const ffmpegPath = require("@ffmpeg-installer/ffmpeg").path;
require('dotenv').config();

// CHAT
const { Configuration, OpenAIApi } = require("openai");

// OPEN AI
const configuration = new Configuration({
    organization: process.env.OPENAI_ORG,
    apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);

// SETUP
ffmpeg.setFfmpegPath(ffmpegPath);

// VARS
let isRecording = false;
const audioFilename = 'recorded_audio.wav';
const micInstance = mic({
    rate: '16000',
    channels: '1',
    fileType: 'wav',
});

// DETECT SPEECH
const file = fs.createWriteStream('determine_speech.wav', { encoding: 'binary' });
const recording = recorder.record();
recording.stream().pipe(file);


recording.stream().on('data', async (data) => {
    let volume = parseInt(calculateVolume(data));
    if (volume > 50 && !isRecording) {
        console.log('You are talking.');
        await recordAudio(audioFilename);
    } else {
        setTimeout(async () => {
            console.log('You are quiet.');
            micInstance.stop();
            console.log('Finished recording');
            const transcription = await transcribeAudio(audioFilename);
            console.log('Transcription:', transcription);
            setTimeout(async () => {
                await askAI(transcription);
            }, 5000);
        }, 5000);
    }
});

function calculateVolume(data) {
    let sum = 0;

    for (let i = 0; i < data.length; i += 2) {
        const sample = data.readInt16LE(i);
        sum += sample * sample;
    }

    const rms = Math.sqrt(sum / (data.length / 2));

    return rms;
}

// TRANSCRIBE
function recordAudio(filename) {
    const micInputStream = micInstance.getAudioStream();
    const output = fs.createWriteStream(filename);
    const writable = new Readable().wrap(micInputStream);

    console.log('Listening...');

    writable.pipe(output);

    micInstance.start();

    micInputStream.on('error', (err) => {
        console.error(err);
    });
}

// Transcribe audio
async function transcribeAudio(filename) {
    const transcript = await openai.createTranscription(
        fs.createReadStream(filename),
        "whisper-1",
    );
    return transcript.data.text;
}

// CHAT
async function askAI(text) {
    let completion = await openai.createChatCompletion({
        model: "gpt-4",
        temperature: 0.2,
        stream: false,
        messages: [
            { role: "user", content: text },
            { role: "system", content: "Act like you are a rude person." }
        ],
    });

    completion = JSON.stringify(completion.data, null, 2);
    console.log(completion);
}


  


• Dynamically record parts of a video stream using ffmpeg based on incoming events

  12 septembre 2023, par ganjim

  Say I have a RTSP stream running on some server.

  


  I want to record parts of this video stream based on some events that will come up on my machine.
My goal is to record 10 seconds before up to 10 seconds after the PTS in any received event. Consider that I have a way to synchronize the PTS between the sender and the receiver, but by the time I receive the events, its already streamed and is in the past.
So I either need to have the ffmpeg command running already, or to have buffered streaming video in my memory.

  


  I just added some code with comments as a way to simulate the situation, it is not complete as I still don't have a working solution. But I'm looking to understand if ffmpeg has capabilities for dealing with rtsp streams suitable for this situation.

  


  const { spawn } = require('child_process');

function processEvent(event){
  
  let startTime = event.pts - 10
  let endTime = event.pts + 10

  const ffmpegArgs = [
    '-i', "rtspUrl",
    '-ss', startTime.toString(),
    '-to', endTime.toString(),
    '-c', 'copy',
    '-f', 'mp4',
    `output_${startTime}_${endTime}.mp4` // Output filename
  ];
  // Here it is obviously not possible to give ffmpeg a negative startTime.
  // We either have to have spawned the ffmpeg command and somehow give it a starting time for recording on demand or have a buffering system in place.
  // Having a buffer to store every raw frame and then attach them after endTime is also considerably CPU and memory intensive.
  // Looking for alternative ways to achieve the same output.

  const ffmpegProcess = spawn('ffmpeg', ffmpegArgs, {
    stdio: 'inherit'
  });
}

// Code to simulate the events:
// imagine these pts to be relative to Date.now(), so using negative PTS to show the events are going to be in the past by the time we receive them.
setTimeout(() => {
  const event1= { pts: -30 };
  processEvent(event1);
}, 1000);

setTimeout(() => {
  const event2 = { pts: -20 };
  processEvent(event2);
}, 5000);


  


• ffmpeg avformat_open_input() function causes memory leak when receiving live stream

  12 septembre 2023, par george_d

  I have live streams (can be UDP or HLS, video codec is H264), from which I grab frames for further processing.

  


  For this purpose, I use ffmpeg + nvjpeg + cuda libraries.

  


  However I noticed memory leak - memory usage periodically (every 10-20 seconds) is increased by 100-400 KB, the amount and period may vary.

  


  After disabling pieces of code one by one, I realized that it is avformat_open_input() which causes memory leak.

  


  No matter which buffer settings (https://ffmpeg.org/ffmpeg-protocols.html#udp) I choose for UDP, the leak still persists. Same goes for HLS streams.

  


  I tried to find anything related to this problem, but all the sources I found claimed that this problem took place in the past and has been fixed.

  


  Is there some mysterious setting I am missing, so that memory could be freed properly ?

  


  Or is this memory supposed to be freed when processing frames (i.e. using av_read_frame() and av_packet_unref(), etc) ?

  


  Minimal example of code to reproduce the problem :

  


  avformat_example.cpp

  


  #include &#xA;extern "C" {&#xA;    #include <libavformat></libavformat>avformat.h>&#xA;    #include <libavcodec></libavcodec>avcodec.h>&#xA;}&#xA;&#xA;int main(int argc, char *argv[]){&#xA;    if (argc &lt; 2) {&#xA;      return 1;&#xA;    }&#xA;&#xA;    char* inputSource = argv[1];&#xA;    AVFormatContext *ctx = NULL;&#xA;&#xA;    if (avformat_open_input(&amp;ctx, inputSource, NULL, NULL) != 0) {&#xA;        av_log(NULL,&#xA;               AV_LOG_ERROR,&#xA;               "Cannot open &#x27;%s&#x27;",&#xA;               inputSource);&#xA;        return 1;&#xA;    }&#xA;&#xA;    /*&#xA;    This loop is placed here to demonstrate&#xA;    avformat_open_input() causing leak.&#xA;    Actually, instead of noop loop there is logic of getting and processing frames,&#xA;    but it doesn&#x27;t matter now.&#xA;    As loop goes on, the amount of leaked memory increases.&#xA;    */&#xA;    while(true) {&#xA;      sleep(1);&#xA;    }&#xA;&#xA;    return 0;&#xA;}&#xA;

  


  Compile with :

  


  g++ avformat_example.cpp -lavcodec -lavutil -lavformat -I/usr/include/ffmpeg-cuda -o avformat_open_input_example


  


  Run :

  


  ./avformat_open_input_example "udp://127.0.0.1:5000?reuse=1&pkt_size=1316&buffer_size=1310720&fifo_size=40000"


  


  Version of ffmpeg underlying libraries :

  


  libavutil      58.  7.100 / 58.  7.100
libavcodec     60. 11.100 / 60. 11.100
libavformat    60.  5.100 / 60.  5.100
libavdevice    60.  2.100 / 60.  2.100
libavfilter     9.  8.100 /  9.  8.100
libswscale      7.  2.100 /  7.  2.100
libswresample   4. 11.100 /  4. 11.100