Recherche avancée

Sur d’autres sites (6449)

• Twilio Real-Time Media Streaming to WebSocket Receives Only Noise Instead of Speech

  21 février, par dannym25

  I'm setting up a Twilio Voice call with real-time media streaming to a WebSocket server for speech-to-text processing using Google Cloud Speech-to-Text. The connection is established successfully, and I receive a continuous stream of audio data from Twilio. However, when I play back the received audio, all I hear is a rapid clicking/jackhammering noise instead of the actual speech spoken during the call.

  


  Setup :

  


  

  • Twilio sends inbound audio to my WebSocket server.

  


  • WebSocket receives and saves the raw mulaw-encoded audio data from Twilio.

  


  • The audio is processed via Google Speech-to-Text for transcription.

  


  • When I attempt to play back the audio, it sounds like machine-gun-like noise instead of spoken words.

  


  


  1. Confirmed WebSocket Receives Data

  


  • The WebSocket successfully logs incoming audio chunks from Twilio :

  


  🔊 Received 379 bytes of audio from Twilio
🔊 Received 379 bytes of audio from Twilio


  


  • This suggests Twilio is sending audio data, but it's not being interpreted correctly.

  


  2. Saving and Playing Raw Audio

  


  • I save the incoming raw mulaw (8000Hz) audio from Twilio to a file :

  


  fs.appendFileSync('twilio-audio.raw', message);


  


  • Then, I convert it to a .wav file using FFmpeg :

  


  ffmpeg -f mulaw -ar 8000 -ac 1 -i twilio-audio.raw twilio-audio.wav


  


  • Problem : When I play the audio using ffplay, it contains no speech, only rapid clicking sounds.

  


  3. Ensured Correct Audio Encoding

  


  • Twilio sends mulaw 8000Hz mono format.
• Verified that my ffmpeg conversion is using the same settings.
• Tried different conversion methods :

  


  ffmpeg -f mulaw -ar 8000 -ac 1 -i twilio-audio.raw -c:a pcm_s16le twilio-audio-fixed.wav


  


  → Same issue.

  


  4. Checked Google Speech-to-Text Input Format

  


  • Google STT requires proper encoding configuration :

  


  const request = {
    config: {
        encoding: 'MULAW',
        sampleRateHertz: 8000,
        languageCode: 'en-US',
    },
    interimResults: false,
};


  


  • No errors from Google STT, but it never detects speech, likely because the input audio is just noise.

  


  5. Confirmed That Raw Audio is Not a WAV File

  


  • Since Twilio sends raw audio, I checked whether I needed to strip the header before processing.
• Tried manually extracting raw bytes, but the issue persists.

  


  Current Theory :

  


  

  • The WebSocket server might be handling Twilio’s raw audio incorrectly before saving it.

  


  • There might be an additional header in the Twilio stream that needs to be removed before playback.

  


  • Twilio’s <stream></stream> tag expects a WebSocket connection starting with wss:// instead of https://, and switching to wss:// partially fixed some previous connection issues.

  


  


  Code Snippets :

  


  Twilio Setup in TwiML Response

  


  app.post('/voice-response', (req, res) => {
    console.log("📞 Incoming call from Twilio");

    const twiml = new twilio.twiml.VoiceResponse();
    twiml.say("Hello! Welcome to the service. How can I help you?");
    
    // Prevent Twilio from hanging up too early
    twiml.pause({ length: 5 });

    twiml.connect().stream({
        url: `wss://your-ngrok-url/ws`,
        track: "inbound_track"
    });

    console.log("🛠️ Twilio Stream URL:", `wss://your-ngrok-url/ws`);
    
    res.type('text/xml').send(twiml.toString());
});


  


  WebSocket Server Handling Twilio Audio Stream

  


  wss.on('connection', (ws) => {
    console.log("🔗 WebSocket Connected! Waiting for audio input...");

    ws.on('message', (message) => {
        console.log(`🔊 Received ${message.length} bytes of audio from Twilio`);

        // Save raw audio data for debugging
        fs.appendFileSync('twilio-audio.raw', message);

        // Check if audio is non-empty but contains only noise
        if (message.length < 100) {
            console.warn("⚠️ Warning: Audio data from Twilio is very small. Might be silent.");
        }
    });

    ws.on('close', () => {
        console.log("❌ WebSocket Disconnected!");
        
        // Convert Twilio audio for debugging
        exec(`ffmpeg -f mulaw -ar 8000 -ac 1 -i twilio-audio.raw twilio-audio.wav`, (err) => {
            if (err) console.error("❌ FFmpeg Conversion Error:", err);
            else console.log("✅ Twilio Audio Saved as `twilio-audio.wav`");
        });
    });

    ws.on('error', (error) => console.error("⚠️ WebSocket Error:", error));
});


  


  Questions :

  


  

  • Why is the audio from Twilio being received as a clicking noise instead of actual speech ?

  


  • Do I need to strip any additional metadata from the raw bytes before saving ?

  


  • Is there a known issue with Twilio’s mulaw format when streaming audio over WebSockets ?

  


  • How can I confirm that Google STT is receiving properly formatted audio ?

  


  


  Additional Context :

  


  

  • Twilio <stream></stream> is connected and receiving data (confirmed by logs).

  


  • WebSocket successfully receives and saves audio, but it only plays noise.

  


  • Tried multiple ffmpeg conversions, Google STT configurations, and raw data inspection.

  


  • Still no recognizable speech in the audio output.

  


  


  Any help is greatly appreciated ! 🙏

  


• iOS FFMPEG encode images to video

  10 avril 2013, par brad.roush

  I am trying to take a set of UIImages and create a video file out of them with FFMPEG. Seems there are lots of questions about this topic but non have been able to get this working correctly for me. This one was particularly helpful in giving me a starting point. This iFrameExtractor example was also very helpful but I want to do this in reverse, then add audio.

  This is the closest I have gotten and it creates a short silent video with flashing colors and no images :

  // Register all formats and codecs
  
  av_register_all();
  
  
  
  AVCodec *codec;
  
  AVCodecContext *c= NULL;
  
  
  
  int i, out_size, size, outbuf_size;
  
  FILE *file;
  
  AVFrame *picture;
  
  uint8_t *outbuf;
  
  
  
  NSLog(@"Video encoding");
  
  
  
  /* find the mpeg video encoder */
  
  codec = avcodec_find_encoder(CODEC_ID_MPEG2VIDEO);
  
  if (!codec) {
  
      fprintf(stderr, "codec not found\n");
  
      exit(1);
  
  }
  
  
  
  c= avcodec_alloc_context3(codec);
  
  picture= avcodec_alloc_frame();
  
  
  
  /* put sample parameters */
  
  c->bit_rate = 400000;
  
  /* resolution must be a multiple of two */
  
  c->width = 352;
  
  c->height = 288;
  
  /* frames per second */
  
  c->time_base= (AVRational){1,25};
  
  c->gop_size = 10; /* emit one intra frame every ten frames */
  
  c->max_b_frames=1;
  
  c->pix_fmt = PIX_FMT_YUV420P;
  
  
  
  /* open it */
  
  if (avcodec_open2(c, codec, nil) < 0) {
  
      fprintf(stderr, "could not open codec\n");
  
      exit(1);
  
  }
  
  
  
  // Put file in place
  
  
  
  NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
  
  NSString *docs_dir = [paths objectAtIndex:0];
  
  NSString *filePath = [docs_dir stringByAppendingPathComponent:@"test.mpeg"];
  
  
  
  // Seed file
  
  NSString *sourcePath = [[NSBundle mainBundle] pathForResource:@"test" ofType:@"mov"];
  
  NSError* error = nil;
  
  if (![[NSFileManager defaultManager] copyItemAtPath:sourcePath toPath:filePath error:&error]) {
  
      NSLog(@"Test Video creation failed:%@",[error userInfo]);
  
  } else NSLog(@"Test Video Created");
  
  
  
  const char *filename = [filePath UTF8String];
  
  file = fopen(filename, "wb");
  
  if (!file) {
  
      fprintf(stderr, "could not open %s\n", "filename");
  
      exit(1);
  
  }
  
  
  
  /* alloc image and output buffer */
  
  outbuf_size = 100000;
  
  outbuf = malloc(outbuf_size);
  
  size = c->width * c->height;
  
  
  
  //#pragma mark -
  
  AVFrame* outpic = avcodec_alloc_frame();
  
  int nbytes = avpicture_get_size(PIX_FMT_YUV420P, c->width, c->height);
  
  
  
  //create buffer for the output image
  
  uint8_t* outbuffer = (uint8_t*)av_malloc(nbytes);
  
  
  
  
  
  AVPacket packet;
  
  av_init_packet(&packet);
  
  
  
  
  
  //#pragma mark -
  
  for(i=1;i<50;i++) {
  
      fflush(stdout);
  
  
  
      int numBytes = avpicture_get_size(PIX_FMT_YUV420P, c->width, c->height);
  
      uint8_t *buffer = (uint8_t *)av_malloc(numBytes*sizeof(uint8_t));
  
  
  
      UIImage *image = [UIImage imageWithContentsOfFile:[docs_dir stringByAppendingPathComponent:[NSString stringWithFormat:@"%d.png",i]]];
  
      CGImageRef newCgImage = [image CGImage];
  
  
  
      CGDataProviderRef dataProvider = CGImageGetDataProvider(newCgImage);
  
      CFDataRef bitmapData = CGDataProviderCopyData(dataProvider);
  
      buffer = (uint8_t *)CFDataGetBytePtr(bitmapData);
  
  
  
      avpicture_fill((AVPicture*)picture, buffer, PIX_FMT_RGB8, c->width, c->height);
  
      avpicture_fill((AVPicture*)outpic, outbuffer, PIX_FMT_YUV420P, c->width, c->height);
  
  
  
      struct SwsContext* fooContext = sws_getContext(c->width, c->height,
  
                                                     PIX_FMT_RGB8,
  
                                                     c->width, c->height,
  
                                                     PIX_FMT_YUV420P,
  
                                                     SWS_FAST_BILINEAR, NULL, NULL, NULL);
  
  
  
      //perform the conversion
  
  
  
      sws_scale(fooContext, outpic->data, outpic->linesize,
  
                0, c->height, outpic->data, outpic->linesize);
  
      // Tried This but it didn't work
  
      //sws_scale(fooContext, picture->data, picture->linesize, 0, c->height, outpic->data, outpic->linesize);
  
  
  
  
  
      out_size = avcodec_encode_video(c, outbuf, outbuf_size, outpic);
  
      // Tried this but it didn't work
  
      //int test = 0;
  
      //out_size = avcodec_encode_video2(c, &packet, outpic, &test);
  
  
  
  
  
  
  
      printf("encoding frame %3d (size=%5d)\n", i, out_size);
  
      fwrite(outbuf, 1, out_size, file);
  
  
  
      free(buffer);
  
      buffer = NULL;
  
  
  
  }
  
  
  
  /* get the delayed frames */
  
  /*
  
  for(; out_size; i++) {
  
      fflush(stdout);
  
  
  
      out_size = avcodec_encode_video(c, outbuf, outbuf_size, NULL);
  
      printf("write frame %3d (size=%5d)\n", i, out_size);
  
      fwrite(outbuf, 1, outbuf_size, file);
  
  }
  
  */
  
  
  
  /* add sequence end code to have a real mpeg file */
  
  outbuf[0] = 0x00;
  
  outbuf[1] = 0x00;
  
  outbuf[2] = 0x01;
  
  outbuf[3] = 0xb7;
  
  fwrite(outbuf, 1, 4, file);
  
  fclose(file);
  
  free(outbuf);
  
  
  
  avcodec_close(c);
  
  av_free(c);
  
  av_free(picture);
  
  printf("\n");

  Any ideas will be helpful here. If anyone knows of any other good objective-c examples, this would also be great.

• Facing issues with adding text over a video as watermark using ffmpeg in Laravel

  18 septembre 2024, par Kelash

  I am facing issues with adding text over a video as a watermark using pbmedia/laravel-ffmpeg.

  


  Where am I going wrong ?

  


  Code :

  


  $format = new X264();
$format->setAudioCodec('aac');
$format->setVideoCodec('libx264');
$format->setKiloBitrate(0);

$localPath = '/' . $this->video->id . '.mp4';

$ffmpeg = FFMpeg::fromDisk("public")
          ->open($localPath)
          ->addFilter(function ($filters) {
            $filters->custom("drawtext=fontfile=S:/Freelancer/version58trials/version58trials/public/webfonts/arial.TTF:text='Stack Overflow':fontcolor=white:fontsize=24");
          })
          ->export()
          ->toDisk('public')
          ->inFormat($format)
          ->save("watermark_video_added.mp4");


  


  And this is error I am getting :

  


  [2024-08-25 08:53:25] local.INFO: ffprobe running command C:\ffmpeg\bin\ffprobe.exe -help -loglevel quiet  
[2024-08-25 08:53:25] local.INFO: ffprobe executed command successfully  
[2024-08-25 08:53:25] local.INFO: ffprobe running command C:\ffmpeg\bin\ffprobe.exe "S:/Freelancer/version58trials/version58trials/storage/app/public/143.mp4" -show_streams -print_format json  
[2024-08-25 08:53:25] local.INFO: ffprobe executed command successfully  
[2024-08-25 08:53:25] local.INFO: ffmpeg running command C:\ffmpeg\bin\ffmpeg.exe -y -ss 00:00:01.00 -i "S:/Freelancer/version58trials/version58trials/storage/app/public/143.mp4" -vframes 1 -f image2 "S:/Freelancer/version58trials/version58trials/storage/app/public/RzBnC8ZESC40iRSfOuED66cb37513ccd11724594001-poster.jpg"  
[2024-08-25 08:53:26] local.INFO: ffmpeg executed command successfully  
[2024-08-25 08:53:26] local.INFO: ffmpeg running command C:\ffmpeg\bin\ffmpeg.exe -y -i "S:/Freelancer/version58trials/version58trials/storage/app/public/143.mp4" -threads 12 -vcodec libx264 -acodec aac -refs 6 -coder 1 -sc_threshold 40 -flags +loop -me_range 16 -subq 7 -i_qfactor 0.71 -qcomp 0.6 -qdiff 4 -trellis 1 -b:a 128k -vf "[in]drawtext=fontfile=S:/Freelancer/version58trials/version58trials/public/webfonts/arial.TTF:text='Stack Overflow':fontcolor=white:fontsize=24[out]" "S:/Freelancer/version58trials/version58trials/storage/app/public/watermark_video_added.mp4"  
[2024-08-25 08:53:26] local.INFO: ffprobe running command C:\ffmpeg\bin\ffprobe.exe "S:/Freelancer/version58trials/version58trials/storage/app/public/143.mp4" -show_format -print_format json  
[2024-08-25 08:53:26] local.INFO: ffprobe executed command successfully  
[2024-08-25 08:53:26] local.ERROR: ffmpeg failed to execute command C:\ffmpeg\bin\ffmpeg.exe -y -i "S:/Freelancer/version58trials/version58trials/storage/app/public/143.mp4" -threads 12 -vcodec libx264 -acodec aac -refs 6 -coder 1 -sc_threshold 40 -flags +loop -me_range 16 -subq 7 -i_qfactor 0.71 -qcomp 0.6 -qdiff 4 -trellis 1 -b:a 128k -vf "[in]drawtext=fontfile=S:/Freelancer/version58trials/version58trials/public/webfonts/arial.TTF:text='Stack Overflow':fontcolor=white:fontsize=24[out]" "S:/Freelancer/version58trials/version58trials/storage/app/public/watermark_video_added.mp4": ffmpeg version 2024-08-21-git-9d15fe77e3-full_build-www.gyan.dev Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 13.2.0 (Rev5, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libaribb24 --enable-libaribcaption --enable-libdav1d --enable-libdavs2 --enable-libopenjpeg --enable-libquirc --enable-libuavs3d --enable-libxevd --enable-libzvbi --enable-libqrencode --enable-librav1e --enable-libsvtav1 --enable-libvvenc --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxeve --enable-libxvid --enable-libaom --enable-libjxl --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-dxva2 --enable-d3d11va --enable-d3d12va --enable-ffnvcodec --enable-libvpl --enable-nvdec --enable-nvenc --enable-vaapi --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libcodec2 --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      59. 34.100 / 59. 34.100
  libavcodec     61. 11.100 / 61. 11.100
  libavformat    61.  5.101 / 61.  5.101
  libavdevice    61.  2.100 / 61.  2.100
  libavfilter    10.  2.102 / 10.  2.102
  libswscale      8.  2.100 /  8.  2.100
  libswresample   5.  2.100 /  5.  2.100
  libpostproc    58.  2.100 / 58.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'S:/Freelancer/version58trials/version58trials/storage/app/public/143.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp41isom
    creation_time   : 2024-08-25T10:28:56.000000Z
  Duration: 00:00:05.65, start: 0.000000, bitrate: 6491 kb/s
  Stream #0:0[0x1](und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(progressive), 1906x960 [SAR 1:1 DAR 953:480], 6317 kb/s, 30 fps, 30 tbr, 30k tbn (default)
      Metadata:
        creation_time   : 2024-08-25T10:28:56.000000Z
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
        encoder         : AVC Coding
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 192 kb/s (default)
      Metadata:
        creation_time   : 2024-08-25T10:28:56.000000Z
        handler_name    : SoundHandler
        vendor_id       : [0][0][0][0]
[AVFilterGraph @ 0000020e99b5f0c0] No option name near '/Freelancer/version58trials/version58trials/public/webfonts/arial.TTF:text=Stack Overflow:fontcolor=white:fontsize=24'
[AVFilterGraph @ 0000020e99b5f0c0] Error parsing a filter description around: [out]
[AVFilterGraph @ 0000020e99b5f0c0] Error parsing filterchain '[in]drawtext=fontfile=S:/Freelancer/version58trials/version58trials/public/webfonts/arial.TTF:text='Stack Overflow':fontcolor=white:fontsize=24[out]' around: [out]
Error opening output file S:/Freelancer/version58trials/version58trials/storage/app/public/watermark_video_added.mp4.
Error opening output files: Invalid argument


  


  What I tried, is manipulating that filter function but I couldn't make it.

  


  For your knowledge,the video paths and font path is correct the only issue is with ffmpeg adding watermark text logic.