Recherche avancée

Sur d’autres sites (11237)

• RoQ on Dreamcast

  18 mars 2011, par Multimedia Mike — Sega Dreamcast

  I have been working on that challenge to play back video on the Sega Dreamcast. To review, I asserted that the RoQ format would be a good fit for the Sega Dreamcast hardware. The goal was to play 640x480 video at 30 frames/second. Short version : I have determined that it is possible to decode such video in real time. However, I ran into certain data rate caveats.

  First off : Have you ever wondered if the Dreamcast can read an 80mm optical disc ? It can ! I discovered this when I only had 60 MB of RoQ samples to burn on a disc and a spindle full of these 210MB-capacity 80mm CD-Rs that I never have occasion to use.

  
  
  

  New RoQ Library
  There are open source RoQ decoders out there but I decided to write a new one. A few reasons : 1) RoQ is so simple that I didn’t think it would take too long ; 2) it would be nice to have a RoQ library that is license-compatible (BSD-like) with the rest of the KallistiOS distribution ; 3) the idroq.tar.gz distribution, while license-compatible, has enough issues that I didn’t want to correct it.

  Thankfully, I was correct about the task not being too difficult : I put together a new RoQ decoder in short order. I’m a bit embarrassed to admit that the part I had the most trouble with was properly converting YUV -> RGB.
  
  About the approach I took : While the original idroq.tar.gz decoder maintains YUV 4:2:0 codebooks (which led to chroma bugs during motion compensation) and FFmpeg’s decoder maintains YUV 4:4:4 codebooks, this decoder is built to convert the YUV 4:2:0 vectors into RGB565 vectors during the vector unpacking phase. Thus, the entire frame is rendered in RGB565 — no lengthy YUV -> RGB conversion after decoding — and all pixels are shuffled around as 16-bit units (minor speedup vs. shuffling everything as bytes).

  I also entertained the idea of maintaining YUYV codebooks (since the DC supports that colorspace as a texture format). But I scrapped that idea when I remembered it would lead to the same chroma bleeding problem seen in the original idroq.tar.gz decoder.

  Onto The Dreamcast
  I developed the library on a Linux computer, allowing it to output a series of PNM files for visual verification and debugging. Dropping it into a basic DC/KOS-compatible program was trivial and the first order of business was profiling.

  At first, I profiled the entire decode operation : open file, then read and decode each chunk while tossing away the results. I was roundly disappointed to see that, e.g., an 8.5-second RoQ sample needed a little more than 20 seconds to complete. Not real time. I performed a series of optimizations on the decoding library that netted notable performance gains when profiling on Linux. When I brought these same optimizations over to the DC, decoding time didn’t improve at all. This was my first suspicion that perhaps my assumptions regarding the DC’s optical drive’s data rate were not correct.

  Dreamcast Data Rate Profiling
  Let’s start with some definitions : In terms of data rate, an ’X’, i.e., 1X is the minimum data rate needed to read CD quality audio from a disc. At that speed, a drive should be able to stream 75 sectors each second. When reading mode 1/form 1 CD-ROM data, each sector has 2048 bytes (2 kbytes), so a single-speed data rate should achieve 150 kbytes/sec.

  The Dreamcast is supposed to possess a 12X optical drive. This would imply a maximum data rate of 150 kbytes/sec * 12 = 1800 kbytes/sec.

  Rigging up a trivial experiment using the RoQ samples burned on a few different CD-R discs, the best data rate I can see is about 500-525 kbytes/sec, or around 3.5X.

  Where’s the discrepancy ? My first theory has to do with the fact that not all optical media is created equal. This is why optical drives often advertise a slew of numbers which refer to the best theoretical speed for reading a CD vs. writing a CD-R vs. writing a CD-RW, etc. Perhaps the DC drive can’t read CD-Rs very quickly. To test this theory, I tried streaming a large file from a conventionally mastered CD-ROM. This worked well for the closest CD-ROM I had on hand : I was able to stream data at a rate that works out to about 6.5X.

  I smell a science project for another evening : Profiling read speeds from a mastered CD-ROM, burned CD-R, and also a mastered GD-ROM, on each of the 3 Dreamcast consoles I possess (I’ve heard that there’s variance between optical drives depending on manufacturing run).

  The Good News
  I added a little finer-grained code to profile just the video decoding functions. The good news is that the decoder meets my real time goals : That 8.5-second RoQ sample encoded at 640x480x30fps makes its way through the video decoding functions on the DC in a little less than 5 seconds. If the optical drive can supply the data fast enough, the video decoder can take care of the rest.

  The RoQ encoder included with FFmpeg does not honor any bitrate parameters. Instead, I encoded the same file at 320x240. It reportedly decoded in real time and can be streamed in real time as well.

  I say "reportedly" because I’m simply working from textual output at this point ; the next phase is to hook the decoder up to the display hardware.

• ffmpeg issue with Jupyter notebook

  16 novembre 2022, par Nanda

  I am trying to extract frames from a video file using ffmpeg in Python. I installed ffmpeg using Homebrew and ffmpeg-python on the Anaconda-Navigator. Yet when I call ffmpeg on Jupyter notebook as follows

  


  !ffmpeg -i "$file" "$rootdir"/"$folder_name"/frame%04d.png


  


  I get an error saying

  


  zsh:1: command not found: ffmpeg


  


  I clearly see ffmpeg in my usr/local/bin. Can someone please assist me in sorting this ? I am able to use ffmpeg in Google Colab, though.

  


• Using ffmpeg to convert voice recording captured by HTML5

  8 juillet 2014, par user3789242

  I’m building an HTML5 voice recording software with visualizer.I want the user when recording the voice, and after uploading the file as wave in a blob (server-side), the user should be able to select the audio format of that file using ffmpeg. what I Have achieved so far is uploading the file as wave.what I still want to do is :

  • On the server
  • side pick your preferable web programming framework
  • The web programming framework accepts the upload and stores the file on the server
  • The web programming framework runs a ffmpeg (command line) which processes the file
  • The user can download the processed file

  here is my code so far :

  // variables
  
  var leftchannel = [];
  
  var rightchannel = [];
  
  var recorder = null;
  
  var recording = false;
  
  var recordingLength = 0;
  
  var volume = null;
  
  var audioInput = null;
  
  var sampleRate = 44100;
  
  var audioContext = null;
  
  var context = null;
  
  var outputString;
  
  
  
  
  
  
  
  if (!navigator.getUserMedia)
  
      navigator.getUserMedia = navigator.getUserMedia ||
  
      navigator.webkitGetUserMedia ||
  
      navigator.mozGetUserMedia ||
  
      navigator.msGetUserMedia;
  
  
  
  if (navigator.getUserMedia) {
  
      navigator.getUserMedia({
  
          audio: true
  
      }, success, function (e) {
  
          alert('Error capturing audio.');
  
      });
  
  } else alert('getUserMedia not supported in this browser.');
  
  
  
  
  
  
  
  function getVal(value) {
  
  
  
      // if R is pressed, we start recording
  
      if (value == "record") {
  
          recording = true;
  
          // reset the buffers for the new recording
  
          leftchannel.length = rightchannel.length = 0;
  
          recordingLength = 0;
  
          document.getElementById('output').innerHTML = "Recording now...";
  
  
  
          // if S is pressed, we stop the recording and package the WAV file
  
      } else if (value == "stop") {
  
  
  
          // we stop recording
  
          recording = false;
  
          document.getElementById('output').innerHTML = "Building wav file...";
  
  
  
          // we flat the left and right channels down
  
          var leftBuffer = mergeBuffers(leftchannel, recordingLength);
  
          var rightBuffer = mergeBuffers(rightchannel, recordingLength);
  
          // we interleave both channels together
  
          var interleaved = interleave(leftBuffer, rightBuffer);
  
  
  
  
  
  
  
          var buffer = new ArrayBuffer(44 + interleaved.length * 2);
  
          var view = new DataView(buffer);
  
  
  
          // RIFF chunk descriptor
  
          writeUTFBytes(view, 0, 'RIFF');
  
          view.setUint32(4, 44 + interleaved.length * 2, true);
  
          writeUTFBytes(view, 8, 'WAVE');
  
          // FMT sub-chunk
  
          writeUTFBytes(view, 12, 'fmt ');
  
          view.setUint32(16, 16, true);
  
          view.setUint16(20, 1, true);
  
          // stereo (2 channels)
  
          view.setUint16(22, 2, true);
  
          view.setUint32(24, sampleRate, true);
  
          view.setUint32(28, sampleRate * 4, true);
  
          view.setUint16(32, 4, true);
  
          view.setUint16(34, 16, true);
  
          // data sub-chunk
  
          writeUTFBytes(view, 36, 'data');
  
          view.setUint32(40, interleaved.length * 2, true);
  
  
  
  
  
          var lng = interleaved.length;
  
          var index = 44;
  
          var volume = 1;
  
          for (var i = 0; i < lng; i++) {
  
              view.setInt16(index, interleaved[i] * (0x7FFF * volume), true);
  
              index += 2;
  
          }
  
  
  
          var blob = new Blob([view], {
  
              type: 'audio/wav'
  
          });
  
  
  
          // let's save it locally
  
  
  
          document.getElementById('output').innerHTML = 'Handing off the file now...';
  
          var url = (window.URL || window.webkitURL).createObjectURL(blob);
  
  
  
          var li = document.createElement('li');
  
          var au = document.createElement('audio');
  
          var hf = document.createElement('a');
  
  
  
          au.controls = true;
  
          au.src = url;
  
          hf.href = url;
  
          hf.download = 'audio_recording_' + new Date().getTime() + '.wav';
  
          hf.innerHTML = hf.download;
  
          li.appendChild(au);
  
          li.appendChild(hf);
  
          recordingList.appendChild(li);
  
  
  
      }
  
  }
  
  
  
  
  
  function success(e) {
  
  
  
      audioContext = window.AudioContext || window.webkitAudioContext;
  
      context = new audioContext();
  
  
  
  
  
      volume = context.createGain();
  
  
  
      // creates an audio node from the microphone incoming stream(source)
  
      source = context.createMediaStreamSource(e);
  
  
  
      // connect the stream(source) to the gain node
  
      source.connect(volume);
  
  
  
      var bufferSize = 2048;
  
  
  
      recorder = context.createScriptProcessor(bufferSize, 2, 2);
  
  
  
      //node for the visualizer
  
      analyser = context.createAnalyser();
  
      analyser.smoothingTimeConstant = 0.3;
  
      analyser.fftSize = 512;
  
  
  
      splitter = context.createChannelSplitter();
  
      //when recording happens
  
      recorder.onaudioprocess = function (e) {
  
  
  
          if (!recording) return;
  
          var left = e.inputBuffer.getChannelData(0);
  
          var right = e.inputBuffer.getChannelData(1);
  
  
  
          leftchannel.push(new Float32Array(left));
  
          rightchannel.push(new Float32Array(right));
  
          recordingLength += bufferSize;
  
  
  
          // get the average for the first channel
  
          var array = new Uint8Array(analyser.frequencyBinCount);
  
          analyser.getByteFrequencyData(array);
  
  
  
          var c = document.getElementById("myCanvas");
  
          var ctx = c.getContext("2d");
  
          // clear the current state
  
          ctx.clearRect(0, 0, 1000, 325);
  
          var gradient = ctx.createLinearGradient(0, 0, 0, 300);
  
          gradient.addColorStop(1, '#000000');
  
          gradient.addColorStop(0.75, '#ff0000');
  
          gradient.addColorStop(0.25, '#ffff00');
  
          gradient.addColorStop(0, '#ffffff');
  
          // set the fill style
  
          ctx.fillStyle = gradient;
  
          drawSpectrum(array);
  
  
  
          function drawSpectrum(array) {
  
              for (var i = 0; i < (array.length); i++) {
  
                  var value = array[i];
  
                  ctx.fillRect(i * 5, 325 - value, 3, 325);
  
              }
  
  
  
          }
  
      }
  
  
  
      function getAverageVolume(array) {
  
          var values = 0;
  
          var average;
  
  
  
          var length = array.length;
  
  
  
          // get all the frequency amplitudes
  
          for (var i = 0; i < length; i++) {
  
              values += array[i];
  
          }
  
  
  
          average = values / length;
  
          return average;
  
      }
  
  
  
      // we connect the recorder(node to destination(speakers))
  
      volume.connect(splitter);
  
      splitter.connect(analyser, 0, 0);
  
  
  
      analyser.connect(recorder);
  
      recorder.connect(context.destination);
  
  
  
  }
  
  
  
  
  
  
  
  
  
  function mergeBuffers(channelBuffer, recordingLength) {
  
      var result = new Float32Array(recordingLength);
  
      var offset = 0;
  
      var lng = channelBuffer.length;
  
      for (var i = 0; i < lng; i++) {
  
          var buffer = channelBuffer[i];
  
          result.set(buffer, offset);
  
          offset += buffer.length;
  
      }
  
      return result;
  
  }
  
  
  
  function interleave(leftChannel, rightChannel) {
  
      var length = leftChannel.length + rightChannel.length;
  
      var result = new Float32Array(length);
  
  
  
      var inputIndex = 0;
  
  
  
      for (var index = 0; index < length;) {
  
          result[index++] = leftChannel[inputIndex];
  
          result[index++] = rightChannel[inputIndex];
  
          inputIndex++;
  
      }
  
      return result;
  
  }
  
  
  
  
  
  function writeUTFBytes(view, offset, string) {
  
      var lng = string.length;
  
      for (var i = 0; i < lng; i++) {
  
  
  
          view.setUint8(offset + i, string.charCodeAt(i));
  
      }
  
  }