Recherche avancée

Sur d’autres sites (9054)

• Sequencing MIDI From A Chiptune

  28 avril 2013, par Multimedia Mike — Outlandish Brainstorms

  The feature requests for my game music appreciation website project continue to pour in. Many of them take the form of “please add player support for system XYZ and the chiptune library to go with it.” Most of these requests are A) plausible, and B) in process. I have also received recommendations for UI improvements which I take under consideration. Then there are the numerous requests to port everything from Native Client to JavaScript so that it will work everywhere, even on mobile, a notion which might take a separate post to debunk entirely.

  But here’s an interesting request about which I would like to speculate : Automatically convert a chiptune into a MIDI file. I immediately wanted to dismiss it as impossible or highly implausible. But, as is my habit, I started pondering the concept a little more carefully and decided that there’s an outside chance of getting some part of the idea to work.

  Intro to MIDI
  MIDI stands for Musical Instrument Digital Interface. It’s a standard musical interchange format and allows music instruments and computers to exchange musical information. The file interchange format bears the extension .mid and contains a sequence of numbers that translate into commands separated by time deltas. E.g. : turn key on (this note, this velocity) ; wait x ticks ; turn key off ; wait y ticks ; etc. I’m vastly oversimplifying, as usual.

  MIDI fascinated me back in the days of dialup internet and discrete sound cards (see also my write-up on the Gravis Ultrasound). Typical song-length MIDI files often ranged from a few kilobytes to a few 10s of kilobytes. They were significantly smaller than the MOD et al. family of tracker music formats mostly by virtue of the fact that MIDI files aren’t burdened by transporting digital audio samples.
  
  I know I’m missing a lot of details. I haven’t dealt much with MIDI in the past… 15 years or so (ever since computer audio became a blur of MP3 and AAC audio). But I’m led to believe it’s still relevant. The individual who requested this feature expressed an interest in being able to import the sequenced data into any of the many music programs that can interpret .mid files.

  The Pitch
  To limit the scope, let’s focus on music that comes from the 8-bit Nintendo Entertainment System or the original Game Boy. The former features 2 square wave channels, a triangle wave, a noise channel, and a limited digital channel. The latter creates music via 2 square waves, a wave channel, and a noise channel. The roles that these various channels usually play typically break down as : square waves represent the primary melody, triangle wave is used to simulate a bass line, noise channel approximates a variety of percussive sounds, and the DPCM/wave channels are fairly free-form. They can have random game sound effects or, if they are to assist in the music, are often used for more authentic percussive sounds.

  The various channels are controlled via an assortment of memory-mapped hardware registers. These registers are fed values such as frequency, volume, and duty cycle. My idea is to modify the music playback engine to track when various events occur. Whenever a channel is turned on or off, that corresponds to a MIDI key on or off event. If a channel is already playing but a new frequency is written, that would likely count as a note change, so log a key off event followed by a new key on event.

  There is the major obstacle of what specific note is represented by a channel in a particular state. The MIDI standard defines 128 different notes spanning 11 octaves. Empirically, I wonder if I could create a table which maps the assorted frequencies to different MIDI notes ?

  I think this strategy would only work with the square and triangle waves. Noise and digital channels ? I’m not prepared to tackle that challenge.

  Prior Work ?
  I have to wonder if there is any existing work in this area. I’m certain that people have wanted to do this before ; I wonder if anyone has succeeded ?

  Just like reverse engineering a binary program entails trying to obtain a higher level abstraction of a program from a very low level representation, this challenge feels like reverse engineering a piece of music as it is being performed and automatically expressing it in a higher level form.

• Compute PTS and DTS correctly to sync audio and video ffmpeg C++

  14 août 2015, par Kaidul Islam

  I am trying to mux H264 encoded data and G711 PCM data into mov multimedia container. I am creating AVPacket from encoded data and initially the PTS and DTS value of video/audio frames is equivalent to AV_NOPTS_VALUE. So I calculated the DTS using current time information. My code -

  bool AudioVideoRecorder::WriteVideo(const unsigned char *pData, size_t iDataSize, bool const bIFrame) {
  
      .....................................
  
      .....................................
  
      .....................................
  
      AVPacket pkt = {0};
  
      av_init_packet(&pkt);
  
      int64_t dts = av_gettime();
  
      dts = av_rescale_q(dts, (AVRational){1, 1000000}, m_pVideoStream->time_base);
  
      int duration = 90000 / VIDEO_FRAME_RATE;
  
      if(m_prevVideoDts > 0LL) {
  
          duration = dts - m_prevVideoDts;
  
      }
  
      m_prevVideoDts = dts;
  
  
  
      pkt.pts = AV_NOPTS_VALUE;
  
      pkt.dts = m_currVideoDts;
  
      m_currVideoDts += duration;
  
      pkt.duration = duration;
  
      if(bIFrame) {
  
          pkt.flags |= AV_PKT_FLAG_KEY;
  
      }
  
      pkt.stream_index = m_pVideoStream->index;
  
      pkt.data = (uint8_t*) pData;
  
      pkt.size = iDataSize;
  
  
  
      int ret = av_interleaved_write_frame(m_pFormatCtx, &pkt);
  
  
  
      if(ret < 0) {
  
          LogErr("Writing video frame failed.");
  
          return false;
  
      }
  
  
  
      Log("Writing video frame done.");
  
  
  
      av_free_packet(&pkt);
  
      return true;
  
  }
  
  
  
  bool AudioVideoRecorder::WriteAudio(const unsigned char *pEncodedData, size_t iDataSize) {
  
      .................................
  
      .................................
  
      .................................
  
      AVPacket pkt = {0};
  
      av_init_packet(&pkt);
  
  
  
      int64_t dts = av_gettime();
  
      dts = av_rescale_q(dts, (AVRational){1, 1000000}, (AVRational){1, 90000});
  
      int duration = AUDIO_STREAM_DURATION; // 20
  
      if(m_prevAudioDts > 0LL) {
  
          duration = dts - m_prevAudioDts;
  
      }
  
      m_prevAudioDts = dts;
  
      pkt.pts = AV_NOPTS_VALUE;
  
      pkt.dts = m_currAudioDts;
  
      m_currAudioDts += duration;
  
      pkt.duration = duration;
  
  
  
      pkt.stream_index = m_pAudioStream->index;
  
      pkt.flags |= AV_PKT_FLAG_KEY;
  
      pkt.data = (uint8_t*) pEncodedData;
  
      pkt.size = iDataSize;
  
  
  
      int ret = av_interleaved_write_frame(m_pFormatCtx, &pkt);
  
      if(ret < 0) {
  
          LogErr("Writing audio frame failed: %d", ret);
  
          return false;
  
      }
  
  
  
      Log("Writing audio frame done.");
  
  
  
      av_free_packet(&pkt);
  
      return true;
  
  }

  And I added stream like this -

  AVStream* AudioVideoRecorder::AddMediaStream(enum AVCodecID codecID) {
  
      ................................
  
      .................................   
  
      pStream = avformat_new_stream(m_pFormatCtx, codec);
  
      if (!pStream) {
  
          LogErr("Could not allocate stream.");
  
          return NULL;
  
      }
  
      pStream->id = m_pFormatCtx->nb_streams - 1;
  
      pCodecCtx = pStream->codec;
  
      pCodecCtx->codec_id = codecID;
  
  
  
      switch(codec->type) {
  
      case AVMEDIA_TYPE_VIDEO:
  
          pCodecCtx->bit_rate = VIDEO_BIT_RATE;
  
          pCodecCtx->width = PICTURE_WIDTH;
  
          pCodecCtx->height = PICTURE_HEIGHT;
  
          pStream->time_base = (AVRational){1, 90000};
  
          pStream->avg_frame_rate = (AVRational){90000, 1};
  
          pStream->r_frame_rate = (AVRational){90000, 1}; // though the frame rate is variable and around 15 fps
  
          pCodecCtx->pix_fmt = STREAM_PIX_FMT;
  
          m_pVideoStream = pStream;
  
          break;
  
  
  
      case AVMEDIA_TYPE_AUDIO:
  
          pCodecCtx->sample_fmt = AV_SAMPLE_FMT_S16;
  
          pCodecCtx->bit_rate = AUDIO_BIT_RATE;
  
          pCodecCtx->sample_rate = AUDIO_SAMPLE_RATE;
  
          pCodecCtx->channels = 1;
  
          m_pAudioStream = pStream;
  
          break;
  
  
  
      default:
  
          break;
  
      }
  
  
  
      /* Some formats want stream headers to be separate. */
  
      if (m_pOutputFmt->flags & AVFMT_GLOBALHEADER)
  
          m_pFormatCtx->flags |= CODEC_FLAG_GLOBAL_HEADER;
  
  
  
      return pStream;
  
  }

  There are several problems with this calculation :

  1. The video is laggy and lags behind than audio increasingly with time.

  2. Suppose, an audio frame is received (WriteAudio(..)) little lately like 3 seconds, then the late frame should be started playing with 3 second delay, but it’s not. The delayed frame is played consecutively with previous frame.

  3. Sometimes I recorded for 40 seconds but the file duration is much like 2 minutes, but audio/video is played only few moments like 40 seconds and rest of the file contains nothing and seekbar jumps at en immediately after 40 seconds (tested in VLC).

  EDIT :

  According to Ronald S. Bultje’s suggestion, what I’ve understand :

  m_pAudioStream->time_base = (AVRational){1, 9000}; // actually no need to set as 9000 is already default value for audio as you said
  
  m_pVideoStream->time_base = (AVRational){1, 9000};

  should be set as now both audio and video streams are now in same time base units.

  And for video :

  ...................
  
  ...................
  
  
  
  int64_t dts = av_gettime(); // get current time in microseconds
  
  dts *= 9000; 
  
  dts /= 1000000; // 1 second = 10^6 microseconds
  
  pkt.pts = AV_NOPTS_VALUE; // is it okay?
  
  pkt.dts = dts;
  
  // and no need to set pkt.duration, right?

  And for audio : (exactly same as video, right ?)

  ...................
  
  ...................
  
  
  
  int64_t dts = av_gettime(); // get current time in microseconds
  
  dts *= 9000; 
  
  dts /= 1000000; // 1 second = 10^6 microseconds
  
  pkt.pts = AV_NOPTS_VALUE; // is it okay?
  
  pkt.dts = dts;
  
  // and no need to set pkt.duration, right?

  And I think they are now like sharing same currDts, right ? Please correct me if I am wrong anywhere or missing anything.

  Also, if I want to use video stream time base as (AVRational){1, frameRate} and audio stream time base as (AVRational){1, sampleRate}, how the correct code should look like ?

  EDIT 2.0 :

  m_pAudioStream->time_base = (AVRational){1, VIDEO_FRAME_RATE};
  
  m_pVideoStream->time_base = (AVRational){1, VIDEO_FRAME_RATE};

  And

  bool AudioVideoRecorder::WriteAudio(const unsigned char *pEncodedData, size_t iDataSize) {
  
      ...........................
  
      ......................
  
      AVPacket pkt = {0};
  
      av_init_packet(&pkt);
  
  
  
      int64_t dts = av_gettime() / 1000; // convert into millisecond
  
      dts = dts * VIDEO_FRAME_RATE;
  
      if(m_dtsOffset < 0) {
  
          m_dtsOffset = dts;
  
      }
  
  
  
      pkt.pts = AV_NOPTS_VALUE;
  
      pkt.dts = (dts - m_dtsOffset);
  
  
  
      pkt.stream_index = m_pAudioStream->index;
  
      pkt.flags |= AV_PKT_FLAG_KEY;
  
      pkt.data = (uint8_t*) pEncodedData;
  
      pkt.size = iDataSize;
  
  
  
      int ret = av_interleaved_write_frame(m_pFormatCtx, &pkt);
  
      if(ret < 0) {
  
          LogErr("Writing audio frame failed: %d", ret);
  
          return false;
  
      }
  
  
  
      Log("Writing audio frame done.");
  
  
  
      av_free_packet(&pkt);
  
      return true;
  
  }
  
  
  
  bool AudioVideoRecorder::WriteVideo(const unsigned char *pData, size_t iDataSize, bool const bIFrame) {
  
      ........................................
  
      .................................
  
      AVPacket pkt = {0};
  
      av_init_packet(&pkt);
  
      int64_t dts = av_gettime() / 1000;
  
      dts = dts * VIDEO_FRAME_RATE;
  
      if(m_dtsOffset < 0) {
  
          m_dtsOffset = dts;
  
      }
  
      pkt.pts = AV_NOPTS_VALUE;
  
      pkt.dts = (dts - m_dtsOffset);
  
  
  
      if(bIFrame) {
  
          pkt.flags |= AV_PKT_FLAG_KEY;
  
      }
  
      pkt.stream_index = m_pVideoStream->index;
  
      pkt.data = (uint8_t*) pData;
  
      pkt.size = iDataSize;
  
  
  
      int ret = av_interleaved_write_frame(m_pFormatCtx, &pkt);
  
  
  
      if(ret < 0) {
  
          LogErr("Writing video frame failed.");
  
          return false;
  
      }
  
  
  
      Log("Writing video frame done.");
  
  
  
      av_free_packet(&pkt);
  
      return true;
  
  }

  Is the last change okay ? The video and audio seems synced. Only problem is - the audio is played without the delay regardless the packet arrived in delay.
  Like -

  packet arrival : 1 2 3 4... (then next frame arrived after 3 sec) .. 5

  audio played : 1 2 3 4 (no delay) 5

  EDIT 3.0 :

  zeroed audio sample data :

  AVFrame* pSilentData;
  
  pSilentData = av_frame_alloc();
  
  memset(&pSilentData->data[0], 0, iDataSize);
  
  
  
  pkt.data = (uint8_t*) pSilentData;
  
  pkt.size = iDataSize;
  
  
  
  av_freep(&pSilentData->data[0]);
  
  av_frame_free(&pSilentData);

  Is this okay ? But after writing this into file container, there are dot dot noise during playing the media. Whats the problem ?

  EDIT 4.0 :

  Well, For µ-Law audio the zero value is represented as 0xff. So -

  memset(&pSilentData->data[0], 0xff, iDataSize);

  solve my problem.

• How to remove the background sound of sport commentary sound clip in ffmpeg

  28 juillet 2015, par user0001

  I’m working on a project which needs to convert sport commentary to text. For that I have already used Microsoft system speech library. It’s working fine without background noises. Can any one tell me a way of removing this background noise from the given audio file by using ffmpeg-like tool or in some other programmatic way.