Recherche avancée

Sur d’autres sites (6737)

• Emscripten and Web Audio API

  29 avril 2015, par Multimedia Mike — HTML5

  Ha ! They said it couldn’t be done ! Well, to be fair, I said it couldn’t be done. Or maybe that I just didn’t have any plans to do it. But I did it– I used Emscripten to cross-compile a CPU-intensive C/C++ codebase (Game Music Emu) to JavaScript. Then I leveraged the Web Audio API to output audio and visualize the audio using an HTML5 canvas.

  Want to see it in action ? Here’s a demonstration. Perhaps I will be able to expand the reach of my Game Music site when I can drop the odd Native Client plugin. This JS-based player works great on Chrome, Firefox, and Safari across desktop operating systems.

  But this endeavor was not without its challenges.

  Programmatically Generating Audio
  First, I needed to figure out the proper method for procedurally generating audio and making it available to output. Generally, there are 2 approaches for audio output :

  1. Sit in a loop and generate audio, writing it out via a blocking audio call
  2. Implement a callback that the audio system can invoke in order to generate more audio when needed

  Option #1 is not a good idea for an event-driven language like JavaScript. So I hunted through the rather flexible Web Audio API for a method that allowed something like approach #2. Callbacks are everywhere, after all.

  I eventually found what I was looking for with the ScriptProcessorNode. It seems to be intended to apply post-processing effects to audio streams. A program registers a callback which is passed configurable chunks of audio for processing. I subverted this by simply overwriting the input buffers with the audio generated by the Emscripten-compiled library.

  The ScriptProcessorNode interface is fairly well documented and works across multiple browsers. However, it is already marked as deprecated :

  Note : As of the August 29 2014 Web Audio API spec publication, this feature has been marked as deprecated, and is soon to be replaced by Audio Workers.

  Despite being marked as deprecated for 8 months as of this writing, there exists no appreciable amount of documentation for the successor API, these so-called Audio Workers.

  Vive la web standards !

  Visualize This
  The next problem was visualization. The Web Audio API provides the AnalyzerNode API for accessing both time and frequency domain data from a running audio stream (and fetching the data as both unsigned bytes or floating-point numbers, depending on what the application needs). This is a pretty neat idea. I just wish I could make the API work. The simple demos I could find worked well enough. But when I wired up a prototype to fetch and visualize the time-domain wave, all I got were center-point samples (an array of values that were all 128).

  Even if the API did work, I’m not sure if it would have been that useful. Per my reading of the AnalyserNode API, it only returns data as a single channel. Why would I want that ? My application supports audio with 2 channels. I want 2 channels of data for visualization.

  How To Synchronize
  So I rolled my own visualization solution by maintaining a circular buffer of audio when samples were being generated. Then, requestAnimationFrame() provided the rendering callbacks. The next problem was audio-visual sync. But that certainly is not unique to this situation– maintaining proper A/V sync is a perennial puzzle in real-time multimedia programming. I was able to glean enough timing information from the environment to achieve reasonable A/V sync (verify for yourself).

  Pause/Resume
  The next problem I encountered with the Web Audio API was pause/resume facilities, or the lack thereof. For all its bells and whistles, the API’s omission of such facilities seems most unusual, as if the design philosophy was, “Once the user starts playing audio, they will never, ever have cause to pause the audio.”

  Then again, I must understand that mine is not a use case that the design committee considered and I’m subverting the API in ways the designers didn’t intend. Typical use cases for this API seem to include such workloads as :

  • Downloading, decoding, and playing back a compressed audio stream via the network, applying effects, and visualizing the result
  • Accessing microphone input, applying effects, visualizing, encoding and sending the data across the network
  • Firing sound effects in a gaming application
  • MIDI playback via JavaScript (this honestly amazes me)

  What they did not seem to have in mind was what I am trying to do– synthesize audio in real time.

  I implemented pause/resume in a sub-par manner : pausing has the effect of generating 0 values when the ScriptProcessorNode callback is invoked, while also canceling any animation callbacks. Thus, audio output is technically still occurring, it’s just that the audio is pure silence. It’s not a great solution because CPU is still being used.

  Future Work
  I have a lot more player libraries to port to this new system. But I think I have a good framework set up.

• send h264 video to nginx-rtmp server using ffmpeg API

  11 décembre 2019, par Glen

  I have C++ code that grabs frames from a GigE camera and writes them out to a file. I’m using the libx264 codec and ffmpeg version 4.0.

  Writing to the file works fine, however I would also like to send the video to nginx configured with the nginx-rtmp plug-in to make the video available live via HLS.

  I can use the ffmpeg command line program to stream one of my previously captured files to my nginx server and rebroadcast as HLS, however if I try to stream from my C++ code the nginx server closes the connection after one or two frames are sent.

  To test further, I used the ffmpeg command line program to receive a rtmp stream and write it out to a file. I am able to send video to ffmpeg from my C++ program with rtmp, however every frame generates a warning like this :

  [avi @ 0x1b6b6f0] Non-monotonous DTS in output stream 0:0; previous: 1771, current: 53; changing to 1772. This may result in incorrect timestamps in the output file.
  
  [avi @ 0x1b6b6f0] Non-monotonous DTS in output stream 0:0; previous: 1772, current: 53; changing to 1773. This may result in incorrect timestamps in the output file.
  
  [avi @ 0x1b6b6f0] Non-monotonous DTS in output stream 0:0; previous: 1773, current: 53; changing to 1774. This may result in incorrect timestamps in the output file.
  
  [avi @ 0x1b6b6f0] Non-monotonous DTS in output stream 0:0; previous: 1774, current: 53; changing to 1775. This may result in incorrect timestamps in the output file.
  
  [avi @ 0x1b6b6f0] Non-monotonous DTS in output stream 0:0; previous: 1775, current: 53; changing to 1776. This may result in incorrect timestamps in the output file.
  
  [avi @ 0x1b6b6f0] Non-monotonous DTS in output stream 0:0; previous: 1776, current: 53; changing to 1777. This may result in incorrect timestamps in the output file.
  
  [avi @ 0x1b6b6f0] Non-monotonous DTS in output stream 0:0; previous: 1777, current: 53; changing to 1778. This may result in incorrect timestamps in the output file.
  
  [avi @ 0x1b6b6f0] Non-monotonous DTS in output stream 0:0; previous: 1778, current: 53; changing to 1779. This may result in incorrect timestamps in the output file.
  
  [avi @ 0x1b6b6f0] Non-monotonous DTS in output stream 0:0; previous: 1779, current: 53; changing to 1780. This may result in incorrect timestamps in the output file.

  I printed PTS and DTS for my packet before writing it, and the numbers were monotonous (for example, in this last frame the pts and dts printed from my code were 1780, not the ’current : 53’ that ffmpeg reports>

  also, unless I tell ffmpeg what the output framerate should be I end up with a file that plays 2x speed.

  After ffmpeg receives the rtmp stream and writes it to the file, I am then able to successfully send that file to my nginx server using ffmpeg.

  here is some relevant code :

  //configuring the codec context
  
  // make sure that config.codec is something we support
  
  // for now we are only supporting LIBX264
  
  if (config.codec() != codecs::LIBX264) {
  
      throw std::invalid_argument("currently only libx264 codec is supported");
  
  }
  
  
  
  // lookup specified codec
  
  ffcodec_ = avcodec_find_encoder_by_name(config.codec().c_str());
  
  if (!ffcodec_) {
  
      throw std::invalid_argument("unable to get codec " + config.codec());
  
  }
  
  
  
  // unique_ptr to manage the codec_context
  
  codec_context_ = av_pointer::codec_context(avcodec_alloc_context3(ffcodec_));
  
  
  
  if (!codec_context_) {
  
      throw std::runtime_error("unable to initialize AVCodecContext");
  
  }
  
  
  
  // setup codec_context_
  
  codec_context_->width = frame_width;
  
  codec_context_->height = frame_height;
  
  codec_context_->time_base = (AVRational){1, config.target_fps()};
  
  codec_context_->framerate = (AVRational){config.target_fps(), 1};
  
  codec_context_->global_quality = 0;
  
  codec_context_->compression_level = 0;
  
  codec_context_->bits_per_raw_sample = 8;
  
  codec_context_->gop_size = 1;
  
  codec_context_->max_b_frames = 1;
  
  codec_context_->pix_fmt = AV_PIX_FMT_YUV420P;
  
  
  
  // x264 only settings
  
  if (config.codec() == codecs::LIBX264) {
  
      av_opt_set(codec_context_->priv_data, "preset", config.compression_target().c_str(), 0);
  
      av_opt_set(codec_context_->priv_data, "crf", std::to_string(config.crf()).c_str(), 0);
  
  }
  
  
  
  // Open up the codec
  
  if (avcodec_open2(codec_context_.get(), ffcodec_, NULL) < 0) {
  
      throw std::runtime_error("unable to open ffmpeg codec");
  
  }
  
  
  
  // setup the output format context and stream for RTMP
  
  AVFormatContext *tmp_f_context;
  
  avformat_alloc_output_context2(&tmp_f_context, NULL, "flv", uri.c_str());
  
  rtmp_format_context_ = av_pointer::format_context(tmp_f_context);
  
  rtmp_stream_ = avformat_new_stream(rtmp_format_context_.get(), ffcodec_);
  
  avcodec_parameters_from_context(rtmp_stream_->codecpar, codec_context_.get());
  
  rtmp_stream_->time_base = codec_context_->time_base;
  
  rtmp_stream_->r_frame_rate = codec_context_->framerate;
  
  
  
  /* open the output file */
  
  if (!(rtmp_format_context_->flags & AVFMT_NOFILE)) {
  
      int r = avio_open(&rtmp_format_context_->pb, uri.c_str(), AVIO_FLAG_WRITE);
  
      if (r < 0) {
  
          throw std::runtime_error("unable to open " + uri + " : " + av_err2str(r));
  
      }
  
  }
  
  
  
  if (avformat_write_header(rtmp_format_context_.get(), NULL) < 0) {
  
      throw std::runtime_error("unable to write header");
  
  }
  
  
  
  av_dump_format(rtmp_format_context_.get(), 0,uri.c_str() , 1);

  at this point the av_dump_format produces this output :

  Output #0, flv, to 'rtmp://[MY URI]':
  
    Metadata:
  
      encoder         : Lavf58.12.100
  
      Stream #0:0, 0, 1/1000: Video: h264 (libx264), 1 reference frame ([7][0][0][0] / 0x0007), yuv420p, 800x800 (0x0), 0/1, q=-1--1, 30 tbr, 1k tbn

  encoding and writing the frame :

  // send the frame to the encoder, filtering first if necessary
  
  void VideoWriter::Encode(AVFrame *frame)
  
  {
  
      int rval;
  
      if (!apply_filter_) {
  
          //send frame to encoder
  
          rval = avcodec_send_frame(codec_context_.get(), frame);
  
          if (rval < 0) {
  
              throw std::runtime_error("error sending frame for encoding");
  
          }
  
      } else {
  
          // push frame to filter 
  
          // REMOVED, currently testing without filtering
  
      }
  
  
  
      // get packets from encoder
  
      while (rval >= 0) {
  
          // create smart pointer to allocated packet
  
          av_pointer::packet pkt(av_packet_alloc());
  
          if (!pkt) {
  
              throw std::runtime_error("unable to allocate packet");
  
          }
  
  
  
          rval = avcodec_receive_packet(codec_context_.get(), pkt.get());
  
          if (rval == AVERROR(EAGAIN) || rval == AVERROR_EOF) {
  
              return;
  
          } else if (rval < 0) {
  
              throw std::runtime_error("error during encoding");
  
          }
  
  
  
          // if I print pkt->pts and pkt->dts here, I see sequential numbers
  
  
  
          // write packet
  
          rval = av_interleaved_write_frame(rtmp_format_context_.get(), pkt.get());
  
          if (rval < 0 ) {
  
              std::cerr << av_err2str(rval) << std::endl;
  
          }
  
      }
  
  }

  Since I am able to send video from a previously recorded file to nginx with the ffmpeg command line program, I believe the problem is in my code and not my nginx configuration.

  EDIT : I think it may have to do with SPS/PPS as I see a bunch of these error messages in the nginx log before it closes the stream

  2019/12/11 11:11:31 [error] 10180#0: *4 hls: failed to read 5 byte(s), client: XXX, server: 0.0.0.0:1935
  
  2019/12/11 11:11:31 [error] 10180#0: *4 hls: error appenging SPS/PPS NALs, client: XXX, server: 0.0.0.0:1935

  As I mentioned, this code works fine if I set it up to write to an avi file rather stream to rtmp, and I can stream to ffmpeg listening for rtmp but with lots of warnings about the DTS but if I try to send to nginx, it closes the connection almost immediately. My first thought was that there was something wrong with the frame timestamps, but when I print pts and dts prior to writing the packet to the stream they look okay to me.

  My end goal is to capture video to a file, and also be able to turn on the rtmp stream on demand — but for now I’m just trying to get the rtmp stream working continuously (without writing to a file)

  Thanks for any insights.

• What ffmpeg command line will list the key frames ?

  21 décembre 2019, par ChrisJJ

  What ffmpeg command line will list the index numbers of the key frames in an existing .MP4 to stdout ?

  I’ve tried the only applicable answer at https://superuser.com/questions/885452/extracting-the-index-of-key-frames-from-a-video-using-ffmpeg to no avail.