Recherche avancée

Sur d’autres sites (16828)

• streaming H.264 over RTP with libavformat

  16 avril 2012, par Jacob Peddicord

  I've been trying over the past week to implement H.264 streaming over RTP, using x264 as an encoder and libavformat to pack and send the stream. Problem is, as far as I can tell it's not working correctly.

  Right now I'm just encoding random data (x264_picture_alloc) and extracting NAL frames from libx264. This is fairly simple :

  x264_picture_t pic_out;
  
  x264_nal_t* nals;
  
  int num_nals;
  
  int frame_size = x264_encoder_encode(this->encoder, &nals, &num_nals, this->pic_in, &pic_out);
  
  
  
  if (frame_size <= 0)
  
  {
  
      return frame_size;
  
  }
  
  
  
  // push NALs into the queue
  
  for (int i = 0; i < num_nals; i++)
  
  {
  
      // create a NAL storage unit
  
      NAL nal;
  
      nal.size = nals[i].i_payload;
  
      nal.payload = new uint8_t[nal.size];
  
      memcpy(nal.payload, nals[i].p_payload, nal.size);
  
  
  
      // push the storage into the NAL queue
  
      {
  
          // lock and push the NAL to the queue
  
          boost::mutex::scoped_lock lock(this->nal_lock);
  
          this->nal_queue.push(nal);
  
      }
  
  }

  nal_queue is used for safely passing frames over to a Streamer class which will then send the frames out. Right now it's not threaded, as I'm just testing to try to get this to work. Before encoding individual frames, I've made sure to initialize the encoder.

  But I don't believe x264 is the issue, as I can see frame data in the NALs it returns back.
  Streaming the data is accomplished with libavformat, which is first initialized in a Streamer class :

  Streamer::Streamer(Encoder* encoder, string rtp_address, int rtp_port, int width, int height, int fps, int bitrate)
  
  {
  
      this->encoder = encoder;
  
  
  
      // initalize the AV context
  
      this->ctx = avformat_alloc_context();
  
      if (!this->ctx)
  
      {
  
          throw runtime_error("Couldn't initalize AVFormat output context");
  
      }
  
  
  
      // get the output format
  
      this->fmt = av_guess_format("rtp", NULL, NULL);
  
      if (!this->fmt)
  
      {
  
          throw runtime_error("Unsuitable output format");
  
      }
  
      this->ctx->oformat = this->fmt;
  
  
  
      // try to open the RTP stream
  
      snprintf(this->ctx->filename, sizeof(this->ctx->filename), "rtp://%s:%d", rtp_address.c_str(), rtp_port);
  
      if (url_fopen(&(this->ctx->pb), this->ctx->filename, URL_WRONLY) < 0)
  
      {
  
          throw runtime_error("Couldn't open RTP output stream");
  
      }
  
  
  
      // add an H.264 stream
  
      this->stream = av_new_stream(this->ctx, 1);
  
      if (!this->stream)
  
      {
  
          throw runtime_error("Couldn't allocate H.264 stream");
  
      }
  
  
  
      // initalize codec
  
      AVCodecContext* c = this->stream->codec;
  
      c->codec_id = CODEC_ID_H264;
  
      c->codec_type = AVMEDIA_TYPE_VIDEO;
  
      c->bit_rate = bitrate;
  
      c->width = width;
  
      c->height = height;
  
      c->time_base.den = fps;
  
      c->time_base.num = 1;
  
  
  
      // write the header
  
      av_write_header(this->ctx);
  
  }

  This is where things seem to go wrong. av_write_header above seems to do absolutely nothing ; I've used wireshark to verify this. For reference, I use Streamer streamer(&enc, "10.89.6.3", 49990, 800, 600, 30, 40000); to initialize the Streamer instance, with enc being a reference to an Encoder object used to handle x264 previously.

  Now when I want to stream out a NAL, I use this :

  // grab a NAL
  
  NAL nal = this->encoder->nal_pop();
  
  cout << "NAL popped with size " << nal.size << endl;
  
  
  
  // initalize a packet
  
  AVPacket p;
  
  av_init_packet(&p);
  
  p.data = nal.payload;
  
  p.size = nal.size;
  
  p.stream_index = this->stream->index;
  
  
  
  // send it out
  
  av_write_frame(this->ctx, &p);

  At this point, I can see RTP data appearing over the network, and it looks like the frames I've been sending, even including a little copyright blob from x264. But, no player I've used has been able to make any sense of the data. VLC quits wanting an SDP description, which apparently isn't required.

  I then tried to play it through gst-launch :

  gst-launch udpsrc port=49990 ! rtph264depay ! decodebin ! xvimagesink

  This will sit waiting for UDP data, but when it is received, I get :

  ERROR : element /GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0 : No RTP
  format was negotiated. Additional debug info :
  gstbasertpdepayload.c(372) : gst_base_rtp_depayload_chain () :
  /GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0 : Input buffers
  need to have RTP caps set on them. This is usually achieved by setting
  the 'caps' property of the upstream source element (often udpsrc or
  appsrc), or by putting a capsfilter element before the depayloader and
  setting the 'caps' property on that. Also see
  http://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/gst/rtp/README

  As I'm not using GStreamer to stream itself, I'm not quite sure what it means with RTP caps. But, it makes me wonder if I'm not sending enough information over RTP to describe the stream. I'm pretty new to video and I feel like there's some key thing I'm missing here. Any hints ?

• FFMpeg copy stream SEI type 162 error

  13 juillet 2015, par Vishal Shah

  I am using FFMpeg to copy an RTSP stream to an RTMP stream. Both of which are network streams. This is the command I am using.

  ffmpeg -re -i rtsp ://url.to/rtsp/stream -c copy -f flv rtmp ://target.rtmp/uri/stream

  I am receiving data on the youtube live control room. But After a point I get an error like this in my FFMpeg console and then the data transfer on youtube live stops.

  SEI type 162 size 1496 truncated at 1280

  Can someone throw light on what this error means and how to resolve it ?

  Note that the RTSP stream is over the network

• How to play raw h264 produced by MediaCodec encoder ?

  1er novembre 2014, par jackos2500

  I’m a bit new when it comes to MediaCodec (and video encoding/decoding in general), so correct me if anything I say here is wrong.

  I want to play the raw h264 output of MediaCodec with VLC/ffplay. I need this to play becuase my end goal is to stream some live video to a computer, and MediaMuxer only produces a file on disk rather than something I can stream with (very) low latency to a desktop. (I’m open to other solutions, but I have not found anything else that fits the latency requirement)

  Here is the code I’m using encode the video and write it to a file : (it’s based off the MediaCodec example found here, only with the MediaMuxer part removed)

  package com.jackos2500.droidtop;
  
  
  
  import android.media.MediaCodec;
  
  import android.media.MediaCodecInfo;
  
  import android.media.MediaFormat;
  
  import android.opengl.EGL14;
  
  import android.opengl.EGLConfig;
  
  import android.opengl.EGLContext;
  
  import android.opengl.EGLDisplay;
  
  import android.opengl.EGLExt;
  
  import android.opengl.EGLSurface;
  
  import android.opengl.GLES20;
  
  import android.os.Environment;
  
  import android.util.Log;
  
  import android.view.Surface;
  
  
  
  import java.io.BufferedOutputStream;
  
  import java.io.File;
  
  import java.io.FileOutputStream;
  
  import java.io.IOException;
  
  import java.nio.ByteBuffer;
  
  
  
  public class StreamH264 {
  
      private static final String TAG = "StreamH264";
  
      private static final boolean VERBOSE = true;           // lots of logging
  
  
  
      // where to put the output file (note: /sdcard requires WRITE_EXTERNAL_STORAGE permission)
  
      private static final File OUTPUT_DIR = Environment.getExternalStorageDirectory();
  
  
  
      public static int MEGABIT = 1000 * 1000;
  
      private static final int IFRAME_INTERVAL = 10;
  
  
  
      private static final int TEST_R0 = 0;
  
      private static final int TEST_G0 = 136;
  
      private static final int TEST_B0 = 0;
  
      private static final int TEST_R1 = 236;
  
      private static final int TEST_G1 = 50;
  
      private static final int TEST_B1 = 186;
  
  
  
      private MediaCodec codec;
  
      private CodecInputSurface inputSurface;
  
      private BufferedOutputStream out;
  
  
  
      private MediaCodec.BufferInfo bufferInfo;
  
      public StreamH264() {
  
  
  
      }
  
  
  
      private void prepareEncoder() throws IOException {
  
          bufferInfo = new MediaCodec.BufferInfo();
  
  
  
          MediaFormat format = MediaFormat.createVideoFormat("video/avc", 1280, 720);
  
          format.setInteger(MediaFormat.KEY_BIT_RATE, 2 * MEGABIT);
  
          format.setInteger(MediaFormat.KEY_FRAME_RATE, 30);
  
          format.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
  
          format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, IFRAME_INTERVAL);
  
  
  
          codec = MediaCodec.createEncoderByType("video/avc");
  
          codec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
  
          inputSurface = new CodecInputSurface(codec.createInputSurface());
  
          codec.start();
  
  
  
          File dst = new File(OUTPUT_DIR, "test.264");
  
          out = new BufferedOutputStream(new FileOutputStream(dst));
  
      }
  
      private void releaseEncoder() throws IOException {
  
          if (VERBOSE) Log.d(TAG, "releasing encoder objects");
  
          if (codec != null) {
  
              codec.stop();
  
              codec.release();
  
              codec = null;
  
          }
  
          if (inputSurface != null) {
  
              inputSurface.release();
  
              inputSurface = null;
  
          }
  
          if (out != null) {
  
              out.flush();
  
              out.close();
  
              out = null;
  
          }
  
      }
  
      public void stream() throws IOException {
  
          try {
  
              prepareEncoder();
  
              inputSurface.makeCurrent();
  
              for (int i = 0; i < (30 * 5); i++) {
  
                  // Feed any pending encoder output into the file.
  
                  drainEncoder(false);
  
  
  
                  // Generate a new frame of input.
  
                  generateSurfaceFrame(i);
  
                  inputSurface.setPresentationTime(computePresentationTimeNsec(i, 30));
  
  
  
                  // Submit it to the encoder.  The eglSwapBuffers call will block if the input
  
                  // is full, which would be bad if it stayed full until we dequeued an output
  
                  // buffer (which we can't do, since we're stuck here).  So long as we fully drain
  
                  // the encoder before supplying additional input, the system guarantees that we
  
                  // can supply another frame without blocking.
  
                  if (VERBOSE) Log.d(TAG, "sending frame " + i + " to encoder");
  
                  inputSurface.swapBuffers();
  
              }
  
              // send end-of-stream to encoder, and drain remaining output
  
              drainEncoder(true);
  
          } finally {
  
              // release encoder, muxer, and input Surface
  
              releaseEncoder();
  
          }
  
      }
  
  
  
      private void drainEncoder(boolean endOfStream) throws IOException {
  
          final int TIMEOUT_USEC = 10000;
  
          if (VERBOSE) Log.d(TAG, "drainEncoder(" + endOfStream + ")");
  
  
  
          if (endOfStream) {
  
              if (VERBOSE) Log.d(TAG, "sending EOS to encoder");
  
              codec.signalEndOfInputStream();
  
          }
  
          ByteBuffer[] outputBuffers = codec.getOutputBuffers();
  
          while (true) {
  
              int encoderStatus = codec.dequeueOutputBuffer(bufferInfo, TIMEOUT_USEC);
  
              if (encoderStatus == MediaCodec.INFO_TRY_AGAIN_LATER) {
  
                  // no output available yet
  
                  if (!endOfStream) {
  
                      break;      // out of while
  
                  } else {
  
                      if (VERBOSE) Log.d(TAG, "no output available, spinning to await EOS");
  
                  }
  
              } else if (encoderStatus == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
  
                  // not expected for an encoder
  
                  outputBuffers = codec.getOutputBuffers();
  
              } else if (encoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
  
                  // should happen before receiving buffers, and should only happen once
  
                  MediaFormat newFormat = codec.getOutputFormat();
  
                  Log.d(TAG, "encoder output format changed: " + newFormat);
  
              } else if (encoderStatus < 0) {
  
                  Log.w(TAG, "unexpected result from encoder.dequeueOutputBuffer: " + encoderStatus);
  
                  // let's ignore it
  
              } else {
  
                  ByteBuffer encodedData = outputBuffers[encoderStatus];
  
                  if (encodedData == null) {
  
                      throw new RuntimeException("encoderOutputBuffer " + encoderStatus + " was null");
  
                  }
  
  
  
                  if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
  
                      // The codec config data was pulled out and fed to the muxer when we got
  
                      // the INFO_OUTPUT_FORMAT_CHANGED status.  Ignore it.
  
                      if (VERBOSE) Log.d(TAG, "ignoring BUFFER_FLAG_CODEC_CONFIG");
  
                      bufferInfo.size = 0;
  
                  }
  
  
  
                  if (bufferInfo.size != 0) {
  
                      // adjust the ByteBuffer values to match BufferInfo (not needed?)
  
                      encodedData.position(bufferInfo.offset);
  
                      encodedData.limit(bufferInfo.offset + bufferInfo.size);
  
  
  
                      byte[] data = new byte[bufferInfo.size];
  
                      encodedData.get(data);
  
                      out.write(data);
  
                      if (VERBOSE) Log.d(TAG, "sent " + bufferInfo.size + " bytes to file");
  
                  }
  
  
  
                  codec.releaseOutputBuffer(encoderStatus, false);
  
  
  
                  if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
  
                      if (!endOfStream) {
  
                          Log.w(TAG, "reached end of stream unexpectedly");
  
                      } else {
  
                          if (VERBOSE) Log.d(TAG, "end of stream reached");
  
                      }
  
                      break;      // out of while
  
                  }
  
              }
  
          }
  
      }
  
      private void generateSurfaceFrame(int frameIndex) {
  
          frameIndex %= 8;
  
  
  
          int startX, startY;
  
          if (frameIndex < 4) {
  
              // (0,0) is bottom-left in GL
  
              startX = frameIndex * (1280 / 4);
  
              startY = 720 / 2;
  
          } else {
  
              startX = (7 - frameIndex) * (1280 / 4);
  
              startY = 0;
  
          }
  
  
  
          GLES20.glClearColor(TEST_R0 / 255.0f, TEST_G0 / 255.0f, TEST_B0 / 255.0f, 1.0f);
  
          GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);
  
  
  
          GLES20.glEnable(GLES20.GL_SCISSOR_TEST);
  
          GLES20.glScissor(startX, startY, 1280 / 4, 720 / 2);
  
          GLES20.glClearColor(TEST_R1 / 255.0f, TEST_G1 / 255.0f, TEST_B1 / 255.0f, 1.0f);
  
          GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);
  
          GLES20.glDisable(GLES20.GL_SCISSOR_TEST);
  
      }
  
      private static long computePresentationTimeNsec(int frameIndex, int frameRate) {
  
          final long ONE_BILLION = 1000000000;
  
          return frameIndex * ONE_BILLION / frameRate;
  
      }
  
  
  
      /**
  
       * Holds state associated with a Surface used for MediaCodec encoder input.
  
       * <p>
  
       * The constructor takes a Surface obtained from MediaCodec.createInputSurface(), and uses that
  
       * to create an EGL window surface.  Calls to eglSwapBuffers() cause a frame of data to be sent
  
       * to the video encoder.
  
       * </p><p>
  
       * This object owns the Surface -- releasing this will release the Surface too.
  
       */
  
      private static class CodecInputSurface {
  
          private static final int EGL_RECORDABLE_ANDROID = 0x3142;
  
  
  
          private EGLDisplay mEGLDisplay = EGL14.EGL_NO_DISPLAY;
  
          private EGLContext mEGLContext = EGL14.EGL_NO_CONTEXT;
  
          private EGLSurface mEGLSurface = EGL14.EGL_NO_SURFACE;
  
  
  
          private Surface mSurface;
  
  
  
          /**
  
           * Creates a CodecInputSurface from a Surface.
  
           */
  
          public CodecInputSurface(Surface surface) {
  
              if (surface == null) {
  
                  throw new NullPointerException();
  
              }
  
              mSurface = surface;
  
  
  
              eglSetup();
  
          }
  
  
  
          /**
  
           * Prepares EGL.  We want a GLES 2.0 context and a surface that supports recording.
  
           */
  
          private void eglSetup() {
  
              mEGLDisplay = EGL14.eglGetDisplay(EGL14.EGL_DEFAULT_DISPLAY);
  
              if (mEGLDisplay == EGL14.EGL_NO_DISPLAY) {
  
                  throw new RuntimeException("unable to get EGL14 display");
  
              }
  
              int[] version = new int[2];
  
              if (!EGL14.eglInitialize(mEGLDisplay, version, 0, version, 1)) {
  
                  throw new RuntimeException("unable to initialize EGL14");
  
              }
  
  
  
              // Configure EGL for recording and OpenGL ES 2.0.
  
              int[] attribList = {
  
                      EGL14.EGL_RED_SIZE, 8,
  
                      EGL14.EGL_GREEN_SIZE, 8,
  
                      EGL14.EGL_BLUE_SIZE, 8,
  
                      EGL14.EGL_ALPHA_SIZE, 8,
  
                      EGL14.EGL_RENDERABLE_TYPE, EGL14.EGL_OPENGL_ES2_BIT,
  
                      EGL_RECORDABLE_ANDROID, 1,
  
                      EGL14.EGL_NONE
  
              };
  
              EGLConfig[] configs = new EGLConfig[1];
  
              int[] numConfigs = new int[1];
  
              EGL14.eglChooseConfig(mEGLDisplay, attribList, 0, configs, 0, configs.length,
  
                      numConfigs, 0);
  
              checkEglError("eglCreateContext RGB888+recordable ES2");
  
  
  
              // Configure context for OpenGL ES 2.0.
  
              int[] attrib_list = {
  
                      EGL14.EGL_CONTEXT_CLIENT_VERSION, 2,
  
                      EGL14.EGL_NONE
  
              };
  
              mEGLContext = EGL14.eglCreateContext(mEGLDisplay, configs[0], EGL14.EGL_NO_CONTEXT,
  
                      attrib_list, 0);
  
              checkEglError("eglCreateContext");
  
  
  
              // Create a window surface, and attach it to the Surface we received.
  
              int[] surfaceAttribs = {
  
                      EGL14.EGL_NONE
  
              };
  
              mEGLSurface = EGL14.eglCreateWindowSurface(mEGLDisplay, configs[0], mSurface,
  
                      surfaceAttribs, 0);
  
              checkEglError("eglCreateWindowSurface");
  
          }
  
  
  
          /**
  
           * Discards all resources held by this class, notably the EGL context.  Also releases the
  
           * Surface that was passed to our constructor.
  
           */
  
          public void release() {
  
              if (mEGLDisplay != EGL14.EGL_NO_DISPLAY) {
  
                  EGL14.eglMakeCurrent(mEGLDisplay, EGL14.EGL_NO_SURFACE, EGL14.EGL_NO_SURFACE,
  
                          EGL14.EGL_NO_CONTEXT);
  
                  EGL14.eglDestroySurface(mEGLDisplay, mEGLSurface);
  
                  EGL14.eglDestroyContext(mEGLDisplay, mEGLContext);
  
                  EGL14.eglReleaseThread();
  
                  EGL14.eglTerminate(mEGLDisplay);
  
              }
  
  
  
              mSurface.release();
  
  
  
              mEGLDisplay = EGL14.EGL_NO_DISPLAY;
  
              mEGLContext = EGL14.EGL_NO_CONTEXT;
  
              mEGLSurface = EGL14.EGL_NO_SURFACE;
  
  
  
              mSurface = null;
  
          }
  
  
  
          /**
  
           * Makes our EGL context and surface current.
  
           */
  
          public void makeCurrent() {
  
              EGL14.eglMakeCurrent(mEGLDisplay, mEGLSurface, mEGLSurface, mEGLContext);
  
              checkEglError("eglMakeCurrent");
  
          }
  
  
  
          /**
  
           * Calls eglSwapBuffers.  Use this to "publish" the current frame.
  
           */
  
          public boolean swapBuffers() {
  
              boolean result = EGL14.eglSwapBuffers(mEGLDisplay, mEGLSurface);
  
              checkEglError("eglSwapBuffers");
  
              return result;
  
          }
  
  
  
          /**
  
           * Sends the presentation time stamp to EGL.  Time is expressed in nanoseconds.
  
           */
  
          public void setPresentationTime(long nsecs) {
  
              EGLExt.eglPresentationTimeANDROID(mEGLDisplay, mEGLSurface, nsecs);
  
              checkEglError("eglPresentationTimeANDROID");
  
          }
  
  
  
          /**
  
           * Checks for EGL errors.  Throws an exception if one is found.
  
           */
  
          private void checkEglError(String msg) {
  
              int error;
  
              if ((error = EGL14.eglGetError()) != EGL14.EGL_SUCCESS) {
  
                  throw new RuntimeException(msg + ": EGL error: 0x" + Integer.toHexString(error));
  
              }
  
          }
  
      }
  
  }
  
  </p>

  However, the file produced from this code does not play with VLC or ffplay. Can anyone tell me what I’m doing wrong ? I believe it is due to an incorrect format (or total lack) of headers required for the playing of raw h264, as I have had success playing .264 files downloaded from the internet with ffplay. Also, I’m not sure exactly how I’m going to stream this video to a computer, so if somebody could give me some suggestions as to how I might do that, I would be very grateful ! Thanks !