Recherche avancée

Sur d’autres sites (13095)

• H.264 muxed to MP4 using libavformat not playing back

  14 mai 2015, par Brad Mitchell

  I am trying to mux H.264 data into a MP4 file. There appear to be no errors in saving this H.264 Annex B data out to an MP4 file, but the file fails to playback.

  I’ve done a binary comparison on the files and the issue seems to be somewhere in what is being written to the footer (trailer) of the MP4 file.

  I suspect it has to be something with the way the stream is being created or something.

  Init :

  AVOutputFormat* fmt = av_guess_format( 0, "out.mp4", 0 );
  
  oc = avformat_alloc_context();
  
  oc->oformat = fmt;
  
  strcpy(oc->filename, filename);

  Part of this prototype app I have is creating a png file for each IFrame. So when the first IFrame is encountered, I create the video stream and write the av header etc :

  void addVideoStream(AVCodecContext* decoder)
  
  {
  
      videoStream = av_new_stream(oc, 0);
  
      if (!videoStream)
  
      {
  
           cout << "ERROR creating video stream" << endl;
  
           return;        
  
      }
  
      vi = videoStream->index;    
  
      videoContext = videoStream->codec;      
  
      videoContext->codec_type = AVMEDIA_TYPE_VIDEO;
  
      videoContext->codec_id = decoder->codec_id;
  
      videoContext->bit_rate = 512000;
  
      videoContext->width = decoder->width;
  
      videoContext->height = decoder->height;
  
      videoContext->time_base.den = 25;
  
      videoContext->time_base.num = 1;    
  
      videoContext->gop_size = decoder->gop_size;
  
      videoContext->pix_fmt = decoder->pix_fmt;       
  
  
  
      if (oc->oformat->flags & AVFMT_GLOBALHEADER)
  
          videoContext->flags |= CODEC_FLAG_GLOBAL_HEADER;
  
  
  
      av_dump_format(oc, 0, filename, 1);
  
  
  
      if (!(oc->oformat->flags & AVFMT_NOFILE))
  
      {
  
          if (avio_open(&oc->pb, filename, AVIO_FLAG_WRITE) < 0) {
  
          cout << "Error opening file" << endl;
  
      }
  
      avformat_write_header(oc, NULL);
  
  }

  I write packets out :

  unsigned char* data = block->getData();
  
  unsigned char videoFrameType = data[4];
  
  int dataLen = block->getDataLen();
  
  
  
  // store pps
  
  if (videoFrameType == 0x68)
  
  {
  
      if (ppsFrame != NULL)
  
      {
  
          delete ppsFrame; ppsFrameLength = 0; ppsFrame = NULL;
  
      }
  
      ppsFrameLength = block->getDataLen();
  
      ppsFrame = new unsigned char[ppsFrameLength];
  
      memcpy(ppsFrame, block->getData(), ppsFrameLength);
  
  }
  
  else if (videoFrameType == 0x67)
  
  {
  
      // sps
  
      if (spsFrame != NULL)
  
      {
  
          delete spsFrame; spsFrameLength = 0; spsFrame = NULL;
  
  }
  
      spsFrameLength = block->getDataLen();
  
      spsFrame = new unsigned char[spsFrameLength];
  
      memcpy(spsFrame, block->getData(), spsFrameLength);                 
  
  }                                           
  
  
  
  if (videoFrameType == 0x65 || videoFrameType == 0x41)
  
  {
  
      videoFrameNumber++;
  
  }
  
  if (videoFrameType == 0x65)
  
  {
  
      decodeIFrame(videoFrameNumber, spsFrame, spsFrameLength, ppsFrame, ppsFrameLength, data, dataLen);
  
  }
  
  
  
  if (videoStream != NULL)
  
  {
  
      AVPacket pkt = { 0 };
  
      av_init_packet(&pkt);
  
      pkt.stream_index = vi;
  
      pkt.flags = 0;                      
  
      pkt.pts = pkt.dts = 0;                                  
  
  
  
      if (videoFrameType == 0x65)
  
      {
  
          // combine the SPS PPS & I frames together
  
          pkt.flags |= AV_PKT_FLAG_KEY;                                                   
  
          unsigned char* videoFrame = new unsigned char[spsFrameLength+ppsFrameLength+dataLen];
  
          memcpy(videoFrame, spsFrame, spsFrameLength);
  
          memcpy(&videoFrame[spsFrameLength], ppsFrame, ppsFrameLength);
  
          memcpy(&videoFrame[spsFrameLength+ppsFrameLength], data, dataLen);
  
  
  
          // overwrite the start code (00 00 00 01 with a 32-bit length)
  
          setLength(videoFrame, spsFrameLength-4);
  
          setLength(&videoFrame[spsFrameLength], ppsFrameLength-4);
  
          setLength(&videoFrame[spsFrameLength+ppsFrameLength], dataLen-4);
  
          pkt.size = dataLen + spsFrameLength + ppsFrameLength;
  
          pkt.data = videoFrame;
  
          av_interleaved_write_frame(oc, &pkt);
  
          delete videoFrame; videoFrame = NULL;
  
      }
  
      else if (videoFrameType != 0x67 && videoFrameType != 0x68)
  
      {   
  
          // Send other frames except pps & sps which are caught and stored                   
  
          pkt.size = dataLen;
  
          pkt.data = data;
  
          setLength(data, dataLen-4);                     
  
          av_interleaved_write_frame(oc, &pkt);
  
      }

  Finally to close the file off :

  av_write_trailer(oc);
  
  int i = 0;
  
  for (i = 0; i < oc->nb_streams; i++)
  
  {
  
      av_freep(&oc->streams[i]->codec);
  
      av_freep(&oc->streams[i]);      
  
  }
  
  
  
  if (!(oc->oformat->flags & AVFMT_NOFILE))
  
  {
  
      avio_close(oc->pb);
  
  }
  
  av_free(oc);

  If I take the H.264 data alone and convert it :

  ffmpeg -i recording.h264 -vcodec copy recording.mp4

  All but the "footer" of the files are the same.

  Output from my program :
  readrec recording.tcp out.mp4
  ** START * 01-03-2013 14:26:01 180000
  Output #0, mp4, to ’out.mp4’ :
  Stream #0:0 : Video : h264, yuv420p, 352x288, q=2-31, 512 kb/s, 90k tbn, 25 tbc
  * END ** 01-03-2013 14:27:01 102000
  Wrote 1499 video frames.

  If I try to convert using ffmpeg the MP4 file created using CODE :

  ffmpeg -i out.mp4 -vcodec copy out2.mp4
  
  ffmpeg version 0.11.1 Copyright (c) 2000-2012 the FFmpeg developers
  
        built on Mar  7 2013 12:49:22 with suncc 0x5110
  
        configuration: --extra-cflags=-KPIC -g --disable-mmx
  
        --disable-protocol=udp --disable-encoder=nellymoser --cc=cc --cxx=CC
  
  libavutil      51. 54.100 / 51. 54.100
  
  libavcodec     54. 23.100 / 54. 23.100
  
  libavformat    54.  6.100 / 54.  6.100
  
  libavdevice    54.  0.100 / 54.  0.100
  
  libavfilter     2. 77.100 /  2. 77.100
  
  libswscale      2.  1.100 /  2.  1.100
  
  libswresample   0. 15.100 /  0. 15.100
  
  h264 @ 12eaac0] no frame!
  
      Last message repeated 1 times
  
  [h264 @ 12eaac0] slice type too large (0) at 0 0
  
  [h264 @ 12eaac0] decode_slice_header error
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 23 times
  
  [h264 @ 12eaac0] slice type too large (0) at 0 0
  
  [h264 @ 12eaac0] decode_slice_header error
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 74 times
  
  [h264 @ 12eaac0] slice type too large (0) at 0 0
  
  [h264 @ 12eaac0] decode_slice_header error
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 64 times
  
  [h264 @ 12eaac0] slice type too large (0) at 0 0
  
  [h264 @ 12eaac0] decode_slice_header error
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 34 times
  
  [h264 @ 12eaac0] slice type too large (0) at 0 0
  
  [h264 @ 12eaac0] decode_slice_header error
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 49 times
  
  [h264 @ 12eaac0] slice type too large (0) at 0 0
  
  [h264 @ 12eaac0] decode_slice_header error
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 24 times
  
  [h264 @ 12eaac0] Partitioned H.264 support is incomplete
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 23 times
  
  [h264 @ 12eaac0] sps_id out of range
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 148 times
  
  [h264 @ 12eaac0] sps_id (32) out of range
  
      Last message repeated 1 times
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 33 times
  
  [h264 @ 12eaac0] slice type too large (0) at 0 0
  
  [h264 @ 12eaac0] decode_slice_header error
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 128 times
  
  [h264 @ 12eaac0] sps_id (32) out of range
  
      Last message repeated 1 times
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 3 times
  
  [h264 @ 12eaac0] slice type too large (0) at 0 0
  
  [h264 @ 12eaac0] decode_slice_header error
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 3 times
  
  [h264 @ 12eaac0] slice type too large (0) at 0 0
  
  [h264 @ 12eaac0] decode_slice_header error
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 309 times
  
  [h264 @ 12eaac0] sps_id (32) out of range
  
      Last message repeated 1 times
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 192 times
  
  [h264 @ 12eaac0] Partitioned H.264 support is incomplete
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 73 times
  
  [h264 @ 12eaac0] sps_id (32) out of range
  
      Last message repeated 1 times
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 99 times
  
  [h264 @ 12eaac0] sps_id (32) out of range
  
      Last message repeated 1 times
  
  [h264 @ 12eaac0] no frame!
  
      Last message repeated 197 times
  
  [mov,mp4,m4a,3gp,3g2,mj2 @ 12e3100] decoding for stream 0 failed
  
  [mov,mp4,m4a,3gp,3g2,mj2 @ 12e3100] Could not find codec parameters
  
  (Video: h264 (avc1 / 0x31637661), 393539 kb/s)
  
  out.mp4: could not find codec parameters

  I really do not know where the issue is, except it has to be something to do with the way the streams are being set up. I’ve looked at bits of code from where other people are doing a similar thing, and tried to use this advice in setting up the streams, but to no avail !

  ---

  The final code which gave me a H.264/AAC muxed (synced) file is as follows. First a bit of background information. The data is coming from an IP camera. The data is presented via a 3rd party API as video/audio packets. The video packets are presented as the RTP payload data (no header) and consist of NALU’s that are reconstructed and converted to H.264 video in Annex B format. AAC audio is presented as raw AAC and is converted to adts format to enable playback. These packets have been put into a bitstream format that allows the transmission of the timestamp (64 bit milliseconds since Jan 1 1970) along with a few other things.

  This is more or less a prototype and is not clean in any respects. It probably leaks bad. I do however, hope this helps anyone else out trying to achieve something similar to what I am.

  Globals :

  AVFormatContext* oc = NULL;
  
  AVCodecContext* videoContext = NULL;
  
  AVStream* videoStream = NULL;
  
  AVCodecContext* audioContext = NULL;
  
  AVStream* audioStream = NULL;
  
  AVCodec* videoCodec = NULL;
  
  AVCodec* audioCodec = NULL;
  
  int vi = 0;  // Video stream
  
  int ai = 1;  // Audio stream
  
  
  
  uint64_t firstVideoTimeStamp = 0;
  
  uint64_t firstAudioTimeStamp = 0;
  
  int audioStartOffset = 0;
  
  
  
  char* filename = NULL;
  
  
  
  Boolean first = TRUE;
  
  
  
  int videoFrameNumber = 0;
  
  int audioFrameNumber = 0;

  Main :

  int main(int argc, char* argv[])
  
  {
  
      if (argc != 3)
  
      {   
  
          cout &lt;&lt; argv[0] &lt;&lt; " <stream playback="playback" file="file"> <output mp4="mp4" file="file">" &lt;&lt; endl;
  
          return 0;
  
      }
  
      char* input_stream_file = argv[1];
  
      filename = argv[2];
  
  
  
      av_register_all();    
  
  
  
      fstream inFile;
  
      inFile.open(input_stream_file, ios::in);
  
  
  
      // Used to store the latest pps &amp; sps frames
  
      unsigned char* ppsFrame = NULL;
  
      int ppsFrameLength = 0;
  
      unsigned char* spsFrame = NULL;
  
      int spsFrameLength = 0;
  
  
  
      // Setup MP4 output file
  
      AVOutputFormat* fmt = av_guess_format( 0, filename, 0 );
  
      oc = avformat_alloc_context();
  
      oc->oformat = fmt;
  
      strcpy(oc->filename, filename);
  
  
  
      // Setup the bitstream filter for AAC in adts format.  Could probably also achieve
  
      // this by stripping the first 7 bytes!
  
      AVBitStreamFilterContext* bsfc = av_bitstream_filter_init("aac_adtstoasc");
  
      if (!bsfc)
  
      {       
  
          cout &lt;&lt; "Error creating adtstoasc filter" &lt;&lt; endl;
  
          return -1;
  
      }
  
  
  
      while (inFile.good())
  
      {
  
          TcpAVDataBlock* block = new TcpAVDataBlock();
  
          block->readStruct(inFile);
  
          DateTime dt = block->getTimestampAsDateTime();
  
          switch (block->getPacketType())
  
          {
  
              case TCP_PACKET_H264:
  
              {       
  
                  if (firstVideoTimeStamp == 0)
  
                      firstVideoTimeStamp = block->getTimeStamp();
  
                  unsigned char* data = block->getData();
  
                  unsigned char videoFrameType = data[4];
  
                  int dataLen = block->getDataLen();
  
  
  
                  // pps
  
                  if (videoFrameType == 0x68)
  
                  {
  
                      if (ppsFrame != NULL)
  
                      {
  
                          delete ppsFrame; ppsFrameLength = 0;
  
                          ppsFrame = NULL;
  
                      }
  
                      ppsFrameLength = block->getDataLen();
  
                      ppsFrame = new unsigned char[ppsFrameLength];
  
                      memcpy(ppsFrame, block->getData(), ppsFrameLength);
  
                  }
  
                  else if (videoFrameType == 0x67)
  
                  {
  
                      // sps
  
                      if (spsFrame != NULL)
  
                      {
  
                          delete spsFrame; spsFrameLength = 0;
  
                          spsFrame = NULL;
  
                      }
  
                      spsFrameLength = block->getDataLen();
  
                      spsFrame = new unsigned char[spsFrameLength];
  
                      memcpy(spsFrame, block->getData(), spsFrameLength);                   
  
                  }                                           
  
  
  
                  if (videoFrameType == 0x65 || videoFrameType == 0x41)
  
                  {
  
                      videoFrameNumber++;
  
                  }
  
                  // Extract a thumbnail for each I-Frame
  
                  if (videoFrameType == 0x65)
  
                  {
  
                      decodeIFrame(h264, spsFrame, spsFrameLength, ppsFrame, ppsFrameLength, data, dataLen);
  
                  }
  
                  if (videoStream != NULL)
  
                  {
  
                      AVPacket pkt = { 0 };
  
                      av_init_packet(&amp;pkt);
  
                      pkt.stream_index = vi;
  
                      pkt.flags = 0;           
  
                      pkt.pts = videoFrameNumber;
  
                      pkt.dts = videoFrameNumber;           
  
                      if (videoFrameType == 0x65)
  
                      {
  
                          pkt.flags = 1;                           
  
  
  
                          unsigned char* videoFrame = new unsigned char[spsFrameLength+ppsFrameLength+dataLen];
  
                          memcpy(videoFrame, spsFrame, spsFrameLength);
  
                          memcpy(&amp;videoFrame[spsFrameLength], ppsFrame, ppsFrameLength);
  
  
  
                          memcpy(&amp;videoFrame[spsFrameLength+ppsFrameLength], data, dataLen);
  
                          pkt.data = videoFrame;
  
                          av_interleaved_write_frame(oc, &amp;pkt);
  
                          delete videoFrame; videoFrame = NULL;
  
                      }
  
                      else if (videoFrameType != 0x67 &amp;&amp; videoFrameType != 0x68)
  
                      {                       
  
                          pkt.size = dataLen;
  
                          pkt.data = data;
  
                          av_interleaved_write_frame(oc, &amp;pkt);
  
                      }                       
  
                  }
  
                  break;
  
              }
  
  
  
          case TCP_PACKET_AAC:
  
  
  
              if (firstAudioTimeStamp == 0)
  
              {
  
                  firstAudioTimeStamp = block->getTimeStamp();
  
                  uint64_t millseconds_difference = firstAudioTimeStamp - firstVideoTimeStamp;
  
                  audioStartOffset = millseconds_difference * 16000 / 1000;
  
                  cout &lt;&lt; "audio offset: " &lt;&lt; audioStartOffset &lt;&lt; endl;
  
              }
  
  
  
              if (audioStream != NULL)
  
              {
  
                  AVPacket pkt = { 0 };
  
                  av_init_packet(&amp;pkt);
  
                  pkt.stream_index = ai;
  
                  pkt.flags = 1;           
  
                  pkt.pts = audioFrameNumber*1024;
  
                  pkt.dts = audioFrameNumber*1024;
  
                  pkt.data = block->getData();
  
                  pkt.size = block->getDataLen();
  
                  pkt.duration = 1024;
  
  
  
                  AVPacket newpacket = pkt;                       
  
                  int rc = av_bitstream_filter_filter(bsfc, audioContext,
  
                      NULL,
  
                      &amp;newpacket.data, &amp;newpacket.size,
  
                      pkt.data, pkt.size,
  
                      pkt.flags &amp; AV_PKT_FLAG_KEY);
  
  
  
                  if (rc >= 0)
  
                  {
  
                      //cout &lt;&lt; "Write audio frame" &lt;&lt; endl;
  
                      newpacket.pts = audioFrameNumber*1024;
  
                      newpacket.dts = audioFrameNumber*1024;
  
                      audioFrameNumber++;
  
                      newpacket.duration = 1024;                   
  
  
  
                      av_interleaved_write_frame(oc, &amp;newpacket);
  
                      av_free_packet(&amp;newpacket);
  
                  }   
  
                  else
  
                  {
  
                      cout &lt;&lt; "Error filtering aac packet" &lt;&lt; endl;
  
  
  
                  }
  
              }
  
              break;
  
  
  
          case TCP_PACKET_START:
  
              break;
  
  
  
          case TCP_PACKET_END:
  
              break;
  
          }
  
          delete block;
  
      }
  
      inFile.close();
  
  
  
      av_write_trailer(oc);
  
      int i = 0;
  
      for (i = 0; i &lt; oc->nb_streams; i++)
  
      {
  
          av_freep(&amp;oc->streams[i]->codec);
  
          av_freep(&amp;oc->streams[i]);       
  
      }
  
  
  
      if (!(oc->oformat->flags &amp; AVFMT_NOFILE))
  
      {
  
          avio_close(oc->pb);
  
      }
  
  
  
      av_free(oc);
  
  
  
      delete spsFrame; spsFrame = NULL;
  
      delete ppsFrame; ppsFrame = NULL;
  
  
  
      cout &lt;&lt; "Wrote " &lt;&lt; videoFrameNumber &lt;&lt; " video frames." &lt;&lt; endl;
  
  
  
      return 0;
  
  }
  
  </output></stream>

  The stream stream/codecs are added and the header is created in a function called addVideoAndAudioStream(). This function is called from decodeIFrame() so there are a few assumptions (which aren’t necessarily good)
  1. A video packet comes first
  2. AAC is present

  The decodeIFrame was kind of a separate prototype by where I was creating a thumbnail for each I Frame. The code to generate thumbnails was from : https://gnunet.org/svn/Extractor/src/plugins/thumbnailffmpeg_extractor.c

  The decodeIFrame function passes an AVCodecContext into addVideoAudioStream :

  void addVideoAndAudioStream(AVCodecContext* decoder = NULL)
  
  {
  
      videoStream = av_new_stream(oc, 0);
  
      if (!videoStream)
  
      {
  
          cout << "ERROR creating video stream" << endl;
  
          return;       
  
      }
  
      vi = videoStream->index;   
  
      videoContext = videoStream->codec;       
  
      videoContext->codec_type = AVMEDIA_TYPE_VIDEO;
  
      videoContext->codec_id = decoder->codec_id;
  
      videoContext->bit_rate = 512000;
  
      videoContext->width = decoder->width;
  
      videoContext->height = decoder->height;
  
      videoContext->time_base.den = 25;
  
      videoContext->time_base.num = 1;
  
      videoContext->gop_size = decoder->gop_size;
  
      videoContext->pix_fmt = decoder->pix_fmt;       
  
  
  
      audioStream = av_new_stream(oc, 1);
  
      if (!audioStream)
  
      {
  
          cout << "ERROR creating audio stream" << endl;
  
          return;
  
      }
  
      ai = audioStream->index;
  
      audioContext = audioStream->codec;
  
      audioContext->codec_type = AVMEDIA_TYPE_AUDIO;
  
      audioContext->codec_id = CODEC_ID_AAC;
  
      audioContext->bit_rate = 64000;
  
      audioContext->sample_rate = 16000;
  
      audioContext->channels = 1;
  
  
  
      if (oc->oformat->flags & AVFMT_GLOBALHEADER)
  
      {
  
          videoContext->flags |= CODEC_FLAG_GLOBAL_HEADER;
  
          audioContext->flags |= CODEC_FLAG_GLOBAL_HEADER;
  
      }
  
  
  
      av_dump_format(oc, 0, filename, 1);
  
  
  
      if (!(oc->oformat->flags & AVFMT_NOFILE))
  
      {
  
          if (avio_open(&oc->pb, filename, AVIO_FLAG_WRITE) < 0) {
  
              cout << "Error opening file" << endl;
  
          }
  
      }
  
  
  
      avformat_write_header(oc, NULL);
  
  }

  As far as I can tell, a number of assumptions didn’t seem to matter, for example :
  1. Bit Rate. The actual video bit rate was 262k whereas I specified 512kbit
  2. AAC channels. I specified mono, although the actual output was Stereo from memory

  You would still need to know what the frame rate (time base) is for the video & audio.

  Contrary to a lot of other examples, when setting pts & dts on the video packets, it was not playable. I needed to know the time base (25fps) and then set the pts & dts according to that time base, i.e. first frame = 0 (PPS, SPS, I), second frame = 1 (intermediate frame, whatever its called ;)).

  AAC I also had to make the assumption that it was 16000 hz. 1024 samples per AAC packet (You can also have AAC @ 960 samples I think) to determine the audio "offset". I added this to the pts & dts. So the pts/dts are the sample number that it is to played back at. You also need to make sure that the duration of 1024 is set in the packet before writing also.

  —

  I have found additionally today that Annex B isn’t really compatible with any other player so AVCC format should really be used.

  These URLS helped :
  Problem to Decode H264 video over RTP with ffmpeg (libavcodec)
  http://aviadr1.blogspot.com.au/2010/05/h264-extradata-partially-explained-for.html

  When constructing the video stream, I filled out the extradata & extradata_size :

  // Extradata contains PPS & SPS for AVCC format
  
  int extradata_len = 8 + spsFrameLen-4 + 1 + 2 + ppsFrameLen-4;
  
  videoContext->extradata = (uint8_t*)av_mallocz(extradata_len);
  
  videoContext->extradata_size = extradata_len;
  
  videoContext->extradata[0] = 0x01;
  
  videoContext->extradata[1] = spsFrame[4+1];
  
  videoContext->extradata[2] = spsFrame[4+2];
  
  videoContext->extradata[3] = spsFrame[4+3];
  
  videoContext->extradata[4] = 0xFC | 3;
  
  videoContext->extradata[5] = 0xE0 | 1;
  
  int tmp = spsFrameLen - 4;
  
  videoContext->extradata[6] = (tmp >> 8) & 0x00ff;
  
  videoContext->extradata[7] = tmp & 0x00ff;
  
  int i = 0;
  
  for (i=0;iextradata[8+i] = spsFrame[4+i];
  
  videoContext->extradata[8+tmp] = 0x01;
  
  int tmp2 = ppsFrameLen-4;   
  
  videoContext->extradata[8+tmp+1] = (tmp2 >> 8) & 0x00ff;
  
  videoContext->extradata[8+tmp+2] = tmp2 & 0x00ff;
  
  for (i=0;iextradata[8+tmp+3+i] = ppsFrame[4+i];

  When writing out the frames, don’t prepend the SPS & PPS frames, just write out the I Frame & P frames. In addition, replace the Annex B start code contained in the first 4 bytes (0x00 0x00 0x00 0x01) with the size of the I/P frame.

• WebVTT Discussions at FOMS

  18 décembre 2013, par silvia

  At the recent FOMS (Foundations of Open Media Software and Standards) Developer Workshop, we had a massive focus on WebVTT and the state of its feature set. You will find links to summaries of the individual discussions in the FOMS Schedule page. Here are some of the key results I went away with.

  1. WebVTT Regions

  The key driving force for improvements to WebVTT continues to be the accurate representation of CEA608/708 captioning. As part of that drive, we’ve introduced regions (the CEA708 “window” concept) to WebVTT. WebVTT regions satisfy multiple requirements of CEA608/708 captions :

  1. support for rollup captions
  2. support for background color and border color on a group of cues independent of the background color of the individual cue
  3. possibility to move a group of cues from one location on screen to a different
  4. support to specify an anchor point and a growth direction for cues when their text size changes
  5. support for specifying a fixed number of lines to be rendered
  6. possibility to specify which region is rendered in front of which other one when regions overlap

  While WebVTT regions enable us to satisfy all of the above points, the specification isn’t actually complete yet and some of the above needs aren’t satisfied yet.

  We have an open bug to move a region elsewhere. A first discussion at FOMS seemed to to indicate that we’ll have to add syntax for updating a region at a particular time and thus give region definitions a way to be valid only for a certain time frame. I can imagine that the region definitions that we have in the header of the WebVTT file now would have an implicitly defined time frame from the start to the end of the file, but can be overruled by a re-definition anywhere within the WebVTT file. That redefinition needs to provide a start and end time.

  We registered a bug to add specifying the width and height of regions (and possibly of cues) by em (i.e. by multiples of the largest character in a font). This should allow us to have the region grow/shrink around the region anchor point with a change of font size by script or a user. em specifications should also be applied to cues – that matches the column count of CEA708/608 better.

  When regions overlap, the original region extension spec already suggested a “layer” cue setting. It will be easy to add it.

  Another change that we will ultimately need is the “scroll” setting : we will need to introduce support for scrolling text down or from left-to-right or right-to-left, e.g. vertical scrolling text seems to be used in some Chinese caption use cases.

  2. Unify Rendering Approach

  The introduction of regions created a second code path in the rendering spec with some duplication. At FOMS we discussed if it was possible to unify that. The suggestion is to render all cues into a region. Those that are not part of a region would be rendered into an anonymous region that covers the complete viewport. There may be some consequences to this, e.g. cue settings should be usable across all cues, no matter whether or not part of a region, and avoiding cue overlap may need to be done within regions.

  Here’s a rough outline of the path of the new rendering algorithm :

  (1) Render the regions :

  Specified Region	Anonymous Region
  Render values as given :	Render following values :
  width lines regionanchor viewportanchor scroll	100% videoheight/lineheight 0,0 0,0 none

  (2) Render the cues :

  • Create a cue box and put it in its region (anonymous if none given).
  • Calculate position & size of cue box from cue settings (position, line, size).
  • Calculate position of cue text inside cue box from remaining cue settings (vertical, align).

  3. Vertical Features

  WebVTT includes vertical rendering, both right-to-left and left-to-right. However, regions are not defined for vertical. Eventually, we’re going to have to look at the vertical features of WebVTT with more details and figure out whether the spec is working for them and what real-world requirements we have missed. We hope we can get some help from users in countries where vertically rendered captions/subtitles are the norm.

  4. Best Practices

  Some of he WebVTT users at FOMS suggested it would be advantageous to start a list of “best practices” for how to author captions with WebVTT. Example recommendations are :

  • Use line numbers only to position cues from top or bottom of viewport. Don’t use otherwise.
  • Note that when the user increases the fontsize in rollup captions and thus introduces new line breaks, your cues will roll by faster because the number of lines of a rollup is fixed.
  • Make sure to use &lrm ; and &rlm ; UTF-8 markers to control the directionality of your text.

  It would be nice if somebody started such a document.

  5. Non-caption use cases

  Instead of continuing to look back and improve our support of captions/subtitles in WebVTT, one session at FOMS also went ahead and looked forward to other use cases. The following requirements came out of this :

  5.1 Preview Thumbnails

  A common use case for timed data is the use of preview thumbnails on the navigation bar of videos. A native implementation of preview thumbnails would allow crawlers and search engines to have a standardised way of extracting timed images for media files, so introduction of a new @kind value “thumbnails” was suggested.

  The content of a “thumbnails” cue could be any of :

  • an image URL
  • a sprite URL to a single image
  • a spatial & temporal media fragment URL to a media resource
  • base64 encoded image (data URI)
  • an iframe offset to the media resource

  The suggestion is to allow anything that would work in a img @src attribute as value in a cue of @kind=”thumbnails”. Responsive images might also be useful for a track of @kind=”thumbnails”. It may even be possible to define an inband thumbnail track based on the track of @kind=”thumbnails”. Such cues should also work in the JavaScript track API.

  5.2 Chapter markers

  There is interest to put richer content than just a chapter title into chapter cues. Often, chapters consist of a title, text and and image. The text is not so important, but the image is used almost everywhere that chapters are used. There may be a need to extend chapter cue content with images, similar to what a @kind=”thumbnails” track offers.

  The conclusion that we arrived at was that we need to make @kind=”thumbnails” work first and then look at using the learnings from that to extend @kind=”chapters”.

  5.3 Inband tracks for live video

  A difficult topic was opened with the question of how to transport text tracks in live video. In live captioning, end times are never created for cues, but are implied by the start time of the next cue. This is a use case that hasn’t been addressed in HTML5/WebVTT yet. An old proposal to allow a special end time value of “NEXT” was discussed and recommended for adoption. Also, there was support for the spec change that stops blocking loading VTT until all cues have been loaded.

  5.4 Cross-domain VTT loading

  A brief discussion centered around the fact that the spec disallows cross-domain loading of WebVTT files, but that no browser implements this. This needs to be discussion at the HTML WG level.

  6. Regions in live captioning

  The final topic that we discussed was how we could provide support for regions in live captioning.

  • The currently active region definitions will need to be come part of every header of every VTT file segment that HLS uses, so it’s available in case the cues in the segment file reference it.
  • “NEXT” in end time markers would make authoring of live captioned VTT files easier.
  • If the application wants to use 1 word at a time and doesn’t want to delay sending the word until the full cue is authored (e.g. in a Hangout type environment), we will need to introduce the concept of “cue continuation markers”, so we know that a cue could be extended with the next VTT file fragment.

  This is an extensive and impressive amount of discussion around WebVTT and a lot of new work to be performed in the future. I’m very grateful for all the people who have contributed to these discussions at FOMS and will hopefully continue to help get the specifications right.

• WebVTT Discussions at FOMS

  1er janvier 2014, par silvia

  At the recent FOMS (Foundations of Open Media Software and Standards) Developer Workshop, we had a massive focus on WebVTT and the state of its feature set. You will find links to summaries of the individual discussions in the FOMS Schedule page. Here are some of the key results I went away with.

  1. WebVTT Regions

  The key driving force for improvements to WebVTT continues to be the accurate representation of CEA608/708 captioning. As part of that drive, we’ve introduced regions (the CEA708 “window” concept) to WebVTT. WebVTT regions satisfy multiple requirements of CEA608/708 captions :

  1. support for rollup captions
  2. support for background color and border color on a group of cues independent of the background color of the individual cue
  3. possibility to move a group of cues from one location on screen to a different
  4. support to specify an anchor point and a growth direction for cues when their text size changes
  5. support for specifying a fixed number of lines to be rendered
  6. possibility to specify which region is rendered in front of which other one when regions overlap

  While WebVTT regions enable us to satisfy all of the above points, the specification isn’t actually complete yet and some of the above needs aren’t satisfied yet.

  We have an open bug to move a region elsewhere. A first discussion at FOMS seemed to to indicate that we’ll have to add syntax for updating a region at a particular time and thus give region definitions a way to be valid only for a certain time frame. I can imagine that the region definitions that we have in the header of the WebVTT file now would have an implicitly defined time frame from the start to the end of the file, but can be overruled by a re-definition anywhere within the WebVTT file. That redefinition needs to provide a start and end time.

  We registered a bug to add specifying the width and height of regions (and possibly of cues) by em (i.e. by multiples of the largest character in a font). This should allow us to have the region grow/shrink around the region anchor point with a change of font size by script or a user. em specifications should also be applied to cues – that matches the column count of CEA708/608 better.

  When regions overlap, the original region extension spec already suggested a “layer” cue setting. It will be easy to add it.

  Another change that we will ultimately need is the “scroll” setting : we will need to introduce support for scrolling text down or from left-to-right or right-to-left, e.g. vertical scrolling text seems to be used in some Chinese caption use cases.

  2. Unify Rendering Approach

  The introduction of regions created a second code path in the rendering spec with some duplication. At FOMS we discussed if it was possible to unify that. The suggestion is to render all cues into a region. Those that are not part of a region would be rendered into an anonymous region that covers the complete viewport. There may be some consequences to this, e.g. cue settings should be usable across all cues, no matter whether or not part of a region, and avoiding cue overlap may need to be done within regions.

  Here’s a rough outline of the path of the new rendering algorithm :

  (1) Render the regions :

  Specified Region	Anonymous Region
  Render values as given :	Render following values :
  width lines regionanchor viewportanchor scroll	100% videoheight/lineheight 0,0 0,0 none

  (2) Render the cues :

  • Create a cue box and put it in its region (anonymous if none given).
  • Calculate position & size of cue box from cue settings (position, line, size).
  • Calculate position of cue text inside cue box from remaining cue settings (vertical, align).

  3. Vertical Features

  WebVTT includes vertical rendering, both right-to-left and left-to-right. However, regions are not defined for vertical. Eventually, we’re going to have to look at the vertical features of WebVTT with more details and figure out whether the spec is working for them and what real-world requirements we have missed. We hope we can get some help from users in countries where vertically rendered captions/subtitles are the norm.

  4. Best Practices

  Some of he WebVTT users at FOMS suggested it would be advantageous to start a list of “best practices” for how to author captions with WebVTT. Example recommendations are :

  • Use line numbers only to position cues from top or bottom of viewport. Don’t use otherwise.
  • Note that when the user increases the fontsize in rollup captions and thus introduces new line breaks, your cues will roll by faster because the number of lines of a rollup is fixed.
  • Make sure to use &lrm ; and &rlm ; UTF-8 markers to control the directionality of your text.

  It would be nice if somebody started such a document.

  5. Non-caption use cases

  Instead of continuing to look back and improve our support of captions/subtitles in WebVTT, one session at FOMS also went ahead and looked forward to other use cases. The following requirements came out of this :

  5.1 Preview Thumbnails

  A common use case for timed data is the use of preview thumbnails on the navigation bar of videos. A native implementation of preview thumbnails would allow crawlers and search engines to have a standardised way of extracting timed images for media files, so introduction of a new @kind value “thumbnails” was suggested.

  The content of a “thumbnails” cue could be any of :

  • an image URL
  • a sprite URL to a single image
  • a spatial & temporal media fragment URL to a media resource
  • base64 encoded image (data URI)
  • an iframe offset to the media resource

  The suggestion is to allow anything that would work in a img @src attribute as value in a cue of @kind=”thumbnails”. Responsive images might also be useful for a track of @kind=”thumbnails”. It may even be possible to define an inband thumbnail track based on the track of @kind=”thumbnails”. Such cues should also work in the JavaScript track API.

  5.2 Chapter markers

  There is interest to put richer content than just a chapter title into chapter cues. Often, chapters consist of a title, text and and image. The text is not so important, but the image is used almost everywhere that chapters are used. There may be a need to extend chapter cue content with images, similar to what a @kind=”thumbnails” track offers.

  The conclusion that we arrived at was that we need to make @kind=”thumbnails” work first and then look at using the learnings from that to extend @kind=”chapters”.

  5.3 Inband tracks for live video

  A difficult topic was opened with the question of how to transport text tracks in live video. In live captioning, end times are never created for cues, but are implied by the start time of the next cue. This is a use case that hasn’t been addressed in HTML5/WebVTT yet. An old proposal to allow a special end time value of “NEXT” was discussed and recommended for adoption. Also, there was support for the spec change that stops blocking loading VTT until all cues have been loaded.

  5.4 Cross-domain VTT loading

  A brief discussion centered around the fact that the spec disallows cross-domain loading of WebVTT files, but that no browser implements this. This needs to be discussion at the HTML WG level.

  6. Regions in live captioning

  The final topic that we discussed was how we could provide support for regions in live captioning.

  • The currently active region definitions will need to be come part of every header of every VTT file segment that HLS uses, so it’s available in case the cues in the segment file reference it.
  • “NEXT” in end time markers would make authoring of live captioned VTT files easier.
  • If the application wants to use 1 word at a time and doesn’t want to delay sending the word until the full cue is authored (e.g. in a Hangout type environment), we will need to introduce the concept of “cue continuation markers”, so we know that a cue could be extended with the next VTT file fragment.

  This is an extensive and impressive amount of discussion around WebVTT and a lot of new work to be performed in the future. I’m very grateful for all the people who have contributed to these discussions at FOMS and will hopefully continue to help get the specifications right.