Recherche avancée

Sur d’autres sites (8460)

• First/single frame encoded with ffmpeg/libavcodec library cannot be immediately decoded

  11 septembre 2023, par Marek Kijo

  I'm using libavcodec library and h264 codec to prepare the video stream on one end, transmit the encoded frames to the other PC and there decode it.

  


  What I noticed after receiving very first packet (first encoded video frame) and feeding decoder with it, it is not possible to decode that frame. Only when I receive another frame the first one can be decoded but 'current' one not. So in the end I have constantly one frame delay on the decoder side.

  


  I was trying different presets (focusing rather on 'ultrafast'), also 'zerolatency' tune, also whole variety of bit_rate values of AVCodecContext.

  


  I also tried to flush (with nullptr packet) after injecting first frame data, just to check if it is maybe because of some internal buffers optimization - the frame still not decoded.
Experimenting with other codecs (like mpeg4) gives even worse 'dalay' in number of frames to the point when when first frames can become decodable.

  


  Is it normal, unavoidable because of some internal mechanisms ? Otherwise how I can achieve real zero latency.

  


  Supplementary setup information :

  


  

  • max_b_frames set to 0 (higher value gives even more delay)

  


  • pix_fmt set to AV_PIX_FMT_YUV420P

  


  


  edit :

  


  Answering some comment question :

  


  


  (1) What is the decoder (or playback system) ?

  


  


  Custom decoder written using libavcodec, the decoded frames are later displayed on screen by OpenGL.

  


  

  • initialization :

  


  


  parser_ = av_parser_init(AV_CODEC_ID_H264);
codec_ = avcodec_find_decoder(AV_CODEC_ID_H264);
context_ = avcodec_alloc_context3(codec_);
context_->width = 1024;
context_->height = 768;
context_->thread_count = 1;
if ((codec_->capabilities & AV_CODEC_CAP_TRUNCATED) == 0)
{
  context_->flags |= AV_CODEC_FLAG_TRUNCATED;
}
if (avcodec_open2(context_, codec_, nullptr) < 0)
{
  throw std::runtime_error{"avcodec_open2 failed"};
}
avcodec_flush_buffers(context_);


  


  

  • then player periodically calls of the method of decoder that suppose to check if the another frame can be retrieved and displayed :

  


  


  auto result = avcodec_receive_frame(context_, frame_);
if (!buffer_.empty())
{  // upload another packet for decoding
  int used;
  if (upload_package(buffer_.data(), buffer_.size(), &used))
  {
    buffer_.erase(buffer_.begin(), buffer_.begin() + used);
  }
}
if (result == AVERROR(EAGAIN) || result == AVERROR_EOF || result < 0)
{
  return false;
}
yuv_to_rgb();
return true;


  


  boolean return value informs if the decoding succeeded, and every time the buffer where the incomming packets are stored is checked and uploaded to libavcodec decoder

  


  

  • and that is how the method that uploads the buffer looks like :

  


  


  bool upload_package(const void* data, const std::size_t size, int* used)&#xA;{&#xA;  auto result = av_parser_parse2(parser_, context_, &amp;packet_->data, &amp;packet_->size, reinterpret_cast<const>(data), size, AV_NOPTS_VALUE, AV_NOPTS_VALUE, 0);&#xA;  if (result &lt; 0)&#xA;  {&#xA;    return false;&#xA;  }&#xA;  *used = result;&#xA;  if (packet_->size != 0)&#xA;  {&#xA;    result = avcodec_send_packet(context_, packet_);&#xA;    if (result &lt; 0)&#xA;    {&#xA;      return false;&#xA;    }&#xA;  }&#xA;  return true;&#xA;}&#xA;</const>

  


  


  (2) If possible, save each one as a .bin file and then share the links with us for testing.

  


  


  I will try to figure out something...

  


  


  (3) Show example C++ code of your encoder settings for H264...

  


  


  

  • initialization :

  


  


  codec_ = avcodec_find_encoder(AV_CODEC_ID_H264);
context_ = avcodec_alloc_context3(codec_);

context_->bit_rate = 1048576; // 1xMbit;
context_->width = 1024;
context_->height = 768;
context_->time_base = {1, 30}; // 30 fps
context_->pix_fmt = AV_PIX_FMT_YUV420P;
context_->thread_count = 1;

av_opt_set(context_->priv_data, "preset", "ultrafast", 0);
av_opt_set(context_->priv_data, "tune", "zerolatency", 0);
avcodec_open2(context_, codec_, nullptr);

frame_->format = AV_PIX_FMT_YUV420P;
frame_->width = 1024;
frame_->height = 768;
av_image_alloc(frame_->data, frame_->linesize, 1024, 768, AV_PIX_FMT_YUV420P, 32);


  


  

  • frame encoding :

  


  


  rgb_to_yuv();
frame_->pts = frame_num_++;
auto result = avcodec_send_frame(context_, frame_);
while (result >= 0)
{
  result = avcodec_receive_packet(context_, packet_);
  if (result == AVERROR(EAGAIN) || result == AVERROR_EOF)
  {
    return;
  }
  else if (result < 0)
  {
    throw std::runtime_error{"avcodec_receive_packet failed"};
  }
  // here the packet is send to the decoder, the whole packet is stored on the mentioned before buffer_ and uploaded with avcodec_send_packet
  // I can also add that the whole buffer/packet us uploaded at once
  stream_video_data(packet_->data, packet_->size); 
}
av_packet_unref(packet_);
}


  


  edit2 :

  


  I think I figured out the issue that I had.

  


  For every incoming data packet (encoded frame) I was calling first av_parser_parse2, and then I was sending the data through avcodec_send_packet.
And I was not recalling that procedure having empty buffer_, so for the first frame data the av_parser_parse2 was never called after uploading it through avcodec_send_packet, for the second frame it was called and the first frame was parsed, so it could be properly decoded, but for that (second) frame the parse2 was also not called, and so on ...

  


  So the issue in my case was wrong sequence of av_parser_parse2 and avcodec_send_packet to handle the encoded data.

  


• HLS encoded video audio (ffmpeg) drops when seeking past buffer on Safari

  13 juin 2019, par eschie

  Some of my HLS encoded videos via ffmpeg drop the audio when seeking past the buffer. The only way to sync the audio and video up again is to restart the video. What would be causing this ?

  Example Profile :

  bitrate: 4800, profile: 'high', level: '4.1', resolution: 1080, framerate: '24000/1001'

  ffmpeg
  
      '-y' 
  
      '-i' input_file.mov                                                 
  
      '-v' error 
  
      '-map' '0:0'                                                        
  
      '-c:v' libx264                                                  
  
      '-x264opts' f'
  
          keyint=23:                  
  
          min-keyint=23:          
  
          no-scenecut                                                     
  
      '
  
      '-vf' f'scale=-1:1080'
  
      '-preset' 'slow'
  
      '-profile:v' 'high'
  
      '-level' '4.1'
  
      '-b:v' '4800k'
  
      '-maxrate' '4800k'
  
      '-movflags' 'faststart'
  
      '-bufsize' '9600k' 
  
      '-write_tmcd', '0'
  
      '-r' '24000/1001'                                   
  
      output_dir                                                      

  Segmentation CMD :

  FFMPEG 
  
      '-i' output_dir 
  
      '-v' 'error' 
  
      '-acodec' 'copy'
  
      '-vcodec' 'copy'
  
      '-hls_time' '4' #seconds
  
      '-hls_list_size' '0'
  
      '-hls_flags' 'single_file' 
  
      os.path.join(output_dir, f'{run_id}_{bitrate}.m3u8'

  Added : apple’s mediastreamvalidator outputs a few different errors :

  Error: Playlist vs segment duration mismatch
  
  --> Detail:  Segment duration 98.0146, Playlist duration: 4.7968
  
  --> Source:  1559962503399_2200k.m3u8 - 1559962503399_2200k.ts:1746520@0

  Error: Measured peak bitrate compared to master playlist declared value exceeds error tolerance
  
  --> Detail:  Measured: 3182.61 kb/s, Master playlist: 2173.82 kb/s, Error: 46.41%, Combined rendition name: English
  
  --> Source:  ...playlist.m3u8
  
  --> Compare: 1559962503399_2200k.m3u8

  Error: Different target durations detected
  
  --> Detail:  Target duration: 5 vs Target duration: 4
  
  --> Source:  1559962503399_64k.m3u8
  
  --> Compare: 1559962503399_128k.m3u8

• Incorrect duration and bitrate in ffmpeg-encoded audio

  30 mai 2019, par Timmy K

  I am encoding raw data on Android using ffmpeg libraries. The native code reads the audio data from an external device and encodes it into AAC format in an mp4 container. I am finding that the audio data is successfully encoded (I can play it with Groove Music, my default Windows audio player). But the metadata, as reported by ffprobe, has an incorrect duration of 0.05 secs - it’s actually several seconds long. Also the bitrate is reported wrongly as around 65kbps even though I specified 192kbps.

  I’ve tried recordings of various durations but the result is always similar - the (very small) duration and bitrate. I’ve tried various other audio players such as Quicktime but they play only the first 0.05 secs or so of the audio.

  I’ve removed error-checking from the following. The actual code checks every call and no problems are reported.

  Initialisation :

  void AudioWriter::initialise( const char *filePath )
  
  {
  
      AVCodecID avCodecID = AVCodecID::AV_CODEC_ID_AAC;
  
      int bitRate = 192000;
  
      char *containerFormat = "mp4";
  
      int sampleRate = 48000;
  
      int nChannels = 2;
  
  
  
      mAvCodec = avcodec_find_encoder(avCodecID);
  
      mAvCodecContext = avcodec_alloc_context3(mAvCodec);
  
      mAvCodecContext->codec_id = avCodecID;
  
      mAvCodecContext->codec_type = AVMEDIA_TYPE_AUDIO;
  
      mAvCodecContext->sample_fmt = AV_SAMPLE_FMT_FLTP;
  
      mAvCodecContext->bit_rate = bitRate;
  
      mAvCodecContext->sample_rate = sampleRate;
  
      mAvCodecContext->channels = nChannels; 
  
      mAvCodecContext->channel_layout = AV_CH_LAYOUT_STEREO;
  
  
  
      avcodec_open2( mAvCodecContext, mAvCodec, nullptr );
  
  
  
      mAvFormatContext = avformat_alloc_context();
  
  
  
      avformat_alloc_output_context2(&mAvFormatContext, nullptr, containerFormat, nullptr);
  
      mAvFormatContext->audio_codec = mAvCodec;
  
      mAvFormatContext->audio_codec_id = avCodecID;
  
      mAvOutputStream = avformat_new_stream(mAvFormatContext, mAvCodec);
  
      avcodec_parameters_from_context(mAvOutputStream->codecpar, mAvCodecContext);
  
      if (!(mAvFormatContext->oformat->flags & AVFMT_NOFILE))
  
      {
  
          avio_open(&mAvFormatContext->pb, filePath, AVIO_FLAG_WRITE);
  
      }
  
  
  
      if ( mAvFormatContext->oformat->flags & AVFMT_GLOBALHEADER )
  
      {
  
          mAvCodecContext->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
  
      }
  
  
  
      avformat_write_header(mAvFormatContext, NULL);
  
  
  
      mAvAudioFrame = av_frame_alloc();
  
      mAvAudioFrame->nb_samples = mAvCodecContext->frame_size;
  
      mAvAudioFrame->format = mAvCodecContext->sample_fmt;
  
      mAvAudioFrame->channel_layout = mAvCodecContext->channel_layout;
  
  
  
      av_samples_get_buffer_size(NULL, mAvCodecContext->channels, mAvCodecContext->frame_size,
  
                                                   mAvCodecContext->sample_fmt, 0);
  
      av_frame_get_buffer(mAvAudioFrame, 0);
  
      av_frame_make_writable(mAvAudioFrame);
  
      mAvPacket = av_packet_alloc();
  
    }

  Encoding :

  // SoundRecording is a custom class with the raw samples to be encoded
  
  bool AudioWriter::encodeToContainer( SoundRecording *soundRecording )
  
  {
  
      int ret;
  
      int frameCount = mAvCodecContext->frame_size;
  
      int nChannels = mAvCodecContext->channels;
  
      float *buf = new float[frameCount*nChannels];
  
  
  
      while ( soundRecording->hasReadableData() )
  
      {
  
          //Populate the frame
  
          int samplesRead = soundRecording->read( buf, frameCount*nChannels );
  
          // Planar data
  
          int nFrames = samplesRead/nChannels;
  
          for ( int i = 0; i < nFrames; ++i )
  
          {
  
              for (int c = 0; c < nChannels; ++c )
  
              {
  
                  samples[c][i] = buf[nChannels*i +c];
  
              }
  
          }
  
          // Fill a gap at the end with silence
  
          if ( samplesRead < frameCount*nChannels )
  
          {
  
              for ( int i = samplesRead; i < frameCount*nChannels; ++i )
  
              {
  
                  for (int c = 0; c < nChannels; ++c )
  
                  {
  
                      samples[c][i] = 0.0;
  
                  }
  
              }
  
          }
  
  
  
      encodeFrame( mAvAudioFrame ) )
  
      }
  
  
  
      finish();
  
   }
  
  
  
  bool AudioWriter::encodeFrame( AVFrame *frame )
  
  {
  
      //send the frame for encoding
  
      int ret;
  
  
  
      if ( frame != nullptr )
  
      {
  
          frame->pts = mAudFrameCounter++;
  
      }
  
      avcodec_send_frame(mAvCodecContext, frame );
  
  
  
      while (ret >= 0)
  
      {
  
          ret = avcodec_receive_packet(mAvCodecContext, mAvPacket);
  
          if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF )
  
          {
  
              break;
  
          }
  
          else
  
              if (ret < 0) {
  
               return false;
  
          }
  
          av_packet_rescale_ts(mAvPacket, mAvCodecContext->time_base, mAvOutputStream->time_base);
  
          mAvPacket->stream_index = mAvOutputStream->index;
  
  
  
          av_interleaved_write_frame(mAvFormatContext, mAvPacket);
  
           av_packet_unref(mAvPacket);
  
      }
  
  
  
      return true;
  
  }
  
  
  
  void AudioWriter::finish()
  
  {
  
      // Flush by sending a null frame
  
      encodeFrame( nullptr );
  
  
  
      av_write_trailer(mAvFormatContext);
  
  }

  Since the resultant file contains the recorded music, the code to manipulate the audio data seems to be correct (unless I am overwriting other memory somehow).

  The inaccurate duration and bitrate suggest that information concerning time is not being properly managed. I set the pts of the frames using a simple increasing integer. I’m unclear what the code that sets the timestamp and stream index achieves - and whether it’s even necessary : I copied it from supposedly working code but I’ve seen other code without it.

  Can anyone see what I’m doing wrong ?