Recherche avancée

Sur d’autres sites (2048)

• How to properly close a FFmpeg stream and AVFormatContext without leaking memory ?

  13 décembre 2019, par Darkwonder

  I have built an app that uses FFmpeg to connect to remote IP cameras in order to receive video and audio frames via RTSP 2.0.

  The app is built using Xcode 10-11 and Objective-C with a custom FFmpeg build config.

  The architecture is the following :

  MyApp
  
  
  
  
  
  Document_0
  
  
  
      RTSPContainerObject_0
  
          RTSPObject_0
  
  
  
      RTSPContainerObject_1
  
          RTSPObject_1
  
  
  
      ...
  
  Document_1
  
  ...

  GOAL :

  1. After closing Document_0 no FFmpeg objects should be leaked.
  2. The closing process should stop-frame reading and destroy all objects which use FFmpeg.

  PROBLEM :

  

  1. Somehow Xcode’s memory debugger shows two instances of MyApp.

  FACTS :

  • macOS’es Activity Monitor doesn’t show two instances of MyApp.

  • macOS’es Activity Monitor doesn’t any instances of FFmpeg or other child processes.

  • The issue is not related to some leftover memory due to a late memory snapshot since it can be reproduced easily.

  • Xcode’s memory debugger shows that the second instance only having RTSPObject's AVFormatContext and no other objects.

    2. The second instance has an AVFormatContext and the RTPSObject still has a pointer to the AVFormatContext.

  FACTS :

  • Opening and closing the second document Document_1 leads to the same problem and having two objects leaked. This means that there is a bug that creates scalable problems. More and more memory is used and unavailable.

  Here is my termination code :

     - (void)terminate
  
  {
  
      // * Video and audio frame provisioning termination *
  
      [self stopVideoStream];
  
      [self stopAudioStream];
  
      // *
  
  
  
      // * Video codec termination *
  
      avcodec_free_context(&_videoCodecContext); // NULL pointer safe.
  
      self.videoCodecContext = NULL;
  
      // *
  
  
  
  // * Audio codec termination *
  
  avcodec_free_context(&_audioCodecContext); // NULL pointer safe.
  
  self.audioCodecContext = NULL;
  
  // *
  
  
  
  if (self.packet)
  
  {
  
      // Free the packet that was allocated by av_read_frame.
  
      av_packet_unref(&packet); // The documentation doesn't mention NULL safety.
  
      self.packet = NULL;
  
  }
  
  
  
  if (self.currentAudioPacket)
  
  {
  
      av_packet_unref(_currentAudioPacket);
  
      self.currentAudioPacket = NULL;
  
  }
  
  
  
  // Free raw frame data.
  
  av_freep(&_rawFrameData); // NULL pointer safe.
  
  
  
  // Free the swscaler context swsContext.
  
  self.isFrameConversionContextAllocated = NO;
  
  sws_freeContext(scallingContext); // NULL pointer safe.
  
  
  
  [self.audioPacketQueue removeAllObjects];
  
  
  
  self.audioPacketQueue = nil;
  
  
  
  self.audioPacketQueueLock = nil;
  
  self.packetQueueLock = nil;
  
  self.audioStream = nil;
  
  BXLogInDomain(kLogDomainSources, kLogLevelVerbose, @"%s:%d: All streams have been terminated!", __FUNCTION__, __LINE__);
  
  
  
  // * Session context termination *
  
  AVFormatContext *pFormatCtx = self.sessionContext;
  
  BOOL shouldProceedWithInputSessionTermination = self.isInputStreamOpen && self.shouldTerminateStreams && pFormatCtx;
  
  NSLog(@"\nTerminating session context...");
  
  if (shouldProceedWithInputSessionTermination)
  
  {
  
      NSLog(@"\nTerminating...");
  
      //av_write_trailer(pFormatCtx);
  
      // Discard all internally buffered data.
  
      avformat_flush(pFormatCtx); // The documentation doesn't mention NULL safety.
  
      // Close an opened input AVFormatContext and free it and all its contents.
  
      // WARNING: Closing an non-opened stream will cause avformat_close_input to crash.
  
      avformat_close_input(&pFormatCtx); // The documentation doesn't mention NULL safety.
  
      NSLog(@"Logging leftovers - %p, %p  %p", self.sessionContext, _sessionContext, pFormatCtx);
  
      avformat_free_context(pFormatCtx);
  
  
  
      NSLog(@"Logging content = %c", *self.sessionContext);
  
      //avformat_free_context(pFormatCtx); - Not needed because avformat_close_input is closing it.
  
      self.sessionContext = NULL;
  
  }
  
  // *
  
  
  
  }

  IMPORTANT : The termination sequence is :

      New frame will be read.
  
  -[(RTSPObject)StreamInput currentVideoFrameDurationSec]
  
  -[(RTSPObject)StreamInput frameDuration:]
  
  -[(RTSPObject)StreamInput currentCGImageRef]
  
  -[(RTSPObject)StreamInput convertRawFrameToRGB]
  
  -[(RTSPObject)StreamInput pixelBufferFromImage:]
  
  -[(RTSPObject)StreamInput cleanup]
  
  -[(RTSPObject)StreamInput dealloc]
  
  -[(RTSPObject)StreamInput stopVideoStream]
  
  -[(RTSPObject)StreamInput stopAudioStream]
  
  
  
  Terminating session context...
  
  Terminating...
  
  Logging leftovers - 0x109ec6400, 0x109ec6400  0x109ec6400
  
  Logging content = \330
  
  -[Document dealloc]

  NOT WORKING SOLUTIONS :

  • Changing the order of object releases (The AVFormatContext has been freed first but it didn’t lead to any change).
  • Calling RTSPObject's cleanup method much sooner to give FFmpeg more time to handle object releases.
  • Reading a lot of SO answers and FFmpeg documentation to find a clean cleanup process or newer code which might highlight why the object release doesn’t happen properly.

  I am currently reading the documentation on AVFormatContext since I believe that I am forgetting to release something. This believe is based on the memory debuggers output that AVFormatContext is still around.

  Here is my creation code :

  #pragma mark # Helpers - Start
  
  
  
  - (NSError *)openInputStreamWithVideoStreamId:(int)videoStreamId
  
                                  audioStreamId:(int)audioStreamId
  
                                       useFirst:(BOOL)useFirstStreamAvailable
  
                                         inInit:(BOOL)isInitProcess
  
  {
  
      // NSLog(@"%s", __PRETTY_FUNCTION__); // RTSP
  
      self.status = StreamProvisioningStatusStarting;
  
      AVCodec *decoderCodec;
  
      NSString *rtspURL = self.streamURL;
  
      NSString *errorMessage = nil;
  
      NSError *error = nil;
  
  
  
      self.sessionContext = NULL;
  
      self.sessionContext = avformat_alloc_context();
  
  
  
      AVFormatContext *pFormatCtx = self.sessionContext;
  
      if (!pFormatCtx)
  
      {
  
          // Create approp error.
  
          return error;
  
      }
  
  
  
  
  
      // MUST be called before avformat_open_input().
  
      av_dict_free(&_sessionOptions);
  
  
  
          self.sessionOptions = 0;
  
          if (self.usesTcp)
  
          {
  
              // "rtsp_transport" - Set RTSP transport protocols.
  
              // Allowed are: udp_multicast, tcp, udp, http.
  
              av_dict_set(&_sessionOptions, "rtsp_transport", "tcp", 0);
  
          }
  
          av_dict_set(&_sessionOptions, "rtsp_transport", "tcp", 0);
  
  
  
      // Open an input stream and read the header with the demuxer options.
  
      // WARNING: The stream must be closed with avformat_close_input()
  
      if (avformat_open_input(&pFormatCtx, rtspURL.UTF8String, NULL, &_sessionOptions) != 0)
  
      {
  
          // WARNING: Note that a user-supplied AVFormatContext (pFormatCtx) will be freed on failure.
  
          self.isInputStreamOpen = NO;
  
          // Create approp error.
  
          return error;
  
      }
  
  
  
      self.isInputStreamOpen = YES;
  
  
  
      // user-supplied AVFormatContext pFormatCtx might have been modified.
  
      self.sessionContext = pFormatCtx;
  
  
  
      // Retrieve stream information.
  
      if (avformat_find_stream_info(pFormatCtx,NULL) < 0)
  
      {
  
          // Create approp error.
  
          return error;
  
      }
  
  
  
      // Find the first video stream
  
      int streamCount = pFormatCtx->nb_streams;
  
  
  
      if (streamCount == 0)
  
      {
  
          // Create approp error.
  
          return error;
  
      }
  
  
  
      int noStreamsAvailable = pFormatCtx->streams == NULL;
  
  
  
      if (noStreamsAvailable)
  
      {
  
          // Create approp error.
  
          return error;
  
      }
  
  
  
      // Result. An Index can change, an identifier shouldn't.
  
      self.selectedVideoStreamId = STREAM_NOT_FOUND;
  
      self.selectedAudioStreamId = STREAM_NOT_FOUND;
  
  
  
      // Fallback.
  
      int firstVideoStreamIndex = STREAM_NOT_FOUND;
  
      int firstAudioStreamIndex = STREAM_NOT_FOUND;
  
  
  
      self.selectedVideoStreamIndex = STREAM_NOT_FOUND;
  
      self.selectedAudioStreamIndex = STREAM_NOT_FOUND;
  
  
  
      for (int i = 0; i < streamCount; i++)
  
      {
  
          // Looking for video streams.
  
          AVStream *stream = pFormatCtx->streams[i];
  
          if (!stream) { continue; }
  
          AVCodecParameters *codecPar = stream->codecpar;
  
          if (!codecPar) { continue; }
  
  
  
          if (codecPar->codec_type==AVMEDIA_TYPE_VIDEO)
  
          {
  
              if (stream->id == videoStreamId)
  
              {
  
                  self.selectedVideoStreamId = videoStreamId;
  
                  self.selectedVideoStreamIndex = i;
  
              }
  
  
  
              if (firstVideoStreamIndex == STREAM_NOT_FOUND)
  
              {
  
                  firstVideoStreamIndex = i;
  
              }
  
          }
  
          // Looking for audio streams.
  
          if (codecPar->codec_type==AVMEDIA_TYPE_AUDIO)
  
          {
  
              if (stream->id == audioStreamId)
  
              {
  
                  self.selectedAudioStreamId = audioStreamId;
  
                  self.selectedAudioStreamIndex = i;
  
              }
  
  
  
              if (firstAudioStreamIndex == STREAM_NOT_FOUND)
  
              {
  
                  firstAudioStreamIndex = i;
  
              }
  
          }
  
      }
  
  
  
      // Use first video and audio stream available (if possible).
  
  
  
      if (self.selectedVideoStreamIndex == STREAM_NOT_FOUND && useFirstStreamAvailable && firstVideoStreamIndex != STREAM_NOT_FOUND)
  
      {
  
          self.selectedVideoStreamIndex = firstVideoStreamIndex;
  
          self.selectedVideoStreamId = pFormatCtx->streams[firstVideoStreamIndex]->id;
  
      }
  
  
  
      if (self.selectedAudioStreamIndex == STREAM_NOT_FOUND && useFirstStreamAvailable && firstAudioStreamIndex != STREAM_NOT_FOUND)
  
      {
  
          self.selectedAudioStreamIndex = firstAudioStreamIndex;
  
          self.selectedAudioStreamId = pFormatCtx->streams[firstAudioStreamIndex]->id;
  
      }
  
  
  
      if (self.selectedVideoStreamIndex == STREAM_NOT_FOUND)
  
      {
  
          // Create approp error.
  
          return error;
  
      }
  
  
  
      // See AVCodecID for codec listing.
  
  
  
      // * Video codec setup:
  
      // 1. Find the decoder for the video stream with the gived codec id.
  
      AVStream *stream = pFormatCtx->streams[self.selectedVideoStreamIndex];
  
      if (!stream)
  
      {
  
          // Create approp error.
  
          return error;
  
      }
  
      AVCodecParameters *codecPar = stream->codecpar;
  
      if (!codecPar)
  
      {
  
          // Create approp error.
  
          return error;
  
      }
  
  
  
      decoderCodec = avcodec_find_decoder(codecPar->codec_id);
  
      if (decoderCodec == NULL)
  
      {
  
          // Create approp error.
  
          return error;
  
      }
  
  
  
      // Get a pointer to the codec context for the video stream.
  
      // WARNING: The resulting AVCodecContext should be freed with avcodec_free_context().
  
      // Replaced:
  
      // self.videoCodecContext = pFormatCtx->streams[self.selectedVideoStreamIndex]->codec;
  
      // With:
  
      self.videoCodecContext = avcodec_alloc_context3(decoderCodec);
  
      avcodec_parameters_to_context(self.videoCodecContext,
  
                                    codecPar);
  
  
  
      self.videoCodecContext->thread_count = 4;
  
      NSString *description = [NSString stringWithUTF8String:decoderCodec->long_name];
  
  
  
      // 2. Open codec.
  
      if (avcodec_open2(self.videoCodecContext, decoderCodec, NULL) < 0)
  
      {
  
          // Create approp error.
  
          return error;
  
      }
  
  
  
      // * Audio codec setup:
  
      if (self.selectedAudioStreamIndex > -1)
  
      {
  
          [self setupAudioDecoder];
  
      }
  
  
  
      // Allocate a raw video frame data structure. Contains audio and video data.
  
      self.rawFrameData = av_frame_alloc();
  
  
  
      self.outputWidth = self.videoCodecContext->width;
  
      self.outputHeight = self.videoCodecContext->height;
  
  
  
      if (!isInitProcess)
  
      {
  
          // Triggering notifications in init process won't change UI since the object is created locally. All
  
          // objects which need data access to this object will not be able to get it. Thats why we don't notifiy anyone about the changes.
  
          [NSNotificationCenter.defaultCenter postNotificationName:NSNotification.rtspVideoStreamSelectionChanged
  
                                                            object:nil userInfo: self.selectedVideoStream];
  
  
  
          [NSNotificationCenter.defaultCenter postNotificationName:NSNotification.rtspAudioStreamSelectionChanged
  
                                                            object:nil userInfo: self.selectedAudioStream];
  
      }
  
  
  
      return nil;
  
  }

  UPDATE 1

  The initial architecture allowed using any given thread. Most of the below code would mostly run on the main thread. This solution was not appropriate since the opening of the stream input can take several seconds for which the main thread is blocked while waiting for a network response inside FFmpeg. To solve this issue I have implemented the following solution :

  • Creation and the initial setup are only allowed on the background_thread (see code snippet "1" below).
  • Changes are allowed on the current_thread(Any).
  • Termination is allowed on the current_thread(Any).

  After removing main thread checks and dispatch_asyncs to background threads, leaking has stopped and I can’t reproduce the issue anymore :

  // Code that produces the issue.   
  
  dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
  
      // 1 - Create and do initial setup. 
  
      // This block creates the issue. 
  
  [self.rtspObject = [[RTSPObject alloc] initWithURL: ... ];
  
  [self.rtspObject openInputStreamWithVideoStreamId: ...
  
                                  audioStreamId: ...
  
                                       useFirst: ...
  
                                         inInit: ...];
  
  });

  I still don’t understand why Xcode’s memory debugger says that this block is retained ?

  Any advice or idea is welcome.

• ffmpeg background worker runs in debug but not in application

  2 novembre 2019, par Purgitoria

  My application has a function of taking captured images and using an FFmpeg background worker to stitch these into a time-lapse video. The GUI has some simple options for video quality and for the source folder and output file. I had an older version of my application written in VB.NET and that worked without issue but I am rewriting in C# as it supports additional capture and filter capability in the image processing but am having real trouble figuring out what is wrong with this function.

  I have tried relocating FFmpeg to different locations just in case it was a permissions issue but that had no effect and I also tried to put the function in a "try" with a message box to output any exceptions but I got different errors that prevented me from compiling the code. When I run the application from within VS 2015 in the debugging tool the function works just fine and it will create a video from a collection of still images but when I build and install the application it does not work at all and I cannot see what is causing it to fail. In the options for ffmpeg I used the -report to output a log of what happens in the background worker and in debug, it creates this log but from the application, it does not so I presume it is not even running ffmpeg and going straight to the completed stage of the function.

  Function startConversion()
  
  
  
      CheckForIllegalCrossThreadCalls = False
  
      Dim quality As Integer = trbQuality.Value
  
      Dim input As String = tbFolderOpen.Text
  
      Dim output As String = tbFolderSave.Text
  
      Dim exepath As String = Application.StartupPath + "\\bin\ffmpeg.exe"
  
      input = input & "\SCAImg_%1d.bmp"
  
      input = Chr(34) & input & Chr(34)
  
      output = Chr(34) & output & Chr(34)
  
  
  
      Dim sr As StreamReader
  
      Dim ffmpegOutput As String
  
  
  
      ' all parameters required to run the process
  
      proc.StartInfo.UseShellExecute = False
  
      proc.StartInfo.CreateNoWindow = True
  
      proc.StartInfo.RedirectStandardError = True
  
      proc.StartInfo.FileName = exepath
  
      proc.StartInfo.Arguments = "-framerate 25 -start_number 0 -pattern_type sequence -framerate 10 -i " & input & " -r 10 -c:v libx264 -preset slow -crf " & quality & " " & output
  
      proc.Start()
  
  
  
      lblInfo.Text = "Conversion in progress... Please wait..."
  
      sr = proc.StandardError 'standard error is used by ffmpeg
  
      btnMakeVideo.Enabled = False
  
      Do
  
          ffmpegOutput = sr.ReadLine
  
          tbProgress.Text = ffmpegOutput
  
      Loop Until proc.HasExited And ffmpegOutput = Nothing Or ffmpegOutput = ""
  
  
  
      tbProgress.Text = "Finished !"
  
      lblInfo.Text = "Completed!"
  
      MsgBox("Completed!", MsgBoxStyle.Exclamation)
  
      btnMakeVideo.Enabled = True
  
      Return 0
  
  
  
  End Function

  I checked the application folder and it does contain a subfolder \bin withe the ffmpeg.exe located within the folder so I then used cmd to run an instance of the installed ffmpeg from the application folder and it seemed to be throwing out permissions errors :

  Failed to open report "ffmpeg-20191101-191452.log" : Permission denied
  Failed to set value ’1’ for option ’report’ : Permission denied
  Error parsing global options : Permission denied

  This seems then like it is certainly a permissions problem but where I am not sure. I did not run into this error when using VB.NET so I am wondering where I am going wrong now. I thought perhaps it would just be a write permission in the application folder so I the removed the -report and ran ffmpeg again using cmd from my application folder and it then gave the error

  C :\Users\CEAstro\Pictures\AnytimeCap : Permission denied

  Am I missing something really obvious in my code or is there something more fundamental I have wrong in my setup ?

  I should also add that I tried running ffmpeg via cmd from a copy that was manually placed elsewhere (i used the same file) and that actually worked. For some reason, it seems like it just will not work from wherever my application installs it.

• Can't share few FFMPEG encoded videos on WhatsApp

  2 juin 2019, par t6nand

  I am using FFMPEG to stitch a video to another video. However, I have observed that few of these videos are not being shared on WhatsApp with the message "Can’t send this video. Choose a different video and try again".

  I am using the following command for stitching :

  ffmpeg -y -i <input1> -i
  
   <input2> -filter_complex "color=black:input1Width x input1Height:d=(input1time + input2time -1)[base]
  
  ;[0:v]setpts=PTS-STARTPTS[v0];[1:v] scale = iw * min(input1Width / iw\\, input1Height / ih):ih * min(input1Width/iw\\,input1Height/ih),
  
  pad=input1Width:input1Height:(input1Width-iw*min(input1Width/iw\\,input1Height/ih))/2:(input1Height -
  
   ih*min(input1Width/iw\\,input1Height/ih))/2,setsar=1:1,format=yuva420p,fade=in:st=0:d=1.0:alpha=1,
  
  setpts=PTS-STARTPTS+((input1Time - 1)/TB)[v1];[base][v0]overlay[tmp]; [tmp][v1]overlay,format=yuv420p[fv];
  
  [0:a][1:a]acrossfade=d=1[fa]" -map [fv] -map [fa] -crf 23 -c:v libx264 -b:v 300K 
  
  -preset slow outputvideo.mp4
  
  </input2></input1>

  Here : input1Width - Input 1 Video’s width, input1Height - Input 2 Video’s height.

  Interestingly running this command directly on terminal produces a file which plays correctly on media players and I am able to share it across platforms including WhatsApp.

  However, the same command when triggered from a java code produces a similar file in terms of size, bit rate (near about same as produced when directly executed). I am running the same command from java using ProcessBuilder like :

  ProcessBulider stitchVideoCommandArray = new ProcessBuilder(
  
                          "ffmpeg", "-y",
  
                          "-i", <input1>,
  
                          "-i", <input2>,
  
                          "-filter_complex", "color=black:" + String.valueOf(width) + "x" + String.valueOf(height) + ":d=" +
  
                          String.valueOf(originalVideoTime + ASSET_VIDEOSTREAM_DURATION - CROSS_FADE_TIME_DURATION) + "[base];" +
  
                          "[0:v]setpts=PTS-STARTPTS[v0];[1:v]scale=iw*" + String.valueOf(minMultiplicable) + ":ih*" + String.valueOf(minMultiplicable) +
  
                          ",pad=" + String.valueOf(width) + ":" + String.valueOf(height) + ":(" + String.valueOf(width) + "-iw*" + String.valueOf(minMultiplicable) + "/2" +
  
                          "):(" + String.valueOf(height) + "-ih*" + String.valueOf(minMultiplicable) + "/2)" + ",fade=in" +
  
                          ":st=0:d=" + String.valueOf(CROSS_FADE_TIME_DURATION) + ":alpha=1,setpts=PTS-STARTPTS+((" + String.valueOf(originalVideoTime - CROSS_FADE_TIME_DURATION) +
  
                          ")/TB)[v1];[base][v0]overlay[tmp];[tmp][v1]overlay,format=yuv420p[fv];[0:a][1:a]acrossfade=d=1[fa]",
  
                          "-map", "[fv]",
  
                          "-map", "[fa]",
  
                          "-c:v", MP4Transcode.MP4VideoStreamEncoder.H264.videoEncoders,
  
                          "-b:v", "300K",
  
                          "-c:a", MP4Transcode.MP4AudioStreamEncoders.AAC.audioEncoders,
  
                          "-b:a", "128K",
  
                          "-crf", String.valueOf(constantRateFactor),
  
                          "-preset", presetRequired,
  
                      outputVideoPath
  
              );
  
  </input2></input1>

  I then use this object to execute the command using

  command.start()

  and waiting for exitvalue using

  command.waitFor()

  This also produces a file which I am able to play correctly on media players but was not successful in sharing the video on WhatsApp.

  Also, the file produced by executing FFmpeg commands as mentioned above from java produces media files with their loudness slightly increased.

  I am not able to figue out if it’s video specific issue or something else. It only happens to few random videos.

  Here are the links to videos with the problem :
  input 1 - input1Link
  input 2 - inpt2Link

  EDIT 1 :
  To add, adding -loglevel debug in java command takes forever and I haven’t seen it yet responding for even 5-6 minutes. However, using it directly from terminal returns quickly after processing.