Recherche avancée

Sur d’autres sites (7258)

• Programmatically accessing PTS times in MP4 container

  9 novembre 2022, par mcandril

  Background

  


  For a research project, we are recording video data from two cameras and feed a synchronization pulse directly into the microphone ADC every second.

  


  Problem

  


  We want to derive a frame time stamp in the clock of the pulse source for each camera frame to relate the camera images temporally. With our current methods (see below), we get a frame offset of around 2 frames between the cameras. Unfortunately, inspection of the video shows that we are clearly 6 frames off (at least at one point) between the cameras.
I assume that this is because we are relating audio and video signal wrong (see below).

  


  Approach I think I need help with

  


  I read that in the MP4 container, there should be PTS times for video and audio. How do we access those programmatically. Python would be perfect, but if we have to call ffmpeg via system calls, we may do that too ...

  


  What we currently fail with

  


  The original idea was to find video and audio times as

  


  audio_sample_times = range(N_audiosamples)/audio_sampling_rate
video_frame_times = range(N_videoframes)/video_frame_rate


  


  then identify audio_pulse_times in audio_sample_times base, calculate the relative position of each video_time to the audio_pulse_times around it, and select the same relative value to the corresponding source_pulse_times.

  


  However, a first indication that this approach is problematic is already that for some videos, N_audiosamples/audio_sampling_rate differs from N_videoframes/video_frame_rate by multiple frames.

  


  What I have found by now

  


  OpenCV's cv2.CAP_PROP_POS_MSEC seems to do exactly what we do, and not access any PTS ...

  


  Edit : What I took from the winning answer

  


  container = av.open(video_path)
signal = []
audio_sample_times = []
video_sample_times = []

for frame in tqdm(container.decode(video=0, audio=0)):
    if isinstance(frame, av.audio.frame.AudioFrame):
        sample_times = (frame.pts + np.arange(frame.samples)) / frame.sample_rate
        audio_sample_times += list(sample_times)
        signal_f_ch0 = frame.to_ndarray().reshape((-1, len(frame.layout.channels))).T[0]
        signal += list(signal_f_ch0)
    elif isinstance(frame, av.video.frame.VideoFrame):
        video_sample_times.append(float(frame.pts*frame.time_base))

signal = np.abs(np.array(signal))
audio_sample_times = np.array(audio_sample_times)
video_sample_times = np.array(video_sample_times)


  


  Unfortunately, in my particular case, all pts are consecutive and gapless, so the result is the same as with the naive solution ...
By picture clues, we identified a section of 10s in the videos, somewhere in which they desync, but can't find any traces of that in the data.

  


• FFMPEG crop a portrait image square with respect to the zoomed dimensions and x/y pan

  3 octobre 2022, par hugger

  I am making a photo crop component for my mobile app.

  


  If an image is selected from the image picker and it is above 4:5 ratio, using FFMPEG I need to crop this image 1:1 using the dynamic X/Y values along with the scale value from pinching (1.00 - 10).

  


  I trying to use these values with the FFMPEG crop/scale filter, but no matter what I try I cant seem to get the crop to work as expected... It is not matching up from the UI pan / zoom.

  


  My FFMPEG command looks like this, along with some other relevant variable code :

  


  let zoom = this._scale.__getValue(); //set dynamically by the user; ranges from 0.1 to 10
let translateY = this._translateX.__getValue(); //amount from left side
let translateX = this._translateY.__getValue(); // amount from top

//trying to first crop the width and height / the zoom for the zoom scaling..?
//then I am using translate X/Y to get my coordinates (not matching up)
//I tried to use translateXY * zoom to get the scaling factor but it still did not work...
//the image needs to be square, chain scale filter to make this happen after the original crop?)
FFmpegKit.execute(
  `-y -i ${this.state.mediaSource} -vf "crop=iw/${zoom}:ih/${zoom}:${translateX}:${translateY}, scale=iw:iw:0:0" -qscale 0 -frames:v 1 ${filterPathPostCrop}`
).then(async (session) => {
  const returnCode = await session.getReturnCode();
  if (ReturnCode.isSuccess(returnCode)) {
    this.setState({
      mediaSource: filterPathPostCrop,
      videoSourcePreview: `${filterPathPostCrop}?${new Date().getTime()}`,
      ffMPEGinProgress: null,
      aspectRatio: 1080 / 1080,
      videoTime: 0,
      isPlayingVideo: false,
      isCropping: false,
      filterOutputIsAlt: !this.state.filterOutputIsAlt,
      wasCropped: true,
    });
  } else if (ReturnCode.isCancel(returnCode)) {
    // CANCEL
  } else {
    // ERROR
    alert('error');
  }
});


  


  I appreciate any guidance I can get with this, I feel like I am close I just cant seem to get this calculation working...

  


• FFMPEG crop a portrait image square with pan and zoom scale in mind

  30 septembre 2022, par hugger

  I am making a photo crop component for my mobile app.

  


  If an image is selected from the image picker and it is above 4:5 ratio, using FFMPEG I need to crop this image 1:1 using the dynamic X/Y values along with the scale value from pinching (1.00 - 10).

  


  I trying to use these values with the FFMPEG crop/scale filter, but no matter what I try I cant seem to get the crop to work as expected... It is not matching up from the UI pan / zoom.

  


  My FFMPEG command looks like this, along with some other relevant variable code :

  


  let zoom = this._scale.__getValue(); //set dynamically by the user; ranges from 0.1 to 10
let translateY = this._translateX.__getValue(); //amount from left side
let translateX = this._translateY.__getValue(); // amount from top

//trying to first crop the width and height / the zoom for the zoom scaling..?
//then I am using translate X/Y to get my coordinates (not matching up)
//I tried to use translateXY * zoom to get the scaling factor but it still did not work...
//the image needs to be square, chain scale filter to make this happen after the original crop?)
FFmpegKit.execute(
  `-y -i ${this.state.mediaSource} -vf "crop=iw/${zoom}:ih/${zoom}:${translateX}:${translateY}, scale=iw:iw:0:0" -qscale 0 -frames:v 1 ${filterPathPostCrop}`
).then(async (session) => {
  const returnCode = await session.getReturnCode();
  if (ReturnCode.isSuccess(returnCode)) {
    this.setState({
      mediaSource: filterPathPostCrop,
      videoSourcePreview: `${filterPathPostCrop}?${new Date().getTime()}`,
      ffMPEGinProgress: null,
      aspectRatio: 1080 / 1080,
      videoTime: 0,
      isPlayingVideo: false,
      isCropping: false,
      filterOutputIsAlt: !this.state.filterOutputIsAlt,
      wasCropped: true,
    });
  } else if (ReturnCode.isCancel(returnCode)) {
    // CANCEL
  } else {
    // ERROR
    alert('error');
  }
});


  


  I appreciate any guidance I can get with this, I feel like I am close I just cant seem to get this calculation working...