Recherche avancée

Sur d’autres sites (11732)

• ffmpeg crop with negative offset

  12 août 2020, par Ajouve

  I have a nodejs application which is cropping videos using ffmpeg

  


  From time to time I have an error because I am trying to crop out of the video, see attached image where the black is the video and in red the crop zone. My final video has to be a square.

  


  

  


  If I am replacing the negative offset with 0 the final result will not be a square.

  


  I just need to add a black background on the non-existing part

  


  This is my actual code

  


  const cropVideo = (buffer, width, height, x, y) => {

    const inputFile = tmp.fileSync();
    const outputFile = tmp.fileSync();

    fs.writeFileSync(inputFile.name, buffer);

    return new Promise((resolve, reject) => {
        ffmpeg(inputFile.name)
            .videoFilters(`crop=${width}:${height}:${x}:${y}`)
            .format('mp4')
            .on('error', reject)
            .on('end', () => resolve(fs.readFileSync(outputFile.name)))
            .save(outputFile.name);
    })
}


  


• Decoding MediaRecorder produced webm stream

  15 août 2019, par sgmg

  I am trying to decode a video stream from the browser using the ffmpeg API. The stream is produced by the webcam and recorded with MediaRecorder as webm format. What I ultimately need is a vector of opencv cv::Mat objects for further processing.

  I have written a C++ webserver using the uWebsocket library. The video stream is sent via websocket from the browser to the server once per second. On the server, I append the received data to my custom buffer and decode it with the ffmpeg API.

  If I just save the data on the disk and later I play it with a media player, it works fine. So, whatever the browser sends is a valid video.

  I do not think that I correctly understand how should the custom IO behave with network streaming as nothing seems to be working.

  The custom buffer :

   struct Buffer
  
      {
  
          std::vector data;
  
          int currentPos = 0;
  
      };

  The readAVBuffer method for custom IO

  int MediaDecoder::readAVBuffer(void* opaque, uint8_t* buf, int buf_size)
  
  {
  
      MediaDecoder::Buffer* mbuf = (MediaDecoder::Buffer*)opaque;
  
      int count = 0;
  
      for(int i=0;icurrentPos;
  
          if(index >= (int)mbuf->data.size())
  
          {
  
              break;
  
          }
  
          count++;
  
          buf[i] = mbuf->data.at(index);
  
      }
  
      if(count > 0) mbuf->currentPos+=count;
  
  
  
      std::cout << "read : "<currentPos<<", buff size:"<data.size() << std::endl;
  
      if(count <= 0) return AVERROR(EAGAIN); //is this error that should be returned? It cannot be EOF since we're not done yet, most likely
  
      return count;
  
  }

  The big decode method, that’s supposed to return whatever frames it could read

  std::vector MediaDecoder::decode(const char* data, size_t length)
  
  {
  
      std::vector frames;
  
      //add data to the buffer
  
      for(size_t i=0;i/do not invoke the decoders until we have 1MB of data
  
      if(((buf.data.size() - buf.currentPos) < 1*1024*1024) && !initializedCodecs) return frames;
  
  
  
      std::cout << "decoding data length "</initialize ffmpeg objects. Custom I/O, format, decoder, etc.
  
      {       
  
          //these are just members of the class 
  
          avioCtxPtr = std::unique_ptr(
  
                      avio_alloc_context((uint8_t*)av_malloc(4096),4096,0,&buf,&readAVBuffer,nullptr,nullptr),
  
                      avio_context_deleter());
  
          if(!avioCtxPtr)
  
          {
  
              std::cerr << "Could not create IO buffer" << std::endl;
  
              return frames;
  
          }                
  
  
  
          fmt_ctx = std::unique_ptr(avformat_alloc_context(),
  
                                                                            avformat_context_deleter());
  
          fmt_ctx->pb = avioCtxPtr.get();
  
          fmt_ctx->flags |= AVFMT_FLAG_CUSTOM_IO ;
  
          //fmt_ctx->max_analyze_duration = 2 * AV_TIME_BASE; // read 2 seconds of data
  
          {
  
              AVFormatContext *fmtCtxRaw = fmt_ctx.get();            
  
              if (avformat_open_input(&fmtCtxRaw, "", nullptr, nullptr) < 0) {
  
                  std::cerr << "Could not open movie" << std::endl;
  
                  return frames;
  
              }
  
          }
  
          if (avformat_find_stream_info(fmt_ctx.get(), nullptr) < 0) {
  
              std::cerr << "Could not find stream information" << std::endl;
  
              return frames;
  
          }
  
          if((video_stream_idx = av_find_best_stream(fmt_ctx.get(), AVMEDIA_TYPE_VIDEO, -1, -1, nullptr, 0)) < 0)
  
          {
  
              std::cerr << "Could not find video stream" << std::endl;
  
              return frames;
  
          }
  
          AVStream *video_stream = fmt_ctx->streams[video_stream_idx];
  
          AVCodec *dec = avcodec_find_decoder(video_stream->codecpar->codec_id);
  
  
  
          video_dec_ctx = std::unique_ptr (avcodec_alloc_context3(dec),
  
                                                                                avcodec_context_deleter());
  
          if (!video_dec_ctx)
  
          {
  
              std::cerr << "Failed to allocate the video codec context" << std::endl;
  
              return frames;
  
          }
  
          avcodec_parameters_to_context(video_dec_ctx.get(),video_stream->codecpar);
  
          video_dec_ctx->thread_count = 1;
  
         /* video_dec_ctx->max_b_frames = 0;
  
          video_dec_ctx->frame_skip_threshold = 10;*/
  
  
  
          AVDictionary *opts = nullptr;
  
          av_dict_set(&opts, "refcounted_frames", "1", 0);
  
          av_dict_set(&opts, "deadline", "1", 0);
  
          av_dict_set(&opts, "auto-alt-ref", "0", 0);
  
          av_dict_set(&opts, "lag-in-frames", "1", 0);
  
          av_dict_set(&opts, "rc_lookahead", "1", 0);
  
          av_dict_set(&opts, "drop_frame", "1", 0);
  
          av_dict_set(&opts, "error-resilient", "1", 0);
  
  
  
          int width = video_dec_ctx->width;
  
          videoHeight = video_dec_ctx->height;
  
  
  
          if(avcodec_open2(video_dec_ctx.get(), dec, &opts) < 0)
  
          {
  
              std::cerr << "Failed to open the video codec context" << std::endl;
  
              return frames;
  
          }
  
  
  
          AVPixelFormat  pFormat = AV_PIX_FMT_BGR24;
  
          img_convert_ctx = std::unique_ptr(sws_getContext(width, videoHeight,
  
                                           video_dec_ctx->pix_fmt,   width, videoHeight, pFormat,
  
                                           SWS_BICUBIC, nullptr, nullptr,nullptr),swscontext_deleter());
  
  
  
          frame = std::unique_ptr(av_frame_alloc(),avframe_deleter());
  
          frameRGB = std::unique_ptr(av_frame_alloc(),avframe_deleter());
  
  
  
  
  
          int numBytes = av_image_get_buffer_size(pFormat, width, videoHeight,32 /*https://stackoverflow.com/questions/35678041/what-is-linesize-alignment-meaning*/);
  
          std::unique_ptr imageBuffer((uint8_t *) av_malloc(numBytes*sizeof(uint8_t)),avbuffer_deleter());
  
          av_image_fill_arrays(frameRGB->data,frameRGB->linesize,imageBuffer.get(),pFormat,width,videoHeight,32);
  
          frameRGB->width = width;
  
          frameRGB->height = videoHeight;
  
  
  
          initializedCodecs = true;
  
      }    
  
      AVPacket pkt;
  
      av_init_packet(&pkt);
  
      pkt.data = nullptr;
  
      pkt.size = 0;
  
  
  
      int read_frame_return = 0;
  
      while ( (read_frame_return=av_read_frame(fmt_ctx.get(), &pkt)) >= 0)
  
      {
  
          readFrame(&frames,&pkt,video_dec_ctx.get(),frame.get(),img_convert_ctx.get(),
  
                    videoHeight,frameRGB.get());
  
          //if(cancelled) break;
  
      }
  
      avioCtxPtr->eof_reached = 0;
  
      avioCtxPtr->error = 0;
  
  
  
  
  
      //flush
  
     // readFrame(frames.get(),nullptr,video_dec_ctx.get(),frame.get(),
  
       //         img_convert_ctx.get(),videoHeight,frameRGB.get());
  
  
  
      avioCtxPtr->eof_reached = 0;
  
      avioCtxPtr->error = 0;
  
  
  
      if(frames->size() <= 0)
  
      {
  
          std::cout << "buffer pos: "<code>

  What I would expect to happen would be for a continuous extraction of cv::Mat frames as I feed it more and more data. What actually happens is that after the the buffer is fully read I see :

  [matroska,webm @ 0x507b450] Read error at pos. 1278266 (0x13813a)
  
  [matroska,webm @ 0x507b450] Seek to desired resync point failed. Seeking to earliest point available instead.

  And then no more bytes are read from the buffer even if later I increase the size of it.

  There is something terribly wrong I’m doing here and I don’t understand what.

• Python cv2 script that scans a giant image to a video. Why do I need pad two extra lines

  27 avril 2022, par Mahrarena

  I wrote a script that scans a giant image to make a video. Normally I just post my scripts straight to my Code Review account, but this script is ugly, needs to be refactored, implements only horizontal scrolling and most importantly I just fixed a bug but I don't completely understand why it works.

  


  Example :

  


  Original image (Google Drive)

  


  Video Output (Google Drive)

  


  As you can see from the video, everything is working properly except the fact that I don't know how it works.

  


  Full working code

  


  ---

  


  import cv2
import numpy as np
import random
import rpack
from fractions import Fraction
from math import prod

def resize_guide(image_size, target_area):
    aspect_ratio = Fraction(*image_size).limit_denominator()
    horizontal = aspect_ratio.numerator
    vertical = aspect_ratio.denominator
    unit_length = (target_area/(horizontal*vertical))**.5
    return (int(horizontal*unit_length), int(vertical*unit_length))

fourcc = cv2.VideoWriter_fourcc(*'mp4v')
FRAME = np.zeros((1080, 1920, 3), dtype=np.uint8)

def new_frame():
    return np.ndarray.copy(FRAME)

def center(image):
    frame = new_frame()
    h, w = image.shape[:2]
    yoff = round((1080-h)/2)
    xoff = round((1920-w)/2)
    frame[yoff:yoff+h, xoff:xoff+w] = image
    return frame

def image_scanning(file, fps=60, pan_increment=64, horizontal_increment=8):
    image = cv2.imread(file)
    height, width = image.shape[:2]
    assert width*height >= 1920*1080
    video_writer = cv2.VideoWriter(file+'.mp4', fourcc, fps, (1920, 1080))
    fit_height = True
    if height < 1080:
        width = width*1080/height
        image = cv2.resize(image, (width, 1080), interpolation = cv2.INTER_AREA)
    aspect_ratio = width / height
    zooming_needed = False
    if 4/9 <= aspect_ratio <= 16/9:
        new_width = round(width*1080/height)
        fit = cv2.resize(image, (new_width, 1080), interpolation = cv2.INTER_AREA)
        zooming_needed = True
    
    elif 16/9 < aspect_ratio <= 32/9:
        new_height = round(height*1920/width)
        fit = cv2.resize(image, (1920, new_height), interpolation = cv2.INTER_AREA)
        fit_height = False
        zooming_needed = True
    
    centered = center(fit)
    for i in range(fps):
        video_writer.write(centered)
    if fit_height:
        xoff = round((1920 - new_width)/2)
        while xoff:
            if xoff - pan_increment >= 0:
                xoff -= pan_increment
            else:
                xoff = 0
            frame = new_frame()
            frame[0:1080, xoff:xoff+new_width] = fit
            video_writer.write(frame)
    else:
        yoff = round((1080 - new_height)/2)
        while yoff:
            if yoff - pan_increment >= 0:
                yoff -= pan_increment
            else:
                yoff = 0
            frame = new_frame()
            frame[yoff:yoff+new_height, 0:1920] = fit
            video_writer.write(frame)
    
    if zooming_needed:
        if fit_height:
            width_1, height_1 = new_width, 1080
        else:
            width_1, height_1 = 1920, new_height
        new_area = width_1 * height_1
        original_area = width * height
        area_diff = original_area - new_area
        unit_diff = area_diff / fps
        for i in range(1, fps+1):
            zoomed = cv2.resize(image, resize_guide((width_1, height_1), new_area+unit_diff*i), interpolation=cv2.INTER_AREA)
            zheight, zwidth = zoomed.shape[:2]
            zheight = min(zheight, 1080)
            zwidth = min(zwidth, 1920)
            frame = new_frame()
            frame[0:zheight, 0:zwidth] = zoomed[0:zheight, 0:zwidth]
            video_writer.write(frame)
    
    if (width - 1920) % horizontal_increment:
        new_width = ((width - 1920) // horizontal_increment + 1) * horizontal_increment + 1920
        frame = np.zeros([height, new_width, 3], dtype=np.uint8)
        frame[0:height, 0:width] = image
        width = new_width
        image = frame
    
    if height % 1080:
        new_height = (height // 1080 + 2) * 1080
        frame = np.zeros([new_height, width, 3], dtype=np.uint8)
        frame[0:height, 0:width] = image
        height = new_height - 1080
        image = frame
    
    y, x = 0, 0
    for y in range(0, height, 1080):
        for x in range(0, width-1920, horizontal_increment):
            frame = image[y:y+1080, x:x+1920]
            video_writer.write(frame)
        x = width - 1920
        frame = image[y:y+1080, x:x+1920]
        for i in range(round(fps/3)):
            video_writer.write(frame)
    cv2.destroyAllWindows()
    video_writer.release()
    del video_writer


  


  I don't know why I need to pad two extra lines instead of one, meaning if I change this :

  


      if height % 1080:
        new_height = (height // 1080 + 2) * 1080
        frame = np.zeros([new_height, width, 3], dtype=np.uint8)
        frame[0:height, 0:width] = image
        height = new_height - 1080
        image = frame


  


  To this :

  


      if height % 1080:
        new_height = (height // 1080 + 1) * 1080
        frame = np.zeros([new_height, width, 3], dtype=np.uint8)
        frame[0:height, 0:width] = image
        height = new_height
        image = frame


  


  The program raises exceptions :

  


  OpenCV: FFMPEG: tag 0x34363268/&#x27;h264&#x27; is not supported with codec id 27 and format &#x27;mp4 / MP4 (MPEG-4 Part 14)&#x27;&#xA;OpenCV: FFMPEG: fallback to use tag 0x31637661/&#x27;avc1&#x27;&#xA;---------------------------------------------------------------------------&#xA;error                                     Traceback (most recent call last)&#xA; in <module>&#xA;----> 1 image_scanning("D:/collages/91f53ebcea2a.png")&#xA;&#xA; in image_scanning(file, fps, pan_increment, horizontal_increment, fast_decrement)&#xA;    122                     x &#x2B;= horizontal_increment&#xA;    123                     frame = image[y:y&#x2B;1080, x:x&#x2B;1920]&#xA;--> 124                     video_writer.write(frame)&#xA;    125     cv2.destroyAllWindows()&#xA;    126     video_writer.release()&#xA;&#xA;error: Unknown C&#x2B;&#x2B; exception from OpenCV code&#xA;</module>

  


  I guess it was caused by indexing error because the last line would not have enough pixels so padding the height of the image to a multiple of 1080 should work.

  


  But that's not the case, I need to pad two lines, why is that ? I really don't understand why it is working.

  


  ---

  


  No, I really wrote all of it, I understand all the principles, the ideas are all mine, but there is one small problem in implementation. I don't know why I need extra pixels in the bottom to make it work, because if I don't pad the height to a multiple of 1080, I can't get the bottom line, the lowest potion of height % 1080 would be lost.

  


  If I tried to get the lowest part, the program will raise exceptions even if I pad the height to a multiple of 1080, I think it is related to indexing but I don't fully understand it, turns out I need to pad the height and add extra pixels, even 1 pixel would work.

  


  I don't know why it raises exceptions and how add extra pixels got rid of the exception, but I understand everything else perfectly clear, after all I wrote it.

  


  There's a bug in my program, I don't know what caused it, and I want you to help me debugging, and that's the entire point of the question !