Recherche avancée

Sur d’autres sites (10007)

• Watson NarrowBand Speech to Text not accepting ogg file

  19 janvier 2017, par Bob Dill

  NodeJS app using ffmpeg to create ogg files from mp3 & mp4. If the source file is broadband, Watson Speech to Text accepts the file with no issues. If the source file is narrow band, Watson Speech to Text fails to read the ogg file. I’ve tested the output from ffmpeg and the narrowband ogg file has the same audio content (e.g. I can listen to it and hear the same people) as the mp3 file. Yes, in advance, I am changing the call to Watson to correctly specify the model and content_type. Code follows :

  exports.createTranscript = function(req, res, next)
  
  { var _name = getNameBase(req.body.movie);
  
    var _type = getType(req.body.movie);
  
    var _voice = (_type == "mp4") ? "en-US_BroadbandModel" : "en-US_NarrowbandModel" ;
  
    var _contentType = (_type == "mp4") ? "audio/ogg" : "audio/basic" ;
  
    var _audio = process.cwd()+"/HTML/movies/"+_name+'ogg';
  
    var transcriptFile = process.cwd()+"/HTML/movies/"+_name+'json';
  
  
  
    speech_to_text.createSession({model: _voice}, function(error, session) {
  
      if (error) {console.log('error:', error);}
  
      else
  
        {
  
          var params = { content_type: _contentType, continuous: true,
  
           audio: fs.createReadStream(_audio),
  
            session_id: session.session_id
  
            };
  
            speech_to_text.recognize(params, function(error, transcript) {
  
              if (error) {console.log('error:', error);}
  
              else
  
                { fs.writeFile(transcriptFile, JSON.stringify(transcript), function(err) {if (err) {console.log(err);}});
  
                  res.send(transcript);
  
                }
  
            });
  
        }
  
    });
  
  }

  _type is either mp3 (narrowband from phone recording) or mp4 (broadband)
  model: _voice has been traced to ensure correct setting
  content_type: _contentType has been traced to ensure correct setting

  Any ogg file submitted to Speech to Text with narrowband settings fails with Error: No speech detected for 30s. Tested with both real narrowband files and asking Watson to read a broadband ogg file (created from mp4) as narrowband. Same error message. What am I missing ?

• FFMpeg C Lib - Transpose causes corrupt image

  16 décembre 2016, par Victor.dMdB

  I’m trying to set up a transcoding pipeline with ffmpeg C lib, but if I transpose it, the video is corrupted as shown below.

  If I don’t transpose, the video is fine, ie the rest of the pipeline is correctly set up.

  I need to convert the AVFrame to another datatype to use it with other software. I believe the corruption happens on the copy, but I’m not sure why. Possible something to do with rotating YUV420P pixels ?

  

  The constructor (code was taken from here)

  MyFilter::MyFilter(const std::string filter_desc, AVCodecContext *data_ctx){
  
    avfilter_register_all();
  
    buffersrc_ctx = NULL;
  
    buffersink_ctx = NULL;
  
  
  
      filter_graph = avfilter_graph_alloc();
  
  
  
    AVFilter *buffersink = avfilter_get_by_name("buffersink");
  
    if (!buffersink) {
  
      throw error("filtering sink element not found\n");
  
    }
  
  
  
    if (avfilter_graph_create_filter(&buffersink_ctx, buffersink, "out", NULL, NULL, filter_graph) < 0) {
  
      throw error("Cannot create buffer sink\n");
  
    }
  
  
  
  
  
  filterInputs  = avfilter_inout_alloc();
  
    filterInputs->name       = av_strdup("out");
  
    filterInputs->filter_ctx = buffersink_ctx;
  
    filterInputs->pad_idx    = 0;
  
    filterInputs->next       = NULL;
  
  
  
      AVFilter *buffersrc  = avfilter_get_by_name("buffer");
  
      if (!buffersrc) {
  
          throw error("filtering source element not found\n");
  
      }
  
  
  
      char args[512];
  
      snprintf(args, sizeof(args), "video_size=%dx%d:pix_fmt=%d:time_base=%d/%d:pixel_aspect=%d/%d",
  
                      data_ctx->width, data_ctx->height, data_ctx->pix_fmt,
  
                      data_ctx->time_base.num, data_ctx->time_base.den,
  
                      data_ctx->sample_aspect_ratio.num, data_ctx->sample_aspect_ratio.den);
  
  
  
      log(Info, "Setting filter input with %s", args);
  
  
  
  
  
      if (avfilter_graph_create_filter(&buffersrc_ctx, buffersrc, "in", args, NULL, filter_graph) < 0) {
  
           throw error("Cannot create buffer source\n");
  
      }
  
  
  
      filterOutputs = avfilter_inout_alloc();
  
      filterOutputs->name       = av_strdup("in");
  
      filterOutputs->filter_ctx = buffersrc_ctx;
  
      filterOutputs->pad_idx    = 0;
  
      filterOutputs->next       = NULL;
  
  
  
      if ((avfilter_graph_parse(filter_graph, filter_desc.c_str(), filterInputs, filterOutputs, NULL)) < 0)
  
              log(Warning,"Could not parse input filters");
  
  
  
      if ((avfilter_graph_config(filter_graph, NULL)) < 0)
  
          log(Warning,"Could not configure filter graph");
  
  
  
  }

  And the process

  AVFrame * MyFilter::process(AVFrame *inFrame){
  
  
  
      if (av_buffersrc_add_frame_flags(buffersrc_ctx, inFrame->get(), AV_BUFFERSRC_FLAG_PUSH | AV_BUFFERSRC_FLAG_KEEP_REF ) < 0) {
  
           throw error("Error while feeding the filtergraph\n");
  
       }
  
  
  
      int i = 0;
  
      AVFrame* outFrame =  av_frame_alloc();
  
      if( av_buffersink_get_frame(buffersink_ctx, outFrame) < 0 ){
  
       throw error("Couldnt find a frame\n");
  
      }
  
    return outFrame;
  
  }

  And the filter I’m using is :

   std::string filter_desc = "transpose=cclock"

  As an extra note, it seems like the top bar(visible in the screen capture above) is actually composed of properly rotated pixels, and this works for the whole video. It just degrades for the remaining 99% of pixels.

  Using this works :
  std::string filter_desc = "rotate=PI/2", but then the resolution is not properly shifted. If I try
  std::string filter_desc = "rotate='PI/2:ow=ih:oh=iw'"
  the same issue as before starts appearing again. It seems to be associated with the change in resolution.

  I think the corruption might come from a copy thats made after (for compatibility with something else I’m using) :

  void copyToPicture(AVFrame const* avFrame, DataPicture* pic) {
  
      for (size_t comp=0; compgetNumPlanes(); ++comp) {
  
          auto const subsampling = comp == 0 ? 1 : 2;
  
          auto const bytePerPixel = pic->getFormat().format == YUYV422 ? 2 : 1;
  
          // std::cout<<"Pixel format is "<getFormat().format<data[comp];
  
          auto const srcPitch = avFrame->linesize[comp];
  
  
  
          auto dst = pic->getPlane(comp);
  
          auto const dstPitch = pic->getPitch(comp);
  
  
  
          auto const w = avFrame->width * bytePerPixel / subsampling;
  
          auto const h = avFrame->height / subsampling;
  
  
  
          for (int y=0; ycode>

• Could not read frame error when trying to decompress mp4 file with ffmpeg and Python's threading module

  23 janvier 2017, par mdornfe1

  I’m training constitutional neural networks with video data. So far the bottle neck of my application is decompressing the mp4 files before passing the images to the CNN for training. I had the idea to try to have multiple cpu threads decompress the images concurrently and having one thread pass images to the CNN for training. I made a class VideoStream which makes connection to the mp4 file using the ImageIO module which is built on top of ffmpeg. The structure of my program is a follows :

  1) Generate random ints which represent the frame numbers of the mp4 file that will be used in training. Store these ints in list frame_idxs.

  2) Pass this list of ints and an empty list called frame_queue to the worker function decompress_video_data.

  3) Each worker function makes a connection to the mp4 file using VideoStream.

  4) Each worker function then pops of elements of frame_idxs, decompresses that frame, and then stores that frame as numpy array in list frame_queue.

  Here is the code

  import numpy as np
  
  import os, threading, multiprocessing
  
  
  
  
  
  def decompress_video_data(frame_queue, frame_idxs, full_path):
  
      vs = VideoStream(full_path)
  
      while len(frame_idxs) >1 0:
  
          i = frame_idxs.pop()
  
          frame = vs[i]
  
          frame_queue.append(frame)
  
  
  
  video_folder = '/mnt/data_drive/frame_viewer_client'
  
  video_files = os.listdir(video_folder)
  
  video_file = video_files[0]
  
  full_path = os.path.join(video_folder, video_file)
  
  vs = VideoStream(full_path)
  
  
  
  num_samples = 1000
  
  batch_size = 1
  
  frame_queue = []
  
  decompress_threads = []
  
  frame_idxs = list(np.random.randint(0, len(vs), 
  
      size = batch_size * num_samples))
  
  num_cores = multiprocessing.cpu_count()
  
  
  
  for n in range(num_cores - 1):
  
      decompress_thread = threading.Thread(target=decompress_video_data, 
  
          args=(frame_queue, frame_idxs, full_path))
  
      decompress_threads.append(decompress_thread)
  
      decompress_thread.start()

  The program will sucessfuly decompress approximately 200 frames, and then ImageIO will throw an RuntimeError : Could not read frame. The full error is here. Does anyone know why this is happening ? Is it possible to do what I’m trying to do ? Does ffmpeg just not work with multi threading ?