Recherche avancée

Sur d’autres sites (12974)

• FFMPEG making requests for each frame when decoding a stream, slow performance

  3 juillet 2020, par Byti

  I am having an issue playing MOV camera captured files from an iPhone. My FFMPEG implementation has no problem playing most file formats, this issue is exclusive only for camera captured MOV.

  



  When trying to open the file, I can see in the logs that many requests are made, each requests decoding only one frame, before making a new request which results the video being buffered extremely slowly.
It takes roughly a minute to buffer about a few seconds of the video.

  



  Another thing to mention is that the very same problematic file is played without an issue locally. The problem is when trying to decode while streaming.

  



  I compiled my code on Xcode 11, iOS SDK 13, with cocoapods mobile-ffmpeg-https 4.2.

  



  Here is a rough representation of my code, its pretty standard :

  



  

  1. Here is how I open AVFormatContext :

  


  



  AVFormatContext *context = avformat_alloc_context();
    context->interrupt_callback.callback = FFMPEGFormatContextIOHandler_IO_CALLBACK;
    context->interrupt_callback.opaque = (__bridge void *)(handler);

    av_log_set_level(AV_LOG_TRACE);

    int result = avformat_open_input(&context, [request.urlAsString UTF8String], NULL, NULL);

    if (result != 0) {
        if (context != NULL) {
            avformat_free_context(context);
        }

        return nil;
    }

    result = avformat_find_stream_info(context, NULL);

    if (result < 0) {
        avformat_close_input(&context);
        return nil;
    }


  



  

  2. Video decoder is opened like so, audio decoder is nearly identical

  


  



  AVCodecParameters *params = context->streams[streamIndex]->codecpar;
    AVCodec *codec = avcodec_find_decoder(params->codec_id);

    if (codec == NULL) {
        return NULL;
    }

    AVCodecContext *codecContext = avcodec_alloc_context3(codec);

    if (codecContext == NULL) {
        return NULL;
    }

    codecContext->thread_count = 6;

    int result = avcodec_parameters_to_context(codecContext, params);

    if (result < 0) {
        avcodec_free_context(&codecContext);
        return NULL;
    }

    result = avcodec_open2(codecContext, codec, NULL);

    if (result < 0) {
        avcodec_free_context(&codecContext);
        return NULL;
    }


  



  

  3. I read the data from the server like so :

  


  



  AVPacket packet;

int result = av_read_frame(formatContext, &avPacket);

if (result == 0) {
   avcodec_send_packet(codecContext, &avPacket);

   // .... decode ....
}


  



  Logs after opening the decoders :

  



  // [tls] Request is made here
// [tls] Request response headers are here
Probing mov,mp4,m4a,3gp,3g2,mj2 score:100 size:2048
Probing mp3 score:1 size:2048
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x115918e00] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x115918e00] type:'ftyp' parent:'root' sz: 20 8 23077123
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x115918e00] ISO: File Type Major Brand: qt  
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x115918e00] type:'wide' parent:'root' sz: 8 28 23077123
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x115918e00] type:'mdat' parent:'root' sz: 23066642 36 23077123
// [tls] Request is made here
// [tls] Request response headers are here
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x115918e00] stream 0, sample 4, dts 133333
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x115918e00] stream 1, sample 48, dts 1114558
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x115918e00] stream 2, sample 1, dts 2666667
[h264 @ 0x116080200] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 1
// [tls] Request is made here
// [tls] Request response headers are here
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x115918e00] stream 0, sample 4, dts 133333
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x115918e00] stream 1, sample 48, dts 1114558
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x115918e00] stream 2, sample 1, dts 2666667
[h264 @ 0x116080200] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 1
// [tls] Request is made here
// [tls] Request response headers are here
// ...


  



  These are some warnings I found in the log

  



  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x11c030800] interrupted
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x11c030800] stream 0: start_time: 0.000 duration: 11.833
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x11c030800] stream 1: start_time: 0.000 duration: 11.832
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x11c030800] stream 2: start_time: 0.000 duration: 11.833
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x11c030800] stream 3: start_time: 0.000 duration: 11.833
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x11c030800] format: start_time: 0.000 duration: 11.833 bitrate=15601 kb/s
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x11c030800] Could not find codec parameters for stream 0 (Video: h264, 1 reference frame (avc1 / 0x31637661), none(bt709, left), 1920x1080, 1/1200, 15495 kb/s): unspecified pixel format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x11c030800] After avformat_find_stream_info() pos: 23077123 bytes read:16293 seeks:1 frames:0


  



  Also when calling avformat_open_input(...), 2 GET requests are made, before the returning.
Notice the "Probing mp3 score:1", that is not shown for other MOV files or any other files.

  



  I have tried different versions of ffmpeg, I have tried messing around with the delays of the stream, I tried removing my custom interrupt callback, nothing have worked.

  



  Code works fine with any other videos I have tested (mp4, mkv, avi).

  



  Metadata of the test file :

  



  Metadata:
    major_brand     : qt  
    minor_version   : 0
    compatible_brands: qt  
    creation_time   : 2019-04-14T08:17:03.000000Z
    com.apple.quicktime.make: Apple
    com.apple.quicktime.model: iPhone 7
    com.apple.quicktime.software: 12.2
    com.apple.quicktime.creationdate: 2019-04-14T11:17:03+0300
  Duration: 00:00:16.83, bitrate: N/A
    Stream #0:0(und), 0, 1/600: Video: h264, 1 reference frame (avc1 / 0x31637661), none(bt709), 1920x1080 (0x0), 0/1, 15301 kb/s, 30 fps, 30 tbr, 600 tbn (default)
    Metadata:
      creation_time   : 2019-04-14T08:17:03.000000Z
      handler_name    : Core Media Video
      encoder         : H.264
    Stream #0:1(und), 0, 1/44100: Audio: aac (mp4a / 0x6134706D), 44100 Hz, mono, 100 kb/s (default)
    Metadata:
      creation_time   : 2019-04-14T08:17:03.000000Z
      handler_name    : Core Media Audio
    Stream #0:2(und), 0, 1/600: Data: none (mebx / 0x7862656D), 0/1, 0 kb/s (default)
    Metadata:
      creation_time   : 2019-04-14T08:17:03.000000Z
      handler_name    : Core Media Metadata
    Stream #0:3(und), 0, 1/600: Data: none (mebx / 0x7862656D), 0/1, 0 kb/s (default)
    Metadata:
      creation_time   : 2019-04-14T08:17:03.000000Z
      handler_name    : Core Media Metadata


  


• How to make ffmpeg remux an iPhone MOV (HEVC) that works on Apple products ?

  21 janvier 2024, par Mikael Finstad

  I want to remux a HEVC file (without encoding). I have tried most options, even removing audio, but I cannot get an output file that plays smoothly with Apple products.

  



  First I AirDrop any MOV from an iPhone recorded with the camera in HEVC.

  



  Then run :

  



  ffmpeg -i IMG_4605.MOV -c copy out.mov


  



  Then if I send out.mov back to the iPhone and try to play it back, it plays back with glitches while seeking, like this : (original to the right)

  



  


  



  When I try to drag the file into a new iMovie project and try to play it back, it freezes the whole iMovie app and needs a force quit.

  



  Doing the same with the original causes no issues, so obviously there is something wrong with the way ffmpeg remuxes it, or something wrong with Apple's software.

  



  ffmpeg -i IMG_4605.MOV 
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.17)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.2_2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'IMG_4605.MOV':
  Metadata:
    major_brand     : qt  
    minor_version   : 0
    compatible_brands: qt  
    creation_time   : 2019-12-29T10:20:56.000000Z
    com.apple.quicktime.location.ISO6709: +01.3602+103.9897+024.438/
    com.apple.quicktime.make: Apple
    com.apple.quicktime.model: iPhone 11
    com.apple.quicktime.software: 13.3
    com.apple.quicktime.creationdate: 2019-12-29T18:20:56+0800
  Duration: 00:00:09.00, start: 0.000000, bitrate: 24882 kb/s
    Stream #0:0(und): Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv, bt709), 3840x2160, 24659 kb/s, 29.99 fps, 29.97 tbr, 600 tbn, 600 tbc (default)
    Metadata:
      rotate          : 90
      creation_time   : 2019-12-29T10:20:56.000000Z
      handler_name    : Core Media Video
      encoder         : HEVC
    Side data:
      displaymatrix: rotation of -90.00 degrees
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 192 kb/s (default)
    Metadata:
      creation_time   : 2019-12-29T10:20:56.000000Z
      handler_name    : Core Media Audio
    Stream #0:2(und): Data: none (mebx / 0x7862656D), 0 kb/s (default)
    Metadata:
      creation_time   : 2019-12-29T10:20:56.000000Z
      handler_name    : Core Media Metadata
    Stream #0:3(und): Data: none (mebx / 0x7862656D), 0 kb/s (default)
    Metadata:
      creation_time   : 2019-12-29T10:20:56.000000Z
      handler_name    : Core Media Metadata
    Stream #0:4(und): Data: none (mebx / 0x7862656D), 17 kb/s (default)
    Metadata:
      creation_time   : 2019-12-29T10:20:56.000000Z
      handler_name    : Core Media Metadata


  



  ffmpeg -i out.mov
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.17)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.2_2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'out.mov':
  Metadata:
    major_brand     : qt  
    minor_version   : 512
    compatible_brands: qt  
    encoder         : Lavf58.29.100
  Duration: 00:00:09.00, start: 0.000000, bitrate: 24860 kb/s
    Stream #0:0: Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv, bt709), 3840x2160, 24659 kb/s, 29.99 fps, 29.97 tbr, 19200 tbn, 19200 tbc (default)
    Metadata:
      rotate          : 90
      handler_name    : Core Media Video
      encoder         : HEVC
    Side data:
      displaymatrix: rotation of -90.00 degrees
    Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 192 kb/s (default)
    Metadata:
      handler_name    : Core Media Audio


  



  If I open out.mov in QuickTime and do a trim and export it, then it it still "broken", so I'm thinking there's nothing wrong with the actual muxing into MOV, because QuickTime is passing the error on.

  



  I noticed that out.mov has different parameters 19200 tbn, 19200 tbc (vs original 600 tbn, 600 tbc). However, if I force set these to 600 with -video_track_timescale 600 the output file is still broken.

  



  Anyone got some insights ?

  


• Improving Google Cloud Speech-to-Text accuracy

  6 juillet 2020, par lr_optim

  I'm working on a project where I need to perform these steps :

  


  

  1. Record a voice call (.webm -file)

  


  2. Split the webm -file into chunks with ffmpeg and convert the file into wav

  


  3. Transcribe the chunks using SpeechRecognition -library and Google Cloud API

  


  


  I've faced problems with the transcription accuracy and wondering if there is something I could do to improve it. At the time I'm splitting the original file into 30s chunks. I thought there might be one problem, that I might be missing words because of splitting so I've tried also with longer chunks under 60s but didn't notice any improve in accuracy.
Reading trough the speechRecognition docs I decided to set r.energy_threshold = 4000, I also tried to set the energy_treshold dynamically like this :

  


  with sr.AudioFile(name) as source:
    r.dynamic_energy_threshold = True
    r.adjust_for_ambient_noise(source, duration = 1)
    audio = r.record(source)


  


  I've also tested en-US and en-GB to see if there's some difference but there isn't as much as I'd want. The program is supposed to work with english language spoken by nordic people. If someone has experience about choosing a right language model for people speaking with accent, please let me know.

  


  This is the ffmpeg command is use to split the webm file into chunks : command = ['ffmpeg', '-i', filename, '-f', 'segment', '-segment_time', '30', parts_dir + outputname + '%09d.wav']

  


  Is there somethig I could do better ? I'm wondering if the quality is not good enough an Google is having hard time because of that ?

  


  The main problem is I'm getting bad results (lots of wrong words) from Google and wondering if there is something I could do about it.