Recherche avancée

Sur d’autres sites (4349)

• How to write HEVC frames to file using FFMpeg ?

  2 mai 2018, par boneill

  I have the following code used to output H264 frames to an mp4 file and this works fine :

  typedef struct _AvFileStreamContext
  
  {
  
      AVStream                 *pStreams[AV_FRAME_MAX];
  
      AVCodecContext           *streamCodec;
  
      AVFormatContext          *pAvContext;    // initialised with avformat_alloc_output_context2 for mp4 container
  
  
  
  }_AvFileStreamContext;
  
  
  
  static void WriteImageFrameToFile(const unsigned char * frame,
  
                                    const int frameSize,
  
                                    const struct timeval *frameTime,
  
                                    AvFileStreamContext *pContext,
  
                                    int keyFrame)
  
  {
  
      AVStream *stream pContext->pStreams[AV_FRAME_VIDEO];
  
      AVPacket pkt;
  
  
  
      av_init_packet(&pkt);
  
      if (keyFrame)
  
      {
  
          pkt.flags |= AV_PKT_FLAG_KEY;
  
      }
  
      pkt.stream_index = stream->index;
  
      pkt.data = (unsigned char*)frame;
  
      pkt.size = frameSize;
  
  
  
      int ptsValue = round((float)( ( (frameTime->tv_sec - pContext->firstFrameTime.tv_sec ) * 1000000 +
  
                                      (frameTime->tv_usec - pContext->firstFrameTime.tv_usec)) * pContext->streamCodec->time_base.den ) / 1000000);
  
  
  
      // Packets PTS/DTS must be in Stream time-base units before writing so
  
      // rescaling between coder and stream time bases is required.
  
      pkt.pts =  av_rescale_q(ptsValue, pContext->streamCodec->time_base, stream->time_base);
  
      pkt.dts =  av_rescale_q(ptsValue, pContext->streamCodec->time_base, stream->time_base);
  
  
  
      av_interleaved_write_frame(pContext->pAvContext, &pkt);
  
  }

  Once all frames have been received I call the following :

  av_write_trailer(pContext->pAvContext);
  
  avio_close(pContext->pAvContext->pb);

  The above function is supplied from a circular buffer of frames where each entry in the buffer represents a frame for H264. I am trying to understand how I can adapt this to handle H265/HEVC frames. When I blindly try and use this for H265 frames I end up with an mp4 file where each frame only contains a third of a complete frame i.e.

  
  The video continues to play for the correct duration but each frame is only a third of a complete frame. The implementation for receiving frames is the same as H264 and it is my understanding that with H265 each of the buffers I am receiving represents a ’tile’. In my case these are tile columns of which 3 tiles make up one frame. That said, it would seem that the above function would need to be adapted to combine 3 tiles until an end of frame marker is received. I have trawled the FFMpeg v3.3 documentation to find out how I can achieve this but have had limited luck.
  I have tried to use the following function call to combine frames :

  uint8_t * outputBuffer;
  
  unsigned int bufferSize;
  
  stream->parser = av_parser_init (AV_CODEC_ID_HEVC);
  
  int rc = av_parser_parse2(stream->parser, pContext->pAvContext,
  
                            &outputBuffer, &bufferSize,
  
                            frame, frameSize,
  
                            pkt.pts, pkt.dts, -1);

  It seems that the above call will ultimately call ;

  static int hevc_parse(AVCodecParserContext *s,
  
                        AVCodecContext *avctx,
  
                        const uint8_t **poutbuf, int *poutbuf_size,
  
                        const uint8_t *buf, int buf_size)

  Followed by :

  int ff_combine_frame(ParseContext *pc, int next,
  
                       const uint8_t **buf, int *buf_size)

  So it seems this is the correct path, however when I plug this all in the resulting mp4 file is not playable under gstreamer with the following errors :

  Prerolling...
  
  (gst-play-1.0:11225): GStreamer-WARNING **: gstpad.c:4943:store_sticky_event: Sticky event misordering, got 'segment' before 'caps'
  
  Redistribute latency...
  
  
  
  (gst-play-1.0:11225): GStreamer-WARNING **: gstpad.c:4943:store_sticky_event: Sticky event misordering, got 'segment' before 'caps'

  And I get the following errors (snippet of errors) from VLC where the frames appear to be correct in terms of height and width but are corrupt or incomplete during playback :

  [hevc @ 0x7f8128c30440] PPS changed between slices.
  
  [hevc @ 0x7f8128c42080] PPS changed between slices.
  
  [hevc @ 0x7f8128c53ce0] PPS changed between slices.
  
  [00007f8110293be8] freetype spu text error: Breaking unbreakable line
  
  [hevc @ 0x7f8128c1e0a0] First slice in a frame missing.
  
  [hevc @ 0x7f8128c1e0a0] First slice in a frame missing.
  
  [hevc @ 0x7f8128c1e0a0] Could not find ref with POC 7
  
  [hevc @ 0x7f8128c30440] PPS changed between slices.
  
  [hevc @ 0x7f8128c42080] PPS changed between slices.
  
  [hevc @ 0x7f8128c53ce0] PPS changed between slices.
  
  [hevc @ 0x7f8128c1e0a0] First slice in a frame missing.
  
  [hevc @ 0x7f8128c1e0a0] First slice in a frame missing.
  
  [hevc @ 0x7f8128c1e0a0] Could not find ref with POC 15
  
  [hevc @ 0x7f8128c30440] PPS changed between slices.
  
  [hevc @ 0x7f8128c42080] PPS changed between slices.
  
  [hevc @ 0x7f8128c53ce0] PPS changed between slices.
  
  [hevc @ 0x7f8128c1e0a0] First slice in a frame missing.
  
  [hevc @ 0x7f8128c1e0a0] First slice in a frame missing.
  
  [hevc @ 0x7f8128c1e0a0] Could not find ref with POC 23

  Here is an example frame from the VLC playback where you can just about see the outline of someone :
  

  It should be noted that when using the call to av_parser_parse2(), a frame is only passed to av_interleaved_write_frame() when outputBuffer is populated which seems to take a number of tiles (greater than 3) so I am possibly not setting something correctly.

  • Can I tell FFMpeg that a particular tile is an end of frame ?
  • Should I be using some other FFMpeg call to combine H265 tiles ?
  • Am I misunderstanding how HEVC operates ? (probably)
  • Once combined it the call to av_interleaved_write_frame() still valid ?
  • Note that use of libx265 is not possible.

  Any help appreciated.

• Sequencing MIDI From A Chiptune

  28 avril 2013, par Multimedia Mike — Outlandish Brainstorms

  The feature requests for my game music appreciation website project continue to pour in. Many of them take the form of “please add player support for system XYZ and the chiptune library to go with it.” Most of these requests are A) plausible, and B) in process. I have also received recommendations for UI improvements which I take under consideration. Then there are the numerous requests to port everything from Native Client to JavaScript so that it will work everywhere, even on mobile, a notion which might take a separate post to debunk entirely.

  But here’s an interesting request about which I would like to speculate : Automatically convert a chiptune into a MIDI file. I immediately wanted to dismiss it as impossible or highly implausible. But, as is my habit, I started pondering the concept a little more carefully and decided that there’s an outside chance of getting some part of the idea to work.

  Intro to MIDI
  MIDI stands for Musical Instrument Digital Interface. It’s a standard musical interchange format and allows music instruments and computers to exchange musical information. The file interchange format bears the extension .mid and contains a sequence of numbers that translate into commands separated by time deltas. E.g. : turn key on (this note, this velocity) ; wait x ticks ; turn key off ; wait y ticks ; etc. I’m vastly oversimplifying, as usual.

  MIDI fascinated me back in the days of dialup internet and discrete sound cards (see also my write-up on the Gravis Ultrasound). Typical song-length MIDI files often ranged from a few kilobytes to a few 10s of kilobytes. They were significantly smaller than the MOD et al. family of tracker music formats mostly by virtue of the fact that MIDI files aren’t burdened by transporting digital audio samples.
  
  I know I’m missing a lot of details. I haven’t dealt much with MIDI in the past… 15 years or so (ever since computer audio became a blur of MP3 and AAC audio). But I’m led to believe it’s still relevant. The individual who requested this feature expressed an interest in being able to import the sequenced data into any of the many music programs that can interpret .mid files.

  The Pitch
  To limit the scope, let’s focus on music that comes from the 8-bit Nintendo Entertainment System or the original Game Boy. The former features 2 square wave channels, a triangle wave, a noise channel, and a limited digital channel. The latter creates music via 2 square waves, a wave channel, and a noise channel. The roles that these various channels usually play typically break down as : square waves represent the primary melody, triangle wave is used to simulate a bass line, noise channel approximates a variety of percussive sounds, and the DPCM/wave channels are fairly free-form. They can have random game sound effects or, if they are to assist in the music, are often used for more authentic percussive sounds.

  The various channels are controlled via an assortment of memory-mapped hardware registers. These registers are fed values such as frequency, volume, and duty cycle. My idea is to modify the music playback engine to track when various events occur. Whenever a channel is turned on or off, that corresponds to a MIDI key on or off event. If a channel is already playing but a new frequency is written, that would likely count as a note change, so log a key off event followed by a new key on event.

  There is the major obstacle of what specific note is represented by a channel in a particular state. The MIDI standard defines 128 different notes spanning 11 octaves. Empirically, I wonder if I could create a table which maps the assorted frequencies to different MIDI notes ?

  I think this strategy would only work with the square and triangle waves. Noise and digital channels ? I’m not prepared to tackle that challenge.

  Prior Work ?
  I have to wonder if there is any existing work in this area. I’m certain that people have wanted to do this before ; I wonder if anyone has succeeded ?

  Just like reverse engineering a binary program entails trying to obtain a higher level abstraction of a program from a very low level representation, this challenge feels like reverse engineering a piece of music as it is being performed and automatically expressing it in a higher level form.

• What is the best way to check if audio embedded in video is ambisonic ?

  20 mai 2019, par GrzegorzW

  We are making 360 video player for VR headsets using Unity. We are trying to achieve spatial audio. The easiest case is when video and audio are in separated files, but now we decided to support also videos with embedded audio. I need to know before i load video if audio is ambisonic or not. I’m looking for an easy way to decide if audio is ambisonic in order to separate it from video and convert to .tbe file, which is currently supported by our app.

  I was trying to use ffmpeg :

  $./ffmpeg.exe -i ~/Videos/video.mp4

  and got :

  ffmpeg version 4.1.3 Copyright (c) 2000-2019 the FFmpeg developers
  
    built with gcc 8.3.1 (GCC) 20190414
  
    configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth
  
    libavutil      56. 22.100 / 56. 22.100
  
    libavcodec     58. 35.100 / 58. 35.100
  
    libavformat    58. 20.100 / 58. 20.100
  
    libavdevice    58.  5.100 / 58.  5.100
  
    libavfilter     7. 40.101 /  7. 40.101
  
    libswscale      5.  3.100 /  5.  3.100
  
    libswresample   3.  3.100 /  3.  3.100
  
    libpostproc    55.  3.100 / 55.  3.100
  
  Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'C:/Users/Medion/Videos/video.mp4':
  
    Metadata:
  
      major_brand     : isom
  
      minor_version   : 512
  
      compatible_brands: isomiso2avc1mp41
  
      encoder         : Lavf57.56.101
  
    Duration: 00:11:39.40, start: 0.000000, bitrate: 17290 kb/s
  
      Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 3840x2160 [SAR 1:1 DAR 16:9], 16497 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
  
      Metadata:
  
        handler_name    : VideoHandler
  
      Side data:
  
        stereo3d: top and bottom
  
        spherical: equirectangular (0.000000/0.000000/0.000000)
  
      Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 4.0, fltp, 778 kb/s (default)
  
      Metadata:
  
        handler_name    : SoundHandler
  
      Stream #0:2(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 4.0, fltp, 4 kb/s
  
      Metadata:
  
        handler_name    : SoundHandler
  
      Stream #0:3(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 2 kb/s
  
      Metadata:
  
        handler_name    : SoundHandler
  
  At least one output file must be specified

  but I don’t see any line that could indicate that audio is ambisonic. I see that there are 4 channels in streams #0:1 and #0:2, but I bet it’s not enough.

  I’ve also tried MediaInfo but it’s the same thing :

  General
  
  CompleteName                     : C:\Users\Medion\Videos\video.mp4
  
  Format/String                    : MPEG-4
  
  Format_Profile                   : Base Media
  
  CodecID/String                   : isom (isom/iso2/avc1/mp41)
  
  FileSize/String                  : 1.41 GiB
  
  Duration/String                  : 11 min 39 s
  
  OverallBitRate_Mode/String       : Variable
  
  OverallBitRate/String            : 17.3 Mb/s
  
  Encoded_Application/String       : Lavf57.56.101
  
  
  
  Video
  
  ID/String                        : 1
  
  Format/String                    : AVC
  
  Format/Info                      : Advanced Video Codec
  
  Format_Profile                   : High@L5.2
  
  Format_Settings                  : CABAC / 3 Ref Frames
  
  Format_Settings_CABAC/String     : Yes
  
  Format_Settings_RefFrames/String : 3 frames
  
  Format_Settings_GOP              : M=3, N=29
  
  CodecID                          : avc1
  
  CodecID/Info                     : Advanced Video Coding
  
  Duration/String                  : 11 min 39 s
  
  BitRate_Mode/String              : Variable
  
  BitRate/String                   : 16.5 Mb/s
  
  BitRate_Maximum/String           : 20.0 Mb/s
  
  Width/String                     : 3 840 pixels
  
  Height/String                    : 2 160 pixels
  
  DisplayAspectRatio/String        : 16:9
  
  FrameRate_Mode/String            : Constant
  
  FrameRate/String                 : 29.970 (30000/1001) FPS
  
  Standard                         : NTSC
  
  ColorSpace                       : YUV
  
  ChromaSubsampling/String         : 4:2:0
  
  BitDepth/String                  : 8 bits
  
  ScanType/String                  : Progressive
  
  Bits-(Pixel*Frame)               : 0.066
  
  StreamSize/String                : 1.34 GiB (95%)
  
  Language/String                  : English
  
  Tagged_Date                      : UTC 2017-06-13 17:37:51
  
  colour_range                     : Limited
  
  colour_primaries                 : BT.709
  
  transfer_characteristics         : BT.709
  
  matrix_coefficients              : BT.709
  
  Codec configuration box          : avcC
  
  
  
  Audio #1
  
  ID/String                        : 2
  
  Format/String                    : AAC LC
  
  Format/Info                      : Advanced Audio Codec Low Complexity
  
  CodecID                          : mp4a-40-2
  
  Duration/String                  : 11 min 39 s
  
  Source_Duration/String           : 11 min 39 s
  
  BitRate_Mode/String              : Constant
  
  BitRate/String                   : 779 kb/s
  
  Channel(s)/String                : 4 channels
  
  ChannelLayout                    : C L R Cb
  
  SamplingRate/String              : 48.0 kHz
  
  FrameRate/String                 : 46.875 FPS (1024 SPF)
  
  Compression_Mode/String          : Lossy
  
  StreamSize/String                : 64.9 MiB (5%)
  
  Source_StreamSize/String         : 64.9 MiB (5%)
  
  Default/String                   : Yes
  
  AlternateGroup/String            : 1
  
  Tagged_Date                      : UTC 2017-06-13 17:37:51
  
  
  
  Audio #2
  
  ID/String                        : 3
  
  Format/String                    : AAC LC
  
  Format/Info                      : Advanced Audio Codec Low Complexity
  
  CodecID                          : mp4a-40-2
  
  Duration/String                  : 11 min 39 s
  
  Source_Duration/String           : 11 min 39 s
  
  BitRate_Mode/String              : Variable
  
  BitRate/String                   : 4 900 b/s
  
  BitRate_Maximum/String           : 266 kb/s
  
  Channel(s)/String                : 4 channels
  
  ChannelLayout                    : C L R Cb
  
  SamplingRate/String              : 48.0 kHz
  
  FrameRate/String                 : 46.875 FPS (1024 SPF)
  
  Compression_Mode/String          : Lossy
  
  StreamSize/String                : 418 KiB (0%)
  
  Source_StreamSize/String         : 418 KiB (0%)
  
  Default/String                   : No
  
  AlternateGroup/String            : 1
  
  Tagged_Date                      : UTC 2017-06-13 17:37:51
  
  
  
  Audio #3
  
  ID/String                        : 4
  
  Format/String                    : AAC LC
  
  Format/Info                      : Advanced Audio Codec Low Complexity
  
  CodecID                          : mp4a-40-2
  
  Duration/String                  : 11 min 39 s
  
  Source_Duration/String           : 11 min 39 s
  
  BitRate_Mode/String              : Variable
  
  BitRate/String                   : 2 275 b/s
  
  BitRate_Maximum/String           : 128 kb/s
  
  Channel(s)/String                : 2 channels
  
  ChannelLayout                    : L R
  
  SamplingRate/String              : 48.0 kHz
  
  FrameRate/String                 : 46.875 FPS (1024 SPF)
  
  Compression_Mode/String          : Lossy
  
  StreamSize/String                : 194 KiB (0%)
  
  Source_StreamSize/String         : 194 KiB (0%)
  
  Default/String                   : No
  
  AlternateGroup/String            : 1
  
  Tagged_Date                      : UTC 2017-06-13 17:37:51

  I assume that I don’t know what to look for in those outputs.
  Thanks in advance.