Sur d’autres sites (9838)

• Problem decoding h264 over RTP TCP stream

  4 mai 2022, par Kamil.S

  I'm trying to receive RTP stream encoding h264 over TCP from my intercom Hikvision DS-KH8350-WTE1. By reverse engineering I was able to replicate how Hikvision original software Hik-Connect on iPhone and iVMS-4200 on MacOS connects and negotaties streaming. Now I'm getting the very same stream as original apps - verified through Wireshark. Now I need to "make sense" of the stream. I know it's RTP because I inspected how iVMS-4200 uses it using /usr/bin/sample on MacOS. Which yields :

  


    ! :               2 CStreamConvert::InputData(void*, int)  (in libOpenNetStream.dylib) + 52  [0x11ff7c7a6]
+     ! :                 2 SYSTRANS_InputData  (in libSystemTransform.dylib) + 153  [0x114f917f2]
+     ! :                   1 CRTPDemux::ProcessH264(unsigned char*, unsigned int, unsigned int, unsigned int)  (in libSystemTransform.dylib) + 302  [0x114fa2c04]
+     ! :                   | 1 CRTPDemux::AddAVCStartCode()  (in libSystemTransform.dylib) + 47  [0x114fa40f1]
+     ! :                   1 CRTPDemux::ProcessH264(unsigned char*, unsigned int, unsigned int, unsigned int)  (in libSystemTransform.dylib) + 476  [0x114fa2cb2]
+     ! :                     1 CRTPDemux::ProcessVideoFrame(unsigned char*, unsigned int, unsigned int)  (in libSystemTransform.dylib) + 1339  [0x114fa29b3]
+     ! :                       1 CMPEG2PSPack::InputData(unsigned char*, unsigned int, FRAME_INFO*)  (in libSystemTransform.dylib) + 228  [0x114f961d6]
+     ! :                         1 CMPEG2PSPack::PackH264Frame(unsigned char*, unsigned int, FRAME_INFO*)  (in libSystemTransform.dylib) + 238  [0x114f972fe]
+     ! :                           1 CMPEG2PSPack::FindAVCStartCode(unsigned char*, unsigned int)  (in libSystemTransform.dylib) + 23  [0x114f97561]`


  


  I can catch that with lldb and see the arriving packet data making sense as the format I'm describing.

  


  The packet signatures look following :

  


  


  0x24 0x02 0x05 0x85 0x80 0x60 0x01 0x57 0x00 0x00 0x00 0x02 0x00 0x00 0x27 0xde
0x0d 0x80 0x60 0x37 0x94 0x71 0xe3 0x97 0x10 0x77 0x20 0x2c 0x51 | 0x7c
0x85 0xb8 0x00 00 00 00 01 65 0xb8 0x0 0x0 0xa 0x35 ...

  


  0x24 0x02 0x05 0x85 0x80 0x60 0x01 0x58 0x00 0x00 0x00 0x02 0x00 0x00 0x27 0xde
0xd 0x80 0x60 0x37 0x95 0x71 0xe3 0x97 0x10 0x77 0x20 0x2c 0x51 | 0x7c 0x05 0x15 0xac ...

  


  0x24 0x02 0x5 0x85 0x80 0x60 0x01 0x59 0x00 0x0 0x00 0x02 0x00 00x0 0x27 0xde
0xd 0x80 0x60 0x37 0x96 0x71 0xe3 0x97 0x10 0x77 0x20 0x2c 0x51 | 0x7c 0x05 0x5d 0x00 ...

  


  


  By the means of reverse engineering the original software I was able to figure out that 0x7c85 indicates a key frame. 0x7c85 bytes in the genuine software processing do get replaced by h264 00 00 00 01 65 Key frame NALU . That's h264 appendix-B format.
The 0x7c05 packets always follow and are the remaining payload of the key frame. No NALU are added during their handling (the 0x7c05 is stripped away and rest of the bytes is copied). None of the bytes preceding 0x7cXX make it to a mp4 recording (that makes sense as it's the RTP protocol , albeit I'm not sure if it's entirely RTP standard or there's something custom from Hikvision).

  


  If you pay close attention in the Header there are 2 separate bytes indicating order which always match, so I'm sure no packet loss is occurring.

  


  I also observed nonkey frames arriving as 0x7c81 and converted to 00 00 00 01 61 NALU but I want to focus solely on the single key frame for now. Mostly because if I record a movie with the original software it will always begin with 00 00 00 01 65 Key frame (that obviously makes sense).

  


  To get a working mp4 I decided to copy paste a mp4 header of a genuine iVMS-4200 recording (in this sense that's every byte preceding 1st frame NALU 00 00 00 01 65 in the mp4 file). I know that the resolution will match the actual camera footage. With the strategy of waiting for a keyframe , replacing 0x7c85 with 00 00 00 01 65 NALU and appending the remaining bytes, or only appending bytes in the 0x7c05 case I do seem to get something that eventually could work.
When I attempt to ffplay the custom crafted mp4 result I do get something (with a little stretch of imagination that's actually the camera fisheye image forming), but clearly there is a problem.

  


  

  


  It seems around 3-4th 0x7c05 packet (as the failing packet differs on every run), when I copy bytes eventually the h264 stream is incorrect. Just by eye-inspecting the bytes I don't see anything unusual.

  


  This is the failing packet around offset 750 decimal, (I know it's around this place because I keep stripping bytes away to see if there's still same amount frame arriving before it breaks).

More over I did dump those bytes from original software using lldb taking out my own python implementation out of equation. And I run into very same problem with the original packets.

  


  The mp4 header I use should work (since it does for original recordings even if I manipulate number of frames and leave just the first keyframe).
Correct me if I'm wrong but the phase of converting this to MPEG2-PS (which iVMS-4200 does and I don't) should be entirely optional and should not be related to my problem.

  


  Update :
I went the path of setting up recording and only then dumping the original iVMS-4200 packets. I edited the recorded movie to only contain keyframe of interest and it works. I found differences but I cannot explain where they are there yet :

Somehow 00 00 01 E0  13 FA 88 00 02 FF FF is inserted in the genuine recording (that's 4th packet), but I have no idea how this byte string was generated and what is its purpose.
When I fixed the first difference the next one is :

The pattern is striking. But what00 00 01 E0  13 FA 88 00 02 FF FF actually is ? And why is it inserted after 18 03 25 10 & 2F F4 80 FC
  
The 00 00 01 E0 signature would suggest those are Packetized Elementary Stream (PES) headers

  


• Libavformat records RTP stream too fast (too high FPS)

  14 avril 2022, par cydonia

  I am trying to record an RTP stream using libavformat, however the FPS of recorded videos are ridiculously high, I get thousands of FPS, if not ten thousands. I tried to set the FPS to 30 making the following modifications :

  


  AVFormatContext *src; // Assume src is set here to demuxer's context
AVStream *source_stream = src->streams[i];
AVStream *dest_stream = avformat_new_stream(dest, NULL); // dest is muxer's context here
const AVCodec *pCodec = avcodec_find_decoder(source_stream->codecpar->codec_id);
AVCodecContext *avctx = avcodec_alloc_context3(pCodec);
avctx->time_base = av_make_q(1, 30000);
avcodec_parameters_to_context(avctx, source_stream->codecpar);
avcodec_parameters_from_context(dest_stream->codecpar, avctx);
dest_stream->sample_aspect_ratio = source_stream->sample_aspect_ratio;
dest_stream->time_base = avctx->time_base;
dest_stream->avg_frame_rate = av_make_q(30, 1);
dest_stream->r_frame_rate = source_stream->r_frame_rate;
dest_stream->disposition = source_stream->disposition;


  


  Then, during the recording phase, I do the following as I read a new packet :

  


  packet->pts = av_rescale_q(packet->pts, src_stream->time_base, dest_stream->time_base);
packet->dts = av_rescale_q(packet->dts, src_stream->time_base, dest_stream->time_base);
packet->duration = dest_stream->time_base.den / dest_stream->time_base.num / dest_stream->avg_frame_rate.num * dest_stream->avg_frame_rate.den;
av_interleaved_write_frame(dest, packet);


  


  The error log I get is the following, in a repeated manner :

  


  [mp4 @ 0x7fc514001d00] Application provided duration: ... / timestamp: ... is out of range for mov/mp4 format
[mp4 @ 0x7fc514001d00] pts has no value


  


  The libav version I am using is the following :

  


  ffmpeg version 4.3.3 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.2.0-8ubuntu3)
  configuration:
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100


  


  P.S : Upgrading ffmpeg version won't work.

  


• Add, remove and tune filters on running ffmpeg

  28 mai 2023, par Turmund

  Preface

  


  I have been fiddling around with ffmpeg and ffplay using command line adding and tuning filters and effects.

  


  It quickly becomes rather tiresome to

  


  

  1. start playback with some audio file

  


  2. stop

  


  3. modify command

  


  4. back to 1.

  


  


  When for example fine-tuning noise reduction or adding effects and equalizer.

  


  I have played around with using zmq to tune filters by executing commands in a different terminal, but this also becomes somewhat cumbersome.

  


  I want some interface where I can add, remove and tune filters during runtime / while listening to the changes taking effect.

  


  FFMPEG

  


  I use filter to mean effect / filter from here on out. For example afftdn, rubberband, ...

  


  ffmpeg is somewhat intimidating. It's powerful but also complex, at least when starting to dig into it. :0

  


  Looking at the library and examples I am looking at API example for audio decoding and filtering, - which at least at first looks promising as a starting platter.

  


  Output

  


  I imagine it would be best to have multiple sinks or some container with multiple audio tracks :

  


  

  1. Raw audio

  


  2. Audio with effects applied

  


  


  Optionally :

  


  

  1. Raw audio

  


  2. Audio with all filters

  


  3. Audio with filter group 1

  


  4. Audio with filter group 2

  


  5. ... etc.

  


  


  ---

  


  Processing

  


  I imagine the routine would have to be something like :

  


  

  1. Read packet from stream/file/url

  


  2. Unpack the sample

  


  3. Copy / duplicate sample for each filter group / or one for filters

  


  4. Apply filter(s) to these “effect sample” (s)

  


  5. Write raw audio, filtered audio 1, filtered audio 2, filtered audio N, ... to out

  


  


  Or for step 3 - 5 (as one would only be listening to one track at a time (But this is perhaps not the best if one decide to jump back / forth in the audio stream) :

  


  

  3. Apply currently active filter(s)

  


  4. Write raw audio, filtered audio to out

  


  


  Simultaneously one would read and check changes to filters by some interface. I.e. input :

  


  afftdn=rf=-20:nr=20


  


  then, if afftdn is not present in filters add it, else set new values.

  


  Idea is to output "raw-audio". I.e. used in a sampling and tuning phase - then produce a line with filter-options one can use with the ffmpeg-tool to process the audio files once satisfied.

  


  Questions section

  


  

  • Does something similar exist ?

  


  


  General :

  


  

  • Does this seem like a way to do it and use the ffmpeg library ?

    

    • Can one add, remove and change filter values during runtime or do one have to re-initialize the entire stream for each added / removed filter etc ?

    


    • Is the “Processing” part sound ?

    


    


  


  • Would using a container that supports multiple audio tracks be the likely best solution ? E.g. mp4.

    

    • Any container preferred over others ?

    


    • Any drawbacks (i.e. jumping back / forth in the stream)

    


    


  


  


  ---

  


  Sub-note

  


  Dream is to have a Arduino interfacing with this program where I use physical rotary switches, incremental rotary encoders, buttons and whistles. Tuning the various options for the filters using physical knobs. But at first I need some working sample where I use FIFO or what ever to test ffmpeg itself.