Recherche avancée

Sur d’autres sites (13341)

• Error audio loading when runing Whisper Open AI model

  9 juin 2024, par John mick

  The problem I'm trying to solve is that I can't run Whisper model for some audio, it says something related to audio decoding.

  


  payload.wav: Invalid data found when processing input.
raise RuntimeError(f"Failed to load audio: {e.stderr.decode()}") from e


  


  I tried using the micro-machines.wav and it works fine but when i used other audio it gives me an error.

  


  import whisper

model = whisper.load_model("base")
text=model.transcribe('micro-machines.wav',fp16=False)
print(text)
text=model.transcribe('payload.wav',fp16=False)
print(text)


  


  Error I'm getting for payload :

  


  d:\...\venv\lib\site-packages\whisper\transcribe.py:79: UserWarning: FP16 is not supported on CPU; using FP32 instead&#xA;  warnings.warn("FP16 is not supported on CPU; using FP32 instead")                                                                                        &#xA;Traceback (most recent call last):&#xA;  File "d:\...\venv\lib\site-packages\whisper\audio.py", line 42, in load_audio&#xA;    ffmpeg.input(file, threads=0)                                                                                    &#xA;  File "d:\...\venv\lib\site-packages\ffmpeg\_run.py", line 325, in run        &#xA;    raise Error(&#x27;ffmpeg&#x27;, out, err)                                                                                  &#xA;ffmpeg._run.Error: ffmpeg error (see stderr output for detail)                                                       &#xA;&#xA;The above exception was the direct cause of the following exception:&#xA;&#xA;Traceback (most recent call last):&#xA;  File "C:\....\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main&#xA;    return _run_code(code, main_globals, None,&#xA;  File "C:\.....\Python\Python39\lib\runpy.py", line 87, in _run_code&#xA;    exec(code, run_globals)&#xA;  File "D:\...\venv\Scripts\whisper.exe\__main__.py", line 7, in <module>&#xA;  File "d:\...\venv\lib\site-packages\whisper\transcribe.py", line 314, in cli&#xA;    result = transcribe(model, audio_path, temperature=temperature, **args)&#xA;  File "d:\...\venv\lib\site-packages\whisper\transcribe.py", line 85, in transcribe&#xA;    mel = log_mel_spectrogram(audio)&#xA;  File "d:\...\venv\lib\site-packages\whisper\audio.py", line 111, in log_mel_spectrogram&#xA;    audio = load_audio(audio)&#xA;  File "d:\...\venv\lib\site-packages\whisper\audio.py", line 47, in load_audio&#xA;    raise RuntimeError(f"Failed to load audio: {e.stderr.decode()}") from e&#xA;RuntimeError: Failed to load audio: ffmpeg version 6.0-essentials_build-www.gyan.dev Copyright (c) 2000-2023 the FFmpeg developers&#xA;  built with gcc 12.2.0 (Rev10, Built by MSYS2 project)&#xA;  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enab&#xA;le-gmp --enable-lzma --enable-zlib --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-sdl2 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxv&#xA;id --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf &#xA;--enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libgme --enable-libopenmpt --enable-libo&#xA;pencore-amrwb --enable-libmp3lame --enable-libtheora --enable-libvo-amrwbenc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enab&#xA;le-librubberband&#xA;  libavutil      58.  2.100 / 58.  2.100&#xA;  libavcodec     60.  3.100 / 60.  3.100&#xA;  libavformat    60.  3.100 / 60.  3.100&#xA;  libavdevice    60.  1.100 / 60.  1.100&#xA;  libavfilter     9.  3.100 /  9.  3.100&#xA;  libswscale      7.  1.100 /  7.  1.100&#xA;  libswresample   4. 10.100 /  4. 10.100&#xA;  libpostproc    57.  1.100 / 57.  1.100&#xA;payload.wav: Invalid data found when processing input&#xA;</module>

  


  I tried searching for solutions and I found one which says It appears that the code failed to load the audio file for some reason and even failed to display that error because e.stderr did not contain a valid UTF-8 string

  


• Decode a proprietary H264 network video stream

  10 mars 2023, par al_mukthar

  I have incoming byte streams probably encoded in H264 from a RTSP camera through a websocket in my spring boot application,

  


  I need to decode the incoming H264 streams to transmit the video to my frontend clients.
I have tried using javaCV/FFMpeg but nothing works.

  


  Any help would be appreciated

  


  this is the part of hex dump received through socket

  


  00000000: 01 00 00 00 04 48 32 36 34 00 00 00 24 38 65 34    .....H264...$8e4
00000010: 32 39 65 37 61 2D 32 66 34 66 2D 34 37 31 61 2D    29e7a-2f4f-471a-
00000020: 39 61 63 30 2D 66 66 62 38 64 64 37 63 37 64 37    9ac0-ffb8dd7c7d7
00000030: 32 00 00 00 D4 7B 22 49 73 49 6E 69 74 22 3A 66    2...T{"IsInit":f
00000040: 61 6C 73 65 2C 22 49 73 41 75 64 69 6F 22 3A 66    alse,"IsAudio":f
00000050: 61 6C 73 65 2C 22 54 6F 74 61 6C 53 65 63 6F 6E    alse,"TotalSecon
00000060: 64 73 22 3A 30 2E 30 36 2C 22 46 72 61 6D 65 54    ds":0.06,"FrameT
00000070: 69 6D 65 22 3A 22 32 30 32 33 2D 30 32 2D 32 33    ime":"2023-02-23
00000080: 54 30 34 3A 32 31 3A 35 33 2E 35 33 31 5A 22 2C    T04:21:53.531Z",
00000090: 22 53 65 71 75 65 6E 63 65 49 64 22 3A 31 2C 22    "SequenceId":1,"
000000a0: 42 61 73 65 44 65 63 6F 64 65 54 69 6D 65 22 3A    BaseDecodeTime":
000000b0: 32 36 35 38 37 2C 22 4D 65 64 69 61 54 69 6D 65    26587,"MediaTime
000000c0: 22 3A 32 36 35 38 37 2C 22 49 73 46 72 61 6D 65    ":26587,"IsFrame
000000d0: 48 69 64 64 65 6E 22 3A 66 61 6C 73 65 2C 22 49    Hidden":false,"I
000000e0: 73 4B 65 79 46 72 61 6D 65 22 3A 66 61 6C 73 65    sKeyFrame":false
000000f0: 2C 22 49 64 22 3A 34 34 35 2C 22 47 65 6E 65 72    ,"Id":445,"Gener
00000100: 61 74 69 6F 6E 22 3A 31 7D 00 00 3F 50 00 00 00    ation":1}..?P...
00000110: 68 6D 6F 6F 66 00 00 00 10 6D 66 68 64 00 00 00    hmoof....mfhd...
00000120: 00 00 00 01 BD 00 00 00 50 74 72 61 66 00 00 00    ....=...Ptraf...
00000130: 10 74 66 68 64 00 02 00 00 00 00 00 01 00 00 00    .tfhd...........
00000140: 14 74 66 64 74 01 00 00 00 00 00 00 00 00 00 67    .tfdt..........g
00000150: DB 00 00 00 24 74 72 75 6E 01 00 0F 01 00 00 00    [...$trun.......
00000160: 01 00 00 00 70 00 00 00 3C 00 00 3E E0 00 01 00    ....p...<..>`...
00000170: 00 00 00 00 00 00 00 3E E8 6D 64 61 74 00 00 3E    .......>hmdat..>
00000180: DC 41 E1 81 80 93 BE 16 2B 33 77 3D 4C B6 55 8B    \Aa...>.+3w=L6U.
00000190: D2 55 60 92 05 F7 F7 A4 97 54 4B 6C A6 68 48 84    RU`..ww$.TKl&hH.
000001a0: 68 FF D2 B6 6C 02 31 FC 24 01 78 EA BD 20 AD 15    h.R6l.1|$.xj=.-.
000001b0: F1 73 31 4B EB EF 18 1B 50 B3 13 F2 DC C6 4C E1    qs1Kko..P3.r\FLa
000001c0: 75 8B 94 52 6B C5 09 37 55 1E 45 66 6A 92 39 23    u..RkE.7U.Efj.9#
000001d0: C9 2D FD BB EC AD FD CF C4 30 75 FF 44 66 FA 85    I-};l-}OD0u.Dfz.
000001e0: D9 7C 18 72 AE 63 45 60 DD D7 65 44 84 49 95 8D    Y|.r.cE`]WeD.I..
000001f0: 2C 70 6C 57 8E E9 A9 EB B6 F6 78 BD D6 88 99 F6    ,plW.i)k6vx=V..v
00000200: FC 25 B1 0A FF DF CB 77 6A 67 37 24 A5 3D 8F A1    |%1.._Kwjg7$%=.!
00000210: 27 9B 4F 42 0E CD B8 87 6E C9 99 FC 6F 4C 53 4B    '.OB.M8.nI.|oLSK
00000220: 01 EA B6 AF 99 F8 22 C1 8F 1E C1 66 D6 8A 09 D6    .j6/.x"A..AfV..V
00000230: 99 79 91 F7 C1 2A 08 1F 81 CB 5E DD C3 CA 86 8F    .y.wA*...K^]CJ..
00000240: 57 BF 17 A2 64 6B 69 56 AE 19 1F 57 AD A6 D8 C2    W?."dkiV...W-&XB
00000250: 06 28 EB 46 D3 E4 85 51 3E E2 A5 40 50 50 85 7D    .(kFSd.Q>b%@PP.}
00000260: 72 6B 20 87 1A 6E 73 E1 B8 88 9E 20 23 48 6D FE    rk...nsa8...#Hm~
00000270: C2 0D 39 ED 24 B2 6D B5 9B 81 B6 BC F4 EE DE A2    B.9m$2m5..6..bB.{"&.!4.R^5
000002b0: 2A A6 E2 71 D7 4F 96 0A EC AE 8D 39 27 B8 CF 61    *&bqWO..l..9'8Oa
000002c0: CC ED E9 AF 74 C3 95 D3 E3 96 32 20 E6 31 0B E4    Lmi/tC.Sc.2.f1.d
000002d0: DC F4 FF 41 37 36 E7 DB 87 AE B3 7D BF CA F8 05    \t.A76g[..3}?Jx.
000002e0: 72 2A 38 AB B8 8E 98 43 97 C8 5E 80 57 C6 E7 1E    r*8+8..C.H^.WFg.
000002f0: 86 75 CE CD CE BF CF 10 C9 8A C2 C9 6E 33 41 AC    .uNMN?O.I.BIn3A,
00000300: 91 AC A8 F3 1B E6 D5 0A 22 A1 2C 4C 68 19 51 4D    .,(s.fU."!,Lh.QM
00000310: 17 DA AE E1 D7 BC 0E 2D F8 14 61 E2 4F BA 26 A3    .Z.aW<.-x.abO:&#
00000320: 0A E4 A6 BE 08 EA 3C 28 E6 C5 6B CA 3A 86 D2 59    .d&>.j<(fEkJ:.RY
00000330: 34 C2 ED 91 72 5A EF 2C BE D7 38 A4 60 D7 F3 97    4Bm.rZo,>W8$`Ws.
00000340: BB E6 FD C2 D0 29 10 B5 A4 79 D8 3E 61 48 8A F9    ;f}BP).5$yX>aH.y
00000350: C6 D8 13 D0 FD DB D6 FA 24 7F CD 5A BF 06 57 49    FX.P}[Vz$.MZ?.WI
00000360: 51 EC ED B2 74 AB 92 1D 37 68 70 A2 A5 31 B5 5F    Qlm2t+..7hp"%15_
00000370: EA CF 9E 3E 6A B1 78 16 B7 94 D1 46 7B 63 C1 67    jO.>j1x.7.QF{cAg
00000380: D2 B0 08 44 64 1E 68 15 39 80 E3 DD EB C0 E1 71    R0.Dd.h.9.c]k@aq
00000390: E8 EE D0 4D DF 4F 41 E0 96 C5 34 AD BC D3 9E 88    hnPM_OA`.E4-'.7pV%_>.T..
00000430: 7F FC AD 71 CE AF 54 8B 5D DC 27 34 20 A3 0A 73    .|-qN/T.]\'4.#.s
00000440: 76 A5 81 33 22 31 56 6B 1D 82 C4 32 FB 82 15 F6    v%.3"1Vk..D2{..v
00000450: 97 C8 47 29 3C 9E 59 9A C0 83 48 A0 55 CB C8 D6    .HG)<.Y.@.H.UKHV
00000460: 36 92 CC 54 A7 00 E3 28 9E 99 45 B2 E5 7E 88 A7    6.LT'.c(..E2e~.'
00000470: 28 4E CA 75 17 3C D3 B5 6C F5 FD AC 05 55 BF F7    (NJu.?{.T2Yr.h.wYGw
000004d0: 3C 19 C8 7B 81 9B 17 19 E9 81 A0 36 AD C6 62 71    <.H{....i..6-Fbq
000004e0: DB 68 72 8F 6A 37 45 D9 0E 6E DC 2C 5E 52 C2 75    [hr.j7EY.n\,^RBu
000004f0: 51 2F F9 CE 8A 10 12 E9 C8 68 A9 D6 A6 D7 5B 14    Q/yN...iHh)V&W[.
00000500: 11 51 42 FD BE B5 09 56 7F 19 C3 EB A7 A6 DF 6C    .QB}>5.V..Ck'&_l
00000510: 55 A3 11 DC EF 81 C3 CD DD 63 BF 38 F8 5A 4A 45    U#.\o.CM]c?8xZJE
00000520: 33 24 7B A4 55 B3 85 A6 87 75 3B 85 51 5C 03 B7    3${$U3.&.u;.Q\.7


  


  UPDATE TO THE CODE

  


  1st Packet find here
2nd Packet find here

  


  I have updated the code as per one of the comment to read only MDAT box to retrieve H264 stream from the incoming bytes[] through the socket, now I send only MDAT box contents (next byte after MDAT box)

  


  public Map.Entry> hasMdat(byte[] byteArray) {
    for (int i = 0; i < byteArray.length - 3; i++) {
        if (byteArray[i] == (byte) 109 &&
                byteArray[i + 1] == (byte) 100 &&
                byteArray[i + 2] == (byte) 97 &&
                byteArray[i + 3] == (byte) 116) {

            return Map.entry(true, Arrays.asList(i, i + 1, i + 2, i + 3));
        }
    }
    return Map.entry(false, List.of(0));
}


  


  This is my code which handles the byte stream

  


  initSocketConnection(new VideoStreamCallback() {
        @Override
        public void onVideoStreamReceived(byte[] bytes) {
           
Map.Entry> b = hasMdat(bytes);
        if (b.getKey()) {
            byte[] b1 = Arrays.copyOfRange(bytes, b.getValue().get(3) + 1, bytes.length);
  //write b1 back to client using spring SSE
            
        }

        }
    });


  


• avutil : remove deprecated FF_API_INTERLACED_FRAME

  19 février, par James Almer

  avutil : remove deprecated FF_API_INTERLACED_FRAME
  Deprecated since 2023-05-04.
  Signed-off-by : James Almer <jamrial@gmail.com>
  

  • [DH] libavcodec/decode.c
  • [DH] libavcodec/encode.c
  • [DH] libavfilter/buffersrc.c
  • [DH] libavfilter/qsvvpp.c
  • [DH] libavfilter/vf_coreimage.m
  • [DH] libavfilter/vf_deinterlace_vaapi.c
  • [DH] libavfilter/vf_estdif.c
  • [DH] libavfilter/vf_field.c
  • [DH] libavfilter/vf_fieldhint.c
  • [DH] libavfilter/vf_fieldmatch.c
  • [DH] libavfilter/vf_fieldorder.c
  • [DH] libavfilter/vf_idet.c
  • [DH] libavfilter/vf_kerndeint.c
  • [DH] libavfilter/vf_nnedi.c
  • [DH] libavfilter/vf_separatefields.c
  • [DH] libavfilter/vf_setparams.c
  • [DH] libavfilter/vf_telecine.c
  • [DH] libavfilter/vf_tinterlace.c
  • [DH] libavfilter/vf_w3fdif.c
  • [DH] libavfilter/vf_weave.c
  • [DH] libavfilter/vsrc_gradients.c
  • [DH] libavfilter/vsrc_testsrc.c
  • [DH] libavfilter/yadif_common.c
  • [DH] libavutil/frame.c
  • [DH] libavutil/frame.h
  • [DH] libavutil/version.h