Recherche avancée

Sur d’autres sites (5078)

• Is there any open source solution to display a remote stream inside a Hololens2 UWP Vuforia application ?

  19 avril 2023, par T777

  What do we need ?

  


  We are trying to develop an application for quality management in which we show an hologram on a metal part as an assitance marking. (using Hololen2 + Vuforia + ModleTargets) The employee uses an sensor to follow this assitance marking and the data will be analyzed live by a test device. The results are outputed on a screen / are visible at an closed source application of the manufacturer of the test device.

  


  Capturing of the video output :
The current plan is to capture the video stream of the test device via capture card. Add a via mrtk2 videopanel inside the vuforia app and stream the captured video to the Hololens2 using obs or an OpenCV python script for screen recording.

  


  What we have tried so far

  


  1) Sending Raw udp stream
via RMTP and decoding + converting with gstreamer server and writing an own library in Unity for Receiving
Result : Temporary stopped, because receiving the udp streams needs connection/ session management (signalling) frame syncing and agreement on video size, color format, frame rate etc.. and we have no solution.
An own implementation of any of this would have high complexity is consuming a lot of time.

  


  2) Using available protocols that i can find on the web
Actually there are some protocols already developed for session creation and streaming :

  


  

  • HTTP streaming (HLS) (Transport + Session)

  


  • RTMP (Transport + Session),

  


  • RTP (Transport) + RTPS (Session),

  


  • WebRTC : Is possible with different protocol stacks
RTMP/TCP/UDP (Transport) + SDP (standardized format for video paramaters) + ICE (p2p)/ WHIP (http, client-server) / Websocket(client-server) (signaling protocols) that can be used and some good open source streaming servers (gstreamer, mediamtx and srs)

  


  


  When using these the video will be encoded typcially with xh264 and need to be decoded on the HoloLens 2. There are APIs to C/C++ native (hardware) decoding libraries like unity-vlc and ffmpeg.NET that needing media library ffmpeg. I could figure out (not tested) that there is an hardware h264 decoder on the HoloLens2 but I have no clue how to access it. Since there I couldnt disvocer any information about HoloLens2 media libraries.

  


  3) Using Unity packages

  


  


  • Unit package WebRTC (https://docs.unity3d.com/Packages/com.unity.webrtc@2.4/manual/index.html) supports multiple transport protocols seem to have no signaling mechanism and

    


  


  • Unity package Render streaming (https://docs.unity3d.com/Packages/com.unity.renderstreaming@3.1/manual/index.html) is a fully integrated unity to unity and unit to browser streaming package with integrated stream Server with web GUI. It offers various streaming protocols (TCP, UDP,rmtp) and signaling mechanism over websocket, http (seems custom and not whip) or Furiouus.
BUT it doesnt support UWP as noted into the documentation. Implementing an example application we could demonstrate an working example with Vuforia, but it fails on build with target UWP on missing libraries.
Similar to : https://www.youtube.com/watch?v=nHRC0uGBnn8

    


  


  


  Will be testing other compile options tomorrow..

  


  

  • Mixed Reality WebRTC (https://github.com/microsoft/MixedReality-WebRTC) :
Various protocol support, Microsoft brought Webrtc specifically to HoloLens.
Deprecated, as fas as I can see just support for Hololens1 and ARM32. So i can not evaluate if trying it with this is worth it.

  


  


  What are the next options ?

  


  

  • Developing a raw udp streaming library with untiy directly.

  


  • Rebuilding the application with visionlib (ARM32) compatible and MixedRealityWebRTC (ARM32)

  


  • Porting ffmpeg + API to UWP ?

  


  • Also there seem some affords to make WebRTC in general available to UWP platforms : https://github.com/microsoft/winrtc

  


  


  The questions

  


  

  • Does Vuforia support ARM32 ?

  


  • How to access hardware decoder of Hololens2 via Unity Code ?

  


  


• FFMPEG : Converting from raw audio to audio/mp4 (audio is being converted with slow speed)

  29 décembre 2017, par Valdir

  If I convert from mp3 to mp4 directly everything works perfectly. But if I try to convert from raw pcm, the audio speed is slowed down.

  I’ve tried the following (this works) :

  ffmpeg -i mp3/1.mp3 -strict -2 final.mp4

  This doesn’t work as expected :

  ffmpeg -f s16le -i final.raw -strict -2 -r 26 final.mp4

  With the following output :

  Input #0, s16le, from 'final.raw':
  
    Duration: 00:08:37.38, bitrate: 705 kb/s
  
      Stream #0:0: Audio: pcm_s16le, 44100 Hz, 1 channels, s16, 705 kb/s
  
  File 'final.mp4' already exists. Overwrite ? [y/N] y
  
  Output #0, mp4, to 'final.mp4':
  
    Metadata:
  
      encoder         : Lavf56.40.101
  
      Stream #0:0: Audio: aac ([64][0][0][0] / 0x0040), 44100 Hz, mono, fltp, 128 kb/s
  
      Metadata:
  
        encoder         : Lavc56.60.100 aac
  
  Stream mapping:
  
    Stream #0:0 -> #0:0 (pcm_s16le (native) -> aac (native))
  
  Press [q] to stop, [?] for help
  
  size=    8273kB time=00:08:37.38 bitrate= 131.0kbits/s
  
  video:0kB audio:8185kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.073808%

  I’ve tried to set parameters like :

  ffmpeg -ar 44100 -f s16le -i final.raw -strict -2 -r 26 final.mp4

  With no luck.

  In order to get the PCM from mp3 I’m using nodejs lame decoder :

  var decoder = new lame.Decoder({
  
          channels: 2,
  
          bitDepth: 16,
  
          sampleRate: 44100,
  
          bitRate: 128,
  
          outSampleRate: 44100, // 22050
  
          mode: lame.STEREO
  
      });

• Unable to split audio using easy_audio_trimmer

  27 juillet 2023, par Sana Wasim

  can we use the easy_audio_trimmer package to split an audio ? I tried using the ffmpeg but it is conflicting with the above package and not work.

  


  I tried splitting by using these functions and it gave an error at the FlutterFFmpeg() method and i cant find an alternative also the duration(filePath) in the command final durationResult = await flutterSound.duration(filePath) ; shows an error

  


  Future<void> _splitAudio() async {&#xA;    setState(() {&#xA;      _progressVisibility = true;&#xA;    });&#xA;&#xA;    // Get the application documents directory&#xA;    final appDocumentsDirectory = await getApplicationDocumentsDirectory();&#xA;&#xA;    // Get the input audio file path&#xA;    final inputAudioPath = widget.file.path;&#xA;&#xA;    // Get the output file names for the two parts&#xA;    final outputFileName1 = &#x27;split_audio_part1.mp3&#x27;;&#xA;    final outputFileName2 = &#x27;split_audio_part2.mp3&#x27;;&#xA;&#xA;    // Get the output file paths for the two parts&#xA;    final outputPath1 = &#x27;${appDocumentsDirectory.path}/$outputFileName1&#x27;;&#xA;    final outputPath2 = &#x27;${appDocumentsDirectory.path}/$outputFileName2&#x27;;&#xA;&#xA;    // Calculate the duration of the original audio&#xA;    final originalDuration = await _getAudioDuration(inputAudioPath);&#xA;&#xA;    // Calculate the durations of the two parts&#xA;    final part1Duration = _startValue;&#xA;    final part2Duration = originalDuration - _endValue;&#xA;&#xA;    // Construct the FFmpeg command to split the audio&#xA;    final ffmpeg = FlutterFFmpeg();&#xA;    final splitCommand = &#x27;-i $inputAudioPath -ss 0 -t $part1Duration -c copy $outputPath1 -ss $_endValue -t $part2Duration -c copy $outputPath2&#x27;;&#xA;&#xA;    try {&#xA;      // Execute the FFmpeg command to split the audio&#xA;      final int result = await ffmpeg.execute(splitCommand);&#xA;&#xA;      if (result == 0) {&#xA;        setState(() {&#xA;          _progressVisibility = false;&#xA;        });&#xA;        debugPrint(&#x27;Audio split successfully.&#x27;);&#xA;      } else {&#xA;        setState(() {&#xA;          _progressVisibility = false;&#xA;        });&#xA;        debugPrint(&#x27;Failed to split audio.&#x27;);&#xA;      }&#xA;    } catch (error) {&#xA;      setState(() {&#xA;        _progressVisibility = false;&#xA;      });&#xA;      debugPrint(&#x27;Error while splitting audio: $error&#x27;);&#xA;    }&#xA;  }&#xA;&#xA;  Future<int> _getAudioDuration(String filePath) async {&#xA;    final flutterSound = FlutterSound();&#xA;    final durationResult = await flutterSound.duration(filePath);&#xA;    return durationResult.inMilliseconds;&#xA;  }&#xA;</int></void>

  


  Dependencies

  


   path_provider: ^2.0.15
  ffmpeg_kit_flutter: ^5.1.0
  audioplayers: ^4.1.0
  flutter_sound: ^9.2.13