Recherche avancée

Sur d’autres sites (3752)

• Is there any open source solution to display a remote stream inside a Hololens2 UWP Vuforia application ?

  19 avril 2023, par T777

  What do we need ?

  


  We are trying to develop an application for quality management in which we show an hologram on a metal part as an assitance marking. (using Hololen2 + Vuforia + ModleTargets) The employee uses an sensor to follow this assitance marking and the data will be analyzed live by a test device. The results are outputed on a screen / are visible at an closed source application of the manufacturer of the test device.

  


  Capturing of the video output :
The current plan is to capture the video stream of the test device via capture card. Add a via mrtk2 videopanel inside the vuforia app and stream the captured video to the Hololens2 using obs or an OpenCV python script for screen recording.

  


  What we have tried so far

  


  1) Sending Raw udp stream
via RMTP and decoding + converting with gstreamer server and writing an own library in Unity for Receiving
Result : Temporary stopped, because receiving the udp streams needs connection/ session management (signalling) frame syncing and agreement on video size, color format, frame rate etc.. and we have no solution.
An own implementation of any of this would have high complexity is consuming a lot of time.

  


  2) Using available protocols that i can find on the web
Actually there are some protocols already developed for session creation and streaming :

  


  

  • HTTP streaming (HLS) (Transport + Session)

  


  • RTMP (Transport + Session),

  


  • RTP (Transport) + RTPS (Session),

  


  • WebRTC : Is possible with different protocol stacks
RTMP/TCP/UDP (Transport) + SDP (standardized format for video paramaters) + ICE (p2p)/ WHIP (http, client-server) / Websocket(client-server) (signaling protocols) that can be used and some good open source streaming servers (gstreamer, mediamtx and srs)

  


  


  When using these the video will be encoded typcially with xh264 and need to be decoded on the HoloLens 2. There are APIs to C/C++ native (hardware) decoding libraries like unity-vlc and ffmpeg.NET that needing media library ffmpeg. I could figure out (not tested) that there is an hardware h264 decoder on the HoloLens2 but I have no clue how to access it. Since there I couldnt disvocer any information about HoloLens2 media libraries.

  


  3) Using Unity packages

  


  


  • Unit package WebRTC (https://docs.unity3d.com/Packages/com.unity.webrtc@2.4/manual/index.html) supports multiple transport protocols seem to have no signaling mechanism and

    


  


  • Unity package Render streaming (https://docs.unity3d.com/Packages/com.unity.renderstreaming@3.1/manual/index.html) is a fully integrated unity to unity and unit to browser streaming package with integrated stream Server with web GUI. It offers various streaming protocols (TCP, UDP,rmtp) and signaling mechanism over websocket, http (seems custom and not whip) or Furiouus.
BUT it doesnt support UWP as noted into the documentation. Implementing an example application we could demonstrate an working example with Vuforia, but it fails on build with target UWP on missing libraries.
Similar to : https://www.youtube.com/watch?v=nHRC0uGBnn8

    


  


  


  Will be testing other compile options tomorrow..

  


  

  • Mixed Reality WebRTC (https://github.com/microsoft/MixedReality-WebRTC) :
Various protocol support, Microsoft brought Webrtc specifically to HoloLens.
Deprecated, as fas as I can see just support for Hololens1 and ARM32. So i can not evaluate if trying it with this is worth it.

  


  


  What are the next options ?

  


  

  • Developing a raw udp streaming library with untiy directly.

  


  • Rebuilding the application with visionlib (ARM32) compatible and MixedRealityWebRTC (ARM32)

  


  • Porting ffmpeg + API to UWP ?

  


  • Also there seem some affords to make WebRTC in general available to UWP platforms : https://github.com/microsoft/winrtc

  


  


  The questions

  


  

  • Does Vuforia support ARM32 ?

  


  • How to access hardware decoder of Hololens2 via Unity Code ?

  


  


• ffmpeg drop frames on purpose to lower filesize

  20 octobre 2014, par max

  Our security system records and archives our IP cameras streams with ffmpeg -use_wallclock_as_timestamps 1 -i rtsp://192.168.x.x:554/mpeg4 -c copy -t 60 my_input_video.avi

  I run it with crontab every minute so it creates videos of 60 seconds ( 15Mb) for each camera every minute. When an intrusion occurs, the camera sends a picture through FTP and a script called by incrontab :

  1- forwards immediately the picture by email

  2- selects the video covering the minute the intrusion occured, compress it with h264 (to 2,6Mb) and sends it by email

  It is working really well but if a thief crosses the path of various cameras, the connection to the SMTP server is not fast enough so video emails are delayed. I’d like to compress the videos even more to avoid that. I could lower the resolution (640x480 to 320x240 for example) but sometimes 640x480 is handy to zoom on something which looks to be moving...

  So my idea is to drop frames in the video in order to lower the filesize. I don’t care if the thief is walking like a "stop motion Lego" on the video, the most important is I know there is someone so I can act.

  mediainfo my_input_video.avi says Frame rate = 600.000 fps but it is of course wrong. FPS sent by IP cameras are always false because it varies with the network quality ; this is why i use "-use_wallclock_as_timestamps 1" in my command to record the streams.

  with ffmpeg -i  my_input_video.avi -vcodec h264 -preset ultrafast -crf 28 -acodec mp3 -q:a 5 -r 8 output.avi the video is OK but filesize is higher (3Mb)

  with ffmpeg -i  my_input_video.avi -vcodec h264 -preset ultrafast -crf 28 -acodec mp3 -q:a 5 -r 2 output.avi the filesize is lower (2,2Mb) but the video doesn’t work (it is blocked at the first frame).

  Creating a mjpeg video (mjpeg = not interlaced frames) in the middle of the process (first exporting to mjpeg with less frames and then exporting to h264) creates same results.

  Do you know how I can get my thief to walk like a "stop motion Lego" to lower the filesize to a minimum ?

  Thanks for any help

• Why is this ffmpeg conversion turning up trash ?

  12 octobre 2015, par DigitalJedi805

  So, the company I work for still archives some of our data in Windows Media format.

  I’ve written a C# application that loops through all of our WMVs, and in the event that a corresponding MP4 doesn’t exist, it fires off ffmpeg to convert the file.

  What I’m running into, is a combination of problems.

  When I run the following into ffmpeg ( rough C# ) :

  "-i " + File.FullName + " -vf scale=720:480 -b:v 512k -bufsize 512k -vcodec libx264 -acodec aac -strict experimental " + OutputPath

  I end up with a file that cannot play in our browser based player ( JWPlayer ), and cannot play on my local system in Windows Media Player.

  Additionally, the file is larger than my WMV, and is larger than the file I output with [roughly] the same parameters in AVS Video Converter.

  Furthermore, the file details don’t show any values for the video properties - as in, when I right click and go to properties->details, under ’video’, there is a list of length, height, width, frame rate, and bitrate - they are all empty - when I would very much expect some data normally.

  Does anyone have any idea how I can make the conversion more straightforward, or what might be creating the problem in the first place ?

  I’ve tried running this without scaling or bitrate parameters, and added them as an attempt to resolve the more core problem - obviously to no avail.

  For everyone’s appeasement...

  FFMPEG Output :

  video:537604kB audio:158748kB subtitle:0kB other streams:0kB global headers:0kB
  
  muxing overhead: 1.159483%
  
  [libx264 @ 0453d000] frame I:1234  Avg QP:19.98  size: 33645
  
  [libx264 @ 0453d000] frame P:94430 Avg QP:22.92  size:  3984
  
  [libx264 @ 0453d000] frame B:208152 Avg QP:30.67  size:   638
  
  [libx264 @ 0453d000] consecutive B-frames:  7.6%  1.8%  4.0% 86.6%
  
  [libx264 @ 0453d000] mb I  I16..4: 10.5% 61.2% 28.2%
  
  [libx264 @ 0453d000] mb P  I16..4:  0.4%  1.4%  0.3%  P16..4: 26.8%  9.3%  4.8%
  
   0.0%  0.0%    skip:57.0%
  
  [libx264 @ 0453d000] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8: 21.6%  1.1%  0.2%
  
   direct: 0.4%  skip:76.6%  L0:42.6% L1:55.1% BI: 2.3%
  
  [libx264 @ 0453d000] 8x8 transform intra:65.1% inter:71.9%
  
  [libx264 @ 0453d000] coded y,uvDC,uvAC intra: 61.6% 59.3% 30.9% inter: 6.0% 6.7%
  
   1.7%
  
  [libx264 @ 0453d000] i16 v,h,dc,p: 34% 52%  7%  7%
  
  [libx264 @ 0453d000] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 23% 27% 24%  3%  4%  4%  5%
  
   4%  7%
  
  [libx264 @ 0453d000] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 27% 38% 10%  3%  5%  4%  6%
  
   3%  5%
  
  [libx264 @ 0453d000] i8c dc,h,v,p: 47% 35% 14%  4%
  
  [libx264 @ 0453d000] Weighted P-Frames: Y:0.2% UV:0.1%
  
  [libx264 @ 0453d000] ref P L0: 67.8% 12.3% 14.3%  5.6%  0.0%
  
  [libx264 @ 0453d000] ref B L0: 88.7%  9.7%  1.6%
  
  [libx264 @ 0453d000] ref B L1: 92.6%  7.4%
  
  [libx264 @ 0453d000] kb/s:434.44

  I’ll make the video files available shortly.