Recherche avancée

Sur d’autres sites (6012)

• Trying to capture display output for real-time analysis with OpenCV ; I need help with interfacing with the OS for input

  26 juillet 2024, par mirari

  I want to apply operations from the OpenCV computer vision library, in real time, to video captured from my computer display.
The idea in this particular case is to detect interesting features during gameplay in a popular game and provide the user with an enhanced experience ; but I could think of several other scenarios where one would want to have live access to this data as well. 
At any rate, for the development phase it might be acceptable using canned video, but for the final application performance and responsiveness are obviously critical.

  



  I am trying to do this on Ubuntu 10.10 as of now, and would prefer to use a UNIX-like system, but any options are of interest.
My C skills are very limited, so whenever talking to OpenCV through Python is possible, I try to use that instead.
Please note that I am trying to capture NOT from a camera device, but from a live stream of display output ; and I'm at a loss as to how to take the input. As far as I can tell, CaptureFromCAM works only for camera devices, and it seems to me that the requirement for real-time performance in the end result makes storage in file and reading back through CaptureFromFile a bad option.

  



  The most promising route I have found so far seems to be using ffmpeg with the x11grab option to capture from an X11 display ;
(e.g. the command
ffmpeg -f x11grab -sameq -r 25 -s wxga -i :0.0 out.mpg
captures 1366x768 of display 0 to 'out.mpg').
I imagine it should be possible to treat the output stream from ffmpeg as a file to be read by OpenCV (presumably by using the CaptureFromFile function) maybe by using pipes ; but this is all on a much higher level than I have ever dealt with before and I could really use some directions. 
Do you think this approach is feasible ? And more importantly can you think of a better one ? How would you do it ?

  


• Unity : Converting Texture2D to YUV420P and sending with UDP using FFmpeg

  22 juin 2018, par potu1304

  In my Unity game each frame is rendered into a texture and then put together into a video using FFmpeg. Now my questions is if I am doing this right because avcodec_send_frame throws every time an exception.
  I am pretty sure that I am doing something wrong or in the wrong order or simply missing something.

  Here is the code for capturing the texture :

  void Update() {
  
          //StartCoroutine(CaptureFrame());
  
  
  
          if (rt == null)
  
          {
  
              rect = new Rect(0, 0, captureWidth, captureHeight);
  
              rt = new RenderTexture(captureWidth, captureHeight, 24);
  
              frame = new Texture2D(captureWidth, captureHeight, TextureFormat.RGB24, false);
  
          }
  
  
  
          Camera camera = this.GetComponent<camera>(); // NOTE: added because there was no reference to camera in original script; must add this script to Camera
  
          camera.targetTexture = rt;
  
          camera.Render();
  
  
  
          RenderTexture.active = rt;
  
          frame.ReadPixels(rect, 0, 0);
  
          frame.Apply();
  
  
  
          camera.targetTexture = null;
  
          RenderTexture.active = null;
  
  
  
          byte[] fileData = null;
  
          fileData = frame.GetRawTextureData();
  
          encoding(fileData, fileData.Length);
  
  
  
      }
  
  </camera>

  And here is the code for encoding and sending the byte data :

  private unsafe void encoding(byte[] bytes, int size)
  
      {
  
          Debug.Log("Encoding...");
  
          AVCodec* codec;
  
          codec = ffmpeg.avcodec_find_encoder(AVCodecID.AV_CODEC_ID_H264);
  
          int ret, got_output = 0;
  
  
  
          AVCodecContext* codecContext = null;
  
          codecContext = ffmpeg.avcodec_alloc_context3(codec);
  
          codecContext->bit_rate = 400000;
  
          codecContext->width = captureWidth;
  
          codecContext->height = captureHeight;
  
          //codecContext->time_base.den = 25;
  
          //codecContext->time_base.num = 1;
  
  
  
          AVRational timeBase = new AVRational();
  
          timeBase.num = 1;
  
          timeBase.den = 25;
  
          codecContext->time_base = timeBase;
  
          //AVStream* videoAVStream = null;
  
          //videoAVStream->time_base = timeBase;
  
  
  
  
  
  
  
          AVRational frameRate = new AVRational();
  
          frameRate.num = 25;
  
          frameRate.den = 1;
  
          codecContext->framerate = frameRate;
  
  
  
          codecContext->gop_size = 10;
  
          codecContext->max_b_frames = 1;
  
          codecContext->pix_fmt = AVPixelFormat.AV_PIX_FMT_YUV420P;
  
  
  
          AVFrame* inputFrame;
  
          inputFrame = ffmpeg.av_frame_alloc();
  
          inputFrame->format = (int)codecContext->pix_fmt;
  
          inputFrame->width = captureWidth;
  
          inputFrame->height = captureHeight;
  
          inputFrame->linesize[0] = inputFrame->width;
  
  
  
          AVPixelFormat dst_pix_fmt = AVPixelFormat.AV_PIX_FMT_YUV420P, src_pix_fmt = AVPixelFormat.AV_PIX_FMT_RGBA;
  
          int src_w = 1920, src_h = 1080, dst_w = 1920, dst_h = 1080;
  
          SwsContext* sws_ctx;
  
  
  
          GCHandle pinned = GCHandle.Alloc(bytes, GCHandleType.Pinned);
  
          IntPtr address = pinned.AddrOfPinnedObject();
  
  
  
          sbyte** inputData = (sbyte**)address;
  
          sws_ctx = ffmpeg.sws_getContext(src_w, src_h, src_pix_fmt,
  
                               dst_w, dst_h, dst_pix_fmt,
  
                               0, null, null, null);
  
  
  
          fixed (int* lineSize = new int[1])
  
          {
  
              lineSize[0] = 4 * captureHeight;
  
              // Convert RGBA to YUV420P
  
              ffmpeg.sws_scale(sws_ctx, inputData, lineSize, 0, codecContext->width, inputFrame->extended_data, inputFrame->linesize);
  
          }
  
  
  
          inputFrame->pts = counter++;
  
  
  
          if (ffmpeg.avcodec_send_frame(codecContext, inputFrame) < 0)
  
              throw new ApplicationException("Error sending a frame for encoding!");
  
  
  
          AVPacket pkt;
  
          pkt = new AVPacket();
  
          //pkt.data = inData;
  
          AVPacket* packet = &pkt;
  
          ffmpeg.av_init_packet(packet);
  
  
  
          Debug.Log("pkt.size " + pkt.size);
  
          pinned.Free();
  
          AVDictionary* options = null;
  
          ffmpeg.av_dict_set(&options, "pkt_size", "1300", 0);
  
          ffmpeg.av_dict_set(&options, "buffer_size", "65535", 0);
  
          AVIOContext* server = null;
  
          ffmpeg.avio_open2(&server, "udp://192.168.0.1:1111", ffmpeg.AVIO_FLAG_WRITE, null, &options);
  
          Debug.Log("encoded");
  
          ret = ffmpeg.avcodec_encode_video2(codecContext, &pkt, inputFrame, &got_output);
  
          ffmpeg.avio_write(server, pkt.data, pkt.size);
  
          ffmpeg.av_free_packet(&pkt);
  
          pkt.data = null;
  
          pkt.size = 0;
  
      }

  And every time I start the game

    if (ffmpeg.avcodec_send_frame(codecContext, inputFrame) < 0)
  
              throw new ApplicationException("Error sending a frame for encoding!");

  throws the exception.
  Any help in fixing the issue would be greatly appreciated :)

• Revisiting Nosefart and Discovering GME

  30 mai 2011, par Multimedia Mike — Game Hacking

  I found the following screenshot buried deep in an old directory structure of mine :

  
  
  

  I tried to recall how this screenshot came to exist. Had I actually created a functional KDE frontend to Nosefart yet neglected to release it ? I think it’s more likely that I used some designer tool (possibly KDevelop) to prototype a frontend. This would have been sometime in 2000.

  However, this screenshot prompted me to revisit Nosefart.

  Nosefart Background
  Nosefart is a program that can play Nintendo Sound Format (NSF) files. NSF files are files containing components that were surgically separated from Nintendo Entertainment System (NES) ROM dumps. These components contain the music playback engines for various games. An NSF player is a stripped down emulation system that can simulate the NES6502 CPU along with the custom hardware (2 square waves, 1 triangle wave, 1 noise generator, and 1 limited digital channel).

  Nosefart was written by Matt Conte and eventually imported into a Sourceforge project, though it has not seen any development since then. The distribution contains standalone command line players for Linux and DOS, a GTK frontend for the Linux command line version, and plugins for Winamp, XMMS, and CL-Amp.

  The Sourceforge project page notes that Nosefart is also part of XBMC. Let the record show that Nosefart is also incorporated into xine (I did that in 2002, I think).

  Upgrading the API
  When I tried running the command line version of Nosefart under Linux, I hit hard against the legacy audio API : OSS. Remember that ?

  In fairly short order, I was able to upgrade the CL program to use PulseAudio. The program is not especially sophisticated. It’s a single-threaded affair which checks for a keypress, processes an audio frame, and sends the frame out to the OSS file interface. All that was needed was to rewrite open_hardware() and close_hardware() for PA and then replace the write statement in play(). The only quirk that stood out is that including <pulse/pulseaudio.h> is insufficient for programming PA’s simple API. <pulse/simple.h> must be included separately.

  For extra credit, I adapted the program to ALSA. The program uses the most simplistic audio output API possible — just keep filling a buffer and sending it out to the DAC.

  Discovering GME
  I’m not sure what to do with the the program now since, during my research to attempt to bring Nosefart up to date, I became aware of a software library named Game Music Emu, or GME. It’s a pure C++ library that can essentially play any classic video game format you can possible name. Wow. A lot can happen in 10 years when you’re not paying attention.

  It’s such a well-written library that I didn’t need any tutorial or documentation to come up to speed. Just a quick read of the main gme.h header library enabled me in short order to whip up a quick C program that could play NSF and SPC files. Path of least resistance : Client program asks library to open a hardcoded file, synthesize 10 seconds of audio, and dump it into a file ; ask the FLAC command line program to transcode raw data to .flac file ; use ffplay to verify the results.

  I might develop some other uses for this library.