Sur d’autres sites (13934)

• How to get video pixel location from screen pixel location ?

  22 février 2024, par AmLearning

  Wall of Text so I tried breaking it up into sections to make it better sorry in advance

  


  The problem

  


  I have some video files that I am reading with ffmpeg to get the colors at specific pixels, and all seems well, but I just ran into a problem with finding the right pixel to input. I realized (or mistakingly believe) that the pixel location (x,y) on the screen will be different than the local pixel location so to speak of the video (ie. If I want to get pixel 50,0 of the video that will be different than my screen's pixel 50,0 because the resolutions don't match). I was trying to think of a way to convert my screen's pixel location into the "local pixel location", and I have two ideas but I am not sure if any of them is any good. Note I am currently using cmd+shift+4 on macos to get the screen coordinates and the video is playing fullscreen like in the screenshot below.

  


  Ideas

  


  


  1. If I manually measure and account for this vertical offset, would it effectively convert the screen coordinate into the "local" one ?

     


  


  2. If I instead adjust my SwsContext to put the destination height and width as that of my screen, will it effectively replace the need to convert screen coordinates to the video coordinates ?

     


  


  


  Problems with the Ideas

  


  The problems I see with the first solution are that I am assuming there is no hidden horizontal offset (or conversely that all of the width of the video is actually renderable on the screen). Additionally, this solution would only get an approximate result as I would need to manually measure the offsets, screen width, and screen height using the method I currently am using to get the screen coordinates.

  


  With the second solution, aside from the question of if it will even work, the problem becomes that I can no longer measure what the screen coordinates I want are because I can't seem to get rid of those black bars in VLC.

  


  Some Testing I did

  


  Given that if the black bars are part of the video itself, my entire problem would be fixed (maybe ?) I tried seeing if the black bars were part of the video, and when I looked at the frame data's first pixel, it was black. The problem then is that if the black bars are entirely part of the video, then why are the colors I get for some pixels slightly off (I am checking with ColorSync Utility). These colors aren't just slightly off as in wrong but it seems more that they belong to a slightly offset region of the video.

  


  However, this may be somewhat explained if ffmpeg reads right to left. When I put the top left corner of the video into the program and looked again at the pixel data in the frame for that location (location again was calculated by assuming the video location would be the same as the screen location) instead of getting white, I got a bluish color much like the glove in the top right corner.

  


  The Watered Down Code

  


      struct SwsContext *rescaler = NULL;
    rescaler = sws_getContext(codec_context->width, codec_context->height, codec_context->pix_fmt, codec_context->width, codec_context->height, AV_PIX_FMT_RGB0, SWS_FAST_BILINEAR, NULL, NULL, 0);

// Get Packets (containers for frames but not guaranteed to have a full frame) and Frames
    while (av_read_frame(avformatcontext, packet) >= 0)
    {
        
        // determine if packet is video packet
        if (packet->stream_index != video_index)
        {
            continue;
        }
        
        // send packet to decoder
        if (avcodec_send_packet(codec_context, packet) < 0)
        {
            perror("Failed to decode packet");
        }
        
        // get frame from decoder
        int response = avcodec_receive_frame(codec_context, frame);
        if (response == AVERROR(EAGAIN))
        {
            continue;
        }
        else if (response < 0)
        {
            perror("Failed to get frame");
        }
        
        // convert frame to RGB0 colorspace 4 bytes per pixel 1 per channel
        response = sws_scale_frame(rescaler, scaled_frame, frame);
        if(response < 0){
            perror("Failed to change colorspace");
        }
        // get data and write it
        int pixel_number = y*(scaled_frame->linesize[0]/4)+x; // divide by four gets pixel linesize (4 byte per pixel)
        int byte_number = 4*(pixel_number-1); // position of pixel in array
        // start of debugging things
        int temp = scaled_frame->data[0][byte_number]; // R
        int one_after = scaled_frame->data[0][byte_number+1]; // G
        int two_after = scaled_frame->data[0][byte_number+2]; // B
        int als; // where i put the breakpoint
        // end of debugging things
    }


  


  In Summary

  


  I have no idea what is happening.

  


  I take the data for a pixel and compare it to what colorsync utility says should be there, but it is always slightly off as though the pixel I was actually reading was offset from what I thought I was reading. Therefore, I want to find a way to get the pixel location in a video given a screen coordinate when the video is in fullscreen, but I have no idea how to (aside from a few ideas that are probably bad at best).

  


  Also does FFMPEG put the frame data right to left ?

  


  A Video Better Showing My Problem

  


  https://www.youtube.com/watch?v=NSEErs2lC3A

  


• I tried to play the audio on Alexa skill from my S3 Bucket, from the test tab, **it show but in fact, I can't hear any sound

  19 avril 2022, par Siti Mayna

  So I tried to play the audio on Alexa skill from my S3 Bucket, from the test tab, it show but in fact, I can't hear any sound. Another fact is, that I tried to use the sample audio from https://developer.amazon.com/en-US/docs/alexa/custom-skills/ask-soundlibrary.html and it is worked, but why it won't work when it comes from my own S3 Bucket ?

  


  Notes :

  


  I've tried to test the skill using my mobile phone also.

  


  I've tried to encode the audio using FFmpeg.

  


  I've tried to use Jovo to convert the audio. https://v3.jovo.tech/audio-converter

  


  I don't know how to fix this error.

  


  There is no error message on cloud watch.

  


  Assumptions :
There is some problem related to the audio resources or there is more set to play audio from S3 Bucket since the sample audio is working.

  


  Steps to reproduce :

  


  


  Build the interaction model

  


  


  


  Encode the audio to make it Alexa skill friendly (fulfill the requirements, like sample rate, etc), I used and tried all of these :

  


  


  A :

  


  ffmpeg -i  -ac 2 -codec:a libmp3lame -b:a 48k -ar 16000 -write_xing 0 


  


  B :

  


  ffmpeg -i  -ac 2 -codec:a libmp3lame -b:a 48k -ar 24000 -write_xing 0 


  


  C :

  


  ffmpeg -y -i input.mp3 -ar 16000 -ab 48k -codec:a libmp3lame -ac 1 output.mp3


  


  


  Upload the audio resources on S3Bucket
Audio sample on s3 storage but none of them are produce any sounds

  


  


  


  Use the link and insert it to APLA.json

  


  


  
    {
      "type": "APLA",
      "version": "0.91",
      "description": "Simple document that generates speech",
      "mainTemplate": {
        "parameters": [
          "payload"
        ],
        "type": "Sequencer",
        "items": [
          {
            "type": "Audio",
            "source": "https://72578561-d9d8-47b4-811c-cafbcbc5ddb9-us-east-1.s3.amazonaws.com/Media/one-small-step-alexa-24.mp3"
          }
        ]
      }
    }



  


  notes : I change the link sources based on audio that I tried.

  


  


  the intent on lambda_function.py :

  


  


  def _load_apl_document(file_path):
    # type: (str) -> Dict[str, Any]
    """Load the apl json document at the path into a dict object."""
    with open(file_path) as f:
        return json.load(f)

class LaunchRequestHandler(AbstractRequestHandler):
    """Handler for Skill Launch."""
    def can_handle(self, handler_input):
        # type: (HandlerInput) -> bool

        return ask_utils.is_request_type("LaunchRequest")(handler_input)

    def handle(self, handler_input):
        # type: (HandlerInput) -> Response
        logger.info("In LaunchRequestHandler")

        # type: (HandlerInput) -> Response
        speak_output = "Hello World!"
        # .ask("add a reprompt if you want to keep the session open for the user to respond")

        return (
            handler_input.response_builder
                #.speak(speak_output)
                .add_directive(
                        RenderDocumentDirective(
                            token="pagerToken",
                            document=_load_apl_document("APLA.json"),
                            datasources={}
                        )
                    )
                .response
        )


  


  


  Deploy

  


  


  


  Test it

  


  


  


  The result of the test on my end :

The response for testing

  


  


  the JSON response :

  


  {
    "body": {
        "version": "1.0",
        "response": {
            "directives": [
                {
                    "type": "Alexa.Presentation.APLA.RenderDocument",
                    "token": "pagerToken",
                    "document": {
                        "type": "APLA",
                        "version": "0.91",
                        "description": "Simple document that generates speech",
                        "mainTemplate": {
                            "parameters": [
                                "payload"
                            ],
                            "type": "Sequencer",
                            "items": [
                                {
                                    "type": "Audio",
                                    "source": "https://72578561-d9d8-47b4-811c-cafbcbc5ddb9-us-east-1.s3.amazonaws.com/Media/one-small-step-alexa-24.mp3"
                                }
                            ]
                        }
                    },
                    "datasources": {}
                }
            ],
            "type": "_DEFAULT_RESPONSE"
        },
        "sessionAttributes": {},
        "userAgent": "ask-python/1.16.1 Python/3.7.12"
    }
}


  


  


  On my cloud Watch :
Cloud Watch

  


  


• CGO : How do I write to a file in Golang using a pointer to the C data ?

  24 avril 2018, par nevernew

  I’m writing an app for the windows platform using FFmpeg and it’s golang wrapper goav, but I’m having trouble understanding how to use the C pointers to gain access to an array.

  I’m trying to write the frame data, pointed to by a uint8 pointer from C, to a .ppm file in golang.

  Once I have this done, for proof of concept that FFmpeg is doing what I expect it to, I want to set the frames to a texture in OpenGl to make a video player with cool transitions ; any pointers to do that nice and efficiently would be so very helpful ! I’m guessing I need to write some shader code to draw the ppm as a texture...

  I’m starting to understanding how to cast the pointers between C and Go types, but how can I access the data and write it in Go with the same result as C ? In C I just have to set the pointer offset for the data and state how much of it to write :

  for (y = 0; y < height; y++) {
  
      fwrite(pFrame->data[0]+y*pFrame->linesize[0], 1, width*3, pFile);
  
  }

  I’ve stripped out all the relevant parts of the C code, the wrapper and my code, shown below :

  C code - libavutil/frame.h

  #include 
  
  
  
  typedef struct AVFrame {
  
  #define AV_NUM_DATA_POINTERS 8
  
      uint8_t *data[AV_NUM_DATA_POINTERS];
  
      int linesize[AV_NUM_DATA_POINTERS];
  
  }

  Golang goav wrapper

  package avutil
  
  
  
  /*
  
      #cgo pkg-config: libavutil
  
      #include <libavutil></libavutil>frame.h>
  
      #include 
  
  */
  
  import "C"
  
  import (
  
      "unsafe"
  
  )
  
  
  
  type Frame C.struct_AVFrame
  
  
  
  func Data(f *Frame) *uint8 {
  
      return (*uint8)(unsafe.Pointer((*C.uint8_t)(unsafe.Pointer(&amp;f.data))))
  
  }
  
  func Linesize(f *Frame) int {
  
      return int(*(*C.int)(unsafe.Pointer(&amp;f.linesize)))
  
  }

  My Golang code

  package main
  
  
  
  import "github.com/giorgisio/goav/avutil"
  
  
  
  func saveFrame(videoFrame *avutil.Frame, width int, height int, iFrame int) {
  
      var szFilename string
  
      var y int
  
      var file *os.File
  
      var err error
  
  
  
      szFilename = ""
  
  
  
      // Open file
  
      szFilename = fmt.Sprintf("frame%d.ppm", iFrame)
  
  
  
      if file, err = os.Open(szFilename); err != nil {
  
          log.Println("Error Reading")
  
      }
  
  
  
      // Write header
  
      fh := []byte(fmt.Sprintf("P6\n%d %d\n255\n", width, height))
  
      file.Write(fh)
  
      var b byte = 0
  
      // Write pixel data
  
      for y = 0; y < height; y++ {
  
          d := avutil.Data(videoFrame) // d should be a pointer to the first byte of data
  
          l := avutil.Linesize(videoFrame)
  
  
  
          // I'm basically lost trying to figure out how to write this to a file
  
          data := make([]byte, width*3)
  
  
  
          addr := int(*d) + y*l // figure out the address
  
  
  
          for i := 0; i < l; i++ {
  
              // This is where I'm having the problem, I get an "invalid
  
              // memory address or nil pointer dereference" error
  
              byteArrayPtr := (*byte)(unsafe.Pointer(uintptr(addr) + uintptr(i)*unsafe.Sizeof(b)))
  
              data = append(data, *byteArrayPtr)
  
              fmt.Println(*byteArrayPtr)
  
          }
  
          file.Write(data)
  
      }
  
  
  
      file.Close()
  
  }

  So, how can I write to a file using a pointer to the data, like you can do in C ?