Recherche avancée

Sur d’autres sites (6790)

• Anomaly in raw I420 video generated by GStreamer

  16 avril 2024, par Lea

  Situation

  


  I'm trying to convert RGBA image data to YUV420P in multiple threads, then send this data to a main thread which splits the data it receives from each thread into separate frames and combines them in order to a video. Currently, I'm using FFmpeg for this task but I've found GStreamer to do a quicker job at colorspace conversion than FFmpeg.

  


  Problem

  


  The raw video generated by GStreamer does not match the expectations for YUV 4:2:0 planar video data. To test this, I've made a raw RGBA test video of 3 red 4x4 (16 pixel) frames.

  


  ffmpeg -f lavfi -i color=color=red -t 3 -r 1 -s 4x4 -f rawvideo -pix_fmt rgba ./input.rgba


  


  

  


  FFmpeg

  


  Now, first trying to convert it via FFmpeg as I'm doing it currently :

  


  ffmpeg -f rawvideo -pix_fmt rgba -s 4x4 -i input.rgba -f rawvideo -pix_fmt yuv420p ./ffmpeg.yuv420p


  


  This creates a 72 byte file => 1.5 bytes per pixel, 24 bytes per frame : As expected for yuv420p data.

  


  $ hexdump -C ./ffmpeg.yuv420p 
00000000  51 51 51 51 50 50 50 50  50 50 50 50 50 50 50 50  |QQQQPPPPPPPPPPPP|
00000010  5b 5b 5b 5b ee ee ee ee  51 51 51 51 50 50 50 50  |[[[[....QQQQPPPP|
00000020  50 50 50 50 50 50 50 50  5b 5b 5b 5b ee ee ee ee  |PPPPPPPP[[[[....|
00000030  51 51 51 51 50 50 50 50  50 50 50 50 50 50 50 50  |QQQQPPPPPPPPPPPP|
00000040  5b 5b 5b 5b ee ee ee ee                           |[[[[....|


  


  GStreamer

  


  Now trying to do the same via GStreamer, with the I420 format which corresponds to yuv420p as per their documentation :

  


  gst-launch-1.0 filesrc location=./input.rgba ! rawvideoparse format=rgba width=4 height=4 \
! videoconvert ! video/x-raw,format=I420 ! filesink location=./gstreamer.yuv420p


  


  This creates a 96 byte file => 2 bytes per pixel, 32 bytes per frame (?) : Unusual for yuv420p data. Additionally, none of the sections match the FFmpeg output, ruling out some kind of padding.

  


  $ hexdump -C ./gstreamer.yuv420p 
00000000  50 50 50 50 50 50 50 50  50 50 50 50 50 50 50 50  |PPPPPPPPPPPPPPPP|
00000010  5a 5a 00 00 5a 5a 00 00  ee ee 00 00 ed ed 00 00  |ZZ..ZZ..........|
00000020  50 50 50 50 50 50 50 50  50 50 50 50 50 50 50 50  |PPPPPPPPPPPPPPPP|
00000030  5a 5a 63 6b 5a 5a 77 62  ee ee 78 6d ed ed 2c 20  |ZZckZZwb..xm.., |
00000040  50 50 50 50 50 50 50 50  50 50 50 50 50 50 50 50  |PPPPPPPPPPPPPPPP|
00000050  5a 5a 00 00 5a 5a 00 00  ee ee 00 00 ed ed 00 00  |ZZ..ZZ..........|


  


  This output can also not be interpreted correctly as yuv420p by FFmpeg, leading to corrupted frames when trying to do so :

  


  ffmpeg -f rawvideo -pix_fmt yuv420p -s 4x4 -i gstreamer.yuv420p -f image2 "./%d.png"


  


  

  


  Solution ?

  


  For my personal problem I need a way to chop up raw I420 video generated by GStreamer into separate frames to work with. However, I would also like to understand why GStreamer behaves this way and which key piece I'm missing here.

  


  Additional notes

  


  I've ruled out an issue with the input in GStreamer, as piping it to autovideosink leads to a normal result. I'm also aware of multifilesink, but I would like to avoid writing to disk and rather work with the data directly in buffers.

  


• Seam carving

  13 février 2008, par Mikko Koppanen — Imagick, PHP stuff

  Today I was reading trough the ImageMagick ChangeLog and noticed an interesting entry. “Add support for liquid rescaling”. I rushed to check the MagickWand API docs and there it was : MagickLiquidRescaleImage ! After about ten minutes of hacking the Imagick support was done. Needless to say ; I was excited

  For those who don’t know what seam carving is check the demo here. More detailed information about the algorithm can be found here : “Seam Carving for Content-Aware Image Resizing” by Shai Avidan and Ariel Shamir

  To use this functionality you need to install at least ImageMagick 6.3.8-2 and liblqr. Remember to pass –with-lqr to ImageMagick configuration line. You can get liblqr here : http://liblqr.wikidot.com/. The Imagick side of the functionality should appear in the CVS today if everything goes as planned.

  Here is a really simple example just to illustrate the results of the operation. The parameters might be far from optimal (didn’t do much testing yet). The original dimensions of image are 500×375 and the resulting size is 500×200.

  Update : the functionality is pending until license issues are solved.

  php

  < view plain text >

  1. < ?php
  2.  
  3. /* Create new object */
  4. $im = new Imagick( ’test.jpg’ ) ;
  5.  
  6. /* Scale down */
  7. $im->liquidRescaleImage( 500, 200, 3, 25 ) ;
  8.  
  9. /* Display */
  10. header( ’Content-Type : image/jpg’ ) ;
  11. echo $im ;
  12.  
  13.  ?>

  The original image by flickr/jennconspiracy

  

  And the result :

  

  Update. On kenrick’s request here is an image which is scaled down to 300×300

  

• Why is there an audio delay on recording video stream with ffmpeg ?

  25 décembre 2023, par mqwerty

  I am trying to record video and audio stream (Line in Microphone Analog Audio) which are streaming from broadcaster computer with those parameters in the recorder computer ;

  &#xA;

  ffmpeg record parameters :

  &#xA;

  /usr/bin/ffmpeg -y -buffer_size max -thread_queue_size 8192 -i udp://225.0.5.11:1026 -buffer_size max -thread_queue_size 8192 -i udp://225.0.5.11:1032 -map 0:v -map 1:a -metadata title=COMPUTER-01_metadata_file -metadata creation_time="2023-12-25 13:25:29" -threads 0 -c:v copy -c:a copy -movflags +faststart -f segment -segment_time 01:00:00 -segment_atclocktime 1 -reset_timestamps 1 -strftime 1 -segment_format mp4 -t 120 test_record_video_with_audio_%Y-%m-%d_%H-%M-%S.mp4


  &#xA;

  The ffmpeg started and finished successfully, but when I open the recorded video with mpv like (mpv test_record_video_with_audio.mp4), I realized that there is a 5-6 seconds delay in audio. How can I prevent the delay of audio in the recorded mp4 file without using offset ? My last option is setting offset but I think that it is not safe according to any changes in network or etc.

  &#xA;

  FFMPEG version on both computer :

  &#xA;

  ffmpeg version 4.2.9 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 8 (GCC)


  &#xA;

  BROADCASTER COMPUTER :

  &#xA;

  sysctl.conf :

  &#xA;

  No added configurations.


  &#xA;

  ethtool output :

  &#xA;

  Supported ports: [ TP ]
Supported link modes:   100baseT/Full
                        1000baseT/Full
                        10000baseT/Full
                        2500baseT/Full
                        5000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes:  100baseT/Full
                        1000baseT/Full
                        10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
    Current message level: 0x00000007 (7)
                           drv probe link
Link detected: yes


  &#xA;

  ffmpeg video stream :

  &#xA;

  ffmpeg -fflags +genpts -f x11grab -framerate 30 -video_size uhd2160 -i :0 -c:v hevc_nvenc -preset fast -pix_fmt bgr0 -b:v 3M -g 25 -an -f mpegts udp://225.0.5.11:1026


  &#xA;

  ffmpeg audio stream :

  &#xA;

  ffmpeg -f alsa -i hw:0,0 -c:a aac -ar 48000 -b:a 1024K -ab 512k -f rtp_mpegts rtp://225.0.5.11:1032


  &#xA;

  nvidia-smi :

  &#xA;

  | NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA T400 4GB                Off | 00000000:5B:00.0 Off |                  N/A |
| 38%   38C    P8              N/A /  31W |    207MiB /  4096MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A4000               Off | 00000000:9E:00.0 Off |                  Off |
| 41%   59C    P2              41W / 140W |    766MiB / 16376MiB |     17%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      3227      G   /usr/libexec/Xorg                           114MiB |
|    0   N/A  N/A      3423      G   /usr/bin/gnome-shell                         87MiB |
|    1   N/A  N/A      3227      G   /usr/libexec/Xorg                           285MiB |
|    1   N/A  N/A      3423      G   /usr/bin/gnome-shell                         91MiB |
|    1   N/A  N/A      3762      C   ffmpeg                                      372MiB |
+---------------------------------------------------------------------------------------+


  &#xA;

  lscpu output :

  &#xA;

  

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              96
On-line CPU(s) list: 0-95
Thread(s) per core:  2
Core(s) per socket:  24
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
BIOS Vendor ID:      Intel(R) Corporation
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz
BIOS Model name:     Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz
Stepping:            7
CPU MHz:             2200.000
CPU max MHz:         4000.0000
CPU min MHz:         1000.0000
BogoMIPS:            4400.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            36608K
NUMA node0 CPU(s):   0-23,48-71
NUMA node1 CPU(s):   24-47,72-95


  &#xA;

  OS : CentOS Stream release 8


  &#xA;

  RECORDER COMPUTER :

  &#xA;

  sysctl.conf :

  &#xA;

  net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem= 4096 87380 16777216
net.ipv4.tcp_wmem= 4096 65536 16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 0
net.core.netdev_max_backlog = 50000
net.core.optmem_max=25165824


  &#xA;

  lscpu output :

  &#xA;

  Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              96
On-line CPU(s) list: 0-95
Thread(s) per core:  2
Core(s) per socket:  24
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
BIOS Vendor ID:      Intel
CPU family:          6
Model:               106
Model name:          Intel(R) Xeon(R) Gold 5318Y CPU @ 2.10GHz
BIOS Model name:     Intel(R) Xeon(R) Gold 5318Y CPU @ 2.10GHz
Stepping:            6
CPU MHz:             3400.000
CPU max MHz:         3400.0000
CPU min MHz:         800.0000
BogoMIPS:            4200.00
Virtualization:      VT-x
L1d cache:           48K
L1i cache:           32K
L2 cache:            1280K
L3 cache:            36864K


  &#xA;

  ethtool output :

  &#xA;

  Supported ports: [ TP ]
    Supported link modes:   1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseT/Full
                            10000baseT/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 10000Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Twisted Pair
    PHYAD: 12
    Transceiver: internal
    MDI-X: Unknown
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00002081 (8321)
                               drv tx_err hw
    Link detected: yes


  &#xA;

  No NVIDIA Graphic Driver

  &#xA;

  OS : CentOS Stream release 8


  &#xA;

  I tried audio encoding while recording like :

  &#xA;

  "-c:a", "aac",  
"-ar", "48000", 
"-b:a", "128k",


  &#xA;

  I also tried :

  &#xA;

  "aresample=async=1"



  &#xA;

  Unfortunately,these did not have any improvements on preventing latency in audio.

  &#xA;