Recherche avancée

Sur d’autres sites (2677)

• Audio and Video Recording Issues with FFMPEG [closed]

  7 octobre 2023, par Austin Althouse

  I am trying to use FFMPEG to stream a specific window on my computer. While the video works flawlessly, adding the audio is being problematic. Whenever I had audio (using -f dshow), the video's fps dramatically drops to 3 to 5 fps and a ton of dropped frames occur. Here is the command I am using : ffmpeg -f gdigrab -draw_mouse 0 -i title="MainWindow" -f dshow -i audio="CABLE Output (VB-Audio Virtual Cable)" -c:v mpeg2video -b:v 4000k -c:a aac -b:a 64k -f mpegts udp ://localhost:1234

  


  The audio doesn't seem clear also.

  


  What is going on and how can I fix it ?

  


  I tried different commands with no success.

  


• ffmpeg command for faster encoding at a decent bitrate with smaller file size

  28 juillet 2023, par Blair Holmes

  I'm currently using an implementation of ffmpeg on my android app. I'm allowing users to take short videos within my app and then when they upload them to the server, I'm crunching them down with ffmpeg to decrease file size so they're not passing huge amounts of data over the wire.

  



  the problem is, it's taking forever to encode the videos on the android device. These videos usually aren't longer than 45 seconds and can take 20 minutes to encode. I've done some playing around with different switches / parameters on the ffmpeg command line and I now have it at a more comfortable time, but the file sizes are significantly bigger. I'm just not sure what codec I should use (fastest encoding but with decent quality output), how ffmpeg handles changing size (aspect ratio) in terms of speed of encoding etc.

  



  Here are the two commands I've been using. This first one outputs the file size / quality that I want, but it just takes way too long to encode, not to mention it makes my device get really hot while it's encoding for so long :

  



  ffmpeg -i input.mp4 -b:v 1024k -c:a copy -vf scale=960:540 output.mp4


  



  I've tweaked the bitrate some on that as well as changed the scale to a smaller size, but I don't want to have to make the videos really small (in terms of scale) to accomplish a faster encoding. this second command goes a lot faster but makes the file size significantly bigger :

  



  ffmpeg -i input.mp4 -vcodec libx264 -preset fast -c:a copy -s 960x540 output.mp4


  



  I'd like to find a happy medium (smaller file size but faster encoding) keeping the video scale size close to the original. ffmpeg just has so many different parameters / switches that it's difficult to wrap my head around what I should be doing.

  



  EDIT : adding ffmpeg output.

  



  ffmpeg -i input.mp4 -vcodec libx264 -crf 30 -preset veryfast -c:a copy -s 960x540 output.mp4

  



      09-13 11:06:28.330 10881-10881/someapp D/home: ffmpeg version n3.0.1 Copyright (c) 2000-2016 the FFmpeg developers
09-13 11:06:28.330 10881-10881/someapp D/home:   built with gcc 4.8 (GCC)
09-13 11:06:28.331 10881-10881/someapp D/home:   configuration: --target-os=linux --cross-prefix=/home/vagrant/SourceCode/ffmpeg-android/toolchain-android/bin/arm-linux-androideabi- --arch=arm --cpu=cortex-a8 --enable-runtime-cpudetect --sysroot=/home/vagrant/SourceCode/ffmpeg-android/toolchain-android/sysroot --enable-pic --enable-libx264 --enable-libass --enable-libfreetype --enable-libfribidi --enable-libmp3lame --enable-fontconfig --enable-pthreads --disable-debug --disable-ffserver --enable-version3 --enable-hardcoded-tables --disable-ffplay --disable-ffprobe --enable-gpl --enable-yasm --disable-doc --disable-shared --enable-static --pkg-config=/home/vagrant/SourceCode/ffmpeg-android/ffmpeg-pkg-config --prefix=/home/vagrant/SourceCode/ffmpeg-android/build/armeabi-v7a --extra-cflags='-I/home/vagrant/SourceCode/ffmpeg-android/toolchain-android/include -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fno-strict-overflow -fstack-protector-all' --extra-ldflags='-L/home/vagrant/SourceCode/ffmpeg-android/toolchain-android/lib -Wl,-z,relro -Wl,-z,now -pie' --extra-libs='-lpng -lexpat -lm' --extra-cxxflags=
09-13 11:06:28.331 10881-10881/someapp D/home:   libavutil      55. 17.103 / 55. 17.103
09-13 11:06:28.331 10881-10881/someapp D/home:   libavcodec     57. 24.102 / 57. 24.102
09-13 11:06:28.331 10881-10881/someapp D/home:   libavformat    57. 25.100 / 57. 25.100
09-13 11:06:28.331 10881-10881/someapp D/home:   libavdevice    57.  0.101 / 57.  0.101
09-13 11:06:28.331 10881-10881/someapp D/home:   libavfilter     6. 31.100 /  6. 31.100
09-13 11:06:28.331 10881-10881/someapp D/home:   libswscale      4.  0.100 /  4.  0.100
09-13 11:06:28.331 10881-10881/someapp D/home:   libswresample   2.  0.101 /  2.  0.101
09-13 11:06:28.331 10881-10881/someapp D/home:   libpostproc    54.  0.100 / 54.  0.100
09-13 11:06:28.430 10881-10881/someapp D/home: Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/storage/emulated/0/ExpeditionSpot/Videos/20160913110411.mp4':
09-13 11:06:28.430 10881-10881/someapp D/home:   Metadata:
09-13 11:06:28.430 10881-10881/someapp D/home:     major_brand     : mp42
09-13 11:06:28.430 10881-10881/someapp D/home:     minor_version   : 0
09-13 11:06:28.430 10881-10881/someapp D/home:     compatible_brands: isommp42
09-13 11:06:28.430 10881-10881/someapp D/home:     creation_time   : 2016-09-13 17:04:33
09-13 11:06:28.430 10881-10881/someapp D/home:     com.android.version: 6.0.1
09-13 11:06:28.430 10881-10881/someapp D/home:   Duration: 00:00:19.41, start: 0.000000, bitrate: 20222 kb/s
09-13 11:06:28.430 10881-10881/someapp D/home:     Stream #0:0(eng): Video: h264 (Baseline) (avc1 / 0x31637661), yuv420p, 1920x1080, 19963 kb/s, SAR 1:1 DAR 16:9, 30.03 fps, 30 tbr, 90k tbn, 180k tbc (default)
09-13 11:06:28.430 10881-10881/someapp D/home:     Metadata:
09-13 11:06:28.430 10881-10881/someapp D/home:       creation_time   : 2016-09-13 17:04:33
09-13 11:06:28.430 10881-10881/someapp D/home:       handler_name    : VideoHandle
09-13 11:06:28.431 10881-10881/someapp D/home:     Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 96 kb/s (default)
09-13 11:06:28.431 10881-10881/someapp D/home:     Metadata:
09-13 11:06:28.431 10881-10881/someapp D/home:       creation_time   : 2016-09-13 17:04:33
09-13 11:06:28.431 10881-10881/someapp D/home:       handler_name    : SoundHandle
09-13 11:06:28.448 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] using SAR=1/1
09-13 11:06:28.448 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] using cpu capabilities: none!
09-13 11:06:28.516 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] profile High, level 3.1
09-13 11:06:28.516 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] 264 - core 148 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=1 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=2 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=12 lookahead_threads=4 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=1 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=10 rc=crf mbtree=1 crf=27.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
09-13 11:06:28.532 10881-10881/someapp D/home: Output #0, mp4, to '/storage/emulated/0/ExpeditionSpot/.tmp/small-20160913110411.mp4':
09-13 11:06:28.532 10881-10881/someapp D/home:   Metadata:
09-13 11:06:28.532 10881-10881/someapp D/home:     major_brand     : mp42
09-13 11:06:28.533 10881-10881/someapp D/home:     minor_version   : 0
09-13 11:06:28.533 10881-10881/someapp D/home:     compatible_brands: isommp42
09-13 11:06:28.533 10881-10881/someapp D/home:     com.android.version: 6.0.1
09-13 11:06:28.533 10881-10881/someapp D/home:     encoder         : Lavf57.25.100
09-13 11:06:28.533 10881-10881/someapp D/home:     Stream #0:0(eng): Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 960x540 [SAR 1:1 DAR 16:9], q=-1--1, 30 fps, 15360 tbn, 30 tbc (default)
09-13 11:06:28.533 10881-10881/someapp D/home:     Metadata:
09-13 11:06:28.533 10881-10881/someapp D/home:       creation_time   : 2016-09-13 17:04:33
09-13 11:06:28.533 10881-10881/someapp D/home:       handler_name    : VideoHandle
09-13 11:06:28.533 10881-10881/someapp D/home:       encoder         : Lavc57.24.102 libx264
09-13 11:06:28.533 10881-10881/someapp D/home:     Side data:
09-13 11:06:28.533 10881-10881/someapp D/home:       unknown side data type 10 (24 bytes)
09-13 11:06:28.533 10881-10881/someapp D/home:     Stream #0:1(eng): Audio: aac (LC) ([64][0][0][0] / 0x0040), 48000 Hz, stereo, 96 kb/s (default)
09-13 11:06:28.533 10881-10881/someapp D/home:     Metadata:
09-13 11:06:28.533 10881-10881/someapp D/home:       creation_time   : 2016-09-13 17:04:33
09-13 11:06:28.533 10881-10881/someapp D/home:       handler_name    : SoundHandle
09-13 11:06:28.533 10881-10881/someapp D/home: Stream mapping:
09-13 11:06:28.533 10881-10881/someapp D/home:   Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
09-13 11:06:28.533 10881-10881/someapp D/home:   Stream #0:1 -> #0:1 (copy)
09-13 11:06:28.533 10881-10881/someapp D/home: Press [q] to stop, [?] for help
09-13 11:06:29.102 10881-10881/someapp D/home: frame=    7 fps=0.0 q=0.0 size=       0kB time=00:00:01.04 bitrate=   0.4kbits/s speed=2.08x    
09-13 11:06:29.699 10881-10881/someapp D/home: frame=   16 fps= 15 q=0.0 size=       0kB time=00:00:01.04 bitrate=   0.4kbits/s speed=0.998x    
....
09-13 11:07:12.674 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] frame I:3     Avg QP:26.83  size: 21896
09-13 11:07:12.674 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] frame P:279   Avg QP:28.76  size:  5859
09-13 11:07:12.674 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] frame B:296   Avg QP:29.93  size:   863
09-13 11:07:12.674 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] consecutive B-frames: 18.3% 32.2% 23.9% 25.6%
09-13 11:07:12.674 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] mb I  I16..4: 16.9% 54.2% 28.9%
09-13 11:07:12.674 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] mb P  I16..4:  7.7%  9.1%  0.4%  P16..4: 27.5% 11.1%  4.0%  0.0%  0.0%    skip:40.1%
09-13 11:07:12.674 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] mb B  I16..4:  0.8%  0.6%  0.0%  B16..8: 10.9%  2.4%  0.1%  direct: 1.8%  skip:83.5%  L0:35.8% L1:54.2% BI:10.0%
09-13 11:07:12.675 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] 8x8 transform intra:52.0% inter:41.3%
09-13 11:07:12.675 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] coded y,uvDC,uvAC intra: 28.9% 26.0% 2.1% inter: 6.4% 5.0% 0.0%
09-13 11:07:12.675 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] i16 v,h,dc,p: 57% 20% 17%  6%
09-13 11:07:12.675 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 33% 23% 34%  1%  2%  1%  3%  2%  2%
09-13 11:07:12.675 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 36% 21% 17%  2%  7%  4%  6%  4%  4%
09-13 11:07:12.675 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] i8c dc,h,v,p: 59% 16% 23%  2%
09-13 11:07:12.675 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] Weighted P-Frames: Y:16.8% UV:4.7%
09-13 11:07:12.675 10881-10881/someapp D/home: [libx264 @ 0xf71c4400] kb/s:812.09


  


• What's the most desireable way to capture system display and audio in the form of individual encoded audio and video packets in go (language) ? [closed]

  11 janvier 2023, par Tiger Yang

  Question (read the context below first) :

  


  For those of you familiar with the capabilities of go, Is there a better way to go about all this ? Since ffmpeg is so ubiquitous, I'm sure it's been optomized to perfection, but what's the best way to capture system display and audio in the form of individual encoded audio and video packets in go (language), so that they can be then sent via webtransport-go ? I wish for it to prioritize efficiency and low latency, and ideally capture and encode the framebuffer directly like ffmpeg does.

  


  Thanks ! I have many other questions about this, but I think it's best to ask as I go.

  


  Context and what I've done so far :

  


  I'm writing a remote desktop software for my personal use because of grievances with current solutions out there. At the moment, it consists of a web app that uses the webtransport API to send input datagrams and receive AV packets on two dedicated unidirectional streams, and the webcodecs API to decode these packets. On the serverside, I originally planned to use python with the aioquic library as a webtransport server. Upon connection and authentication, the server would start ffmpeg as a subprocess with this command :

  


  ffmpeg -init_hw_device d3d11va -filter_complex ddagrab=video_size=1920x1080:framerate=60 -vcodec hevc_nvenc -tune ll -preset p7 -spatial_aq 1 -temporal_aq 1 -forced-idr 1 -rc cbr -b:v 400K -no-scenecut 1 -g 216000 -f hevc -

  


  What I really appreciate about this is that it uses windows' desktop duplication API to copy the framebuffer of my GPU and hand that directly to the on-die hardware encoder with zero round trips to the CPU. I think it's about as efficient and elegant a solution as I can manage. It then outputs the encoded stream to the stdout, which python can read and send to the client.

  


  As for the audio, there is another ffmpeg instance :

  


  ffmpeg -f dshow -channels 2 -sample_rate 48000 -sample_size 16 -audio_buffer_size 15 -i audio="RD Audio (High Definition Audio Device)" -acodec libopus -vbr on -application audio -mapping_family 0 -apply_phase_inv true -b:a 25K -fec false -packet_loss 0 -map 0 -f data -

  


  which listens to a physical loopback interface, which is literally just a short wire bridging the front panel headphone and microphone jacks (I'm aware of the quality loss of converting to analog and back, but the audio is then crushed down to 25kbps so it's fine) ()

  


  Unfortunately, aioquic was not easy to work with IMO, and I found webtransport-go https://github.com/adriancable/webtransport-go, which was a hell of a lot better in both simplicity and documentation. However, now I'm dealing with a whole new language, and I wanna ask : (above)

  


  EDIT : Here's the code for my server so far :

  


  


  


  package main

import (
    "bytes"
    "context"
    "fmt"
    "log"
    "net/http"
    "os/exec"
    "time"

    "github.com/adriancable/webtransport-go"
)

func warn(str string) {
    fmt.Printf("\n===== WARNING ===================================================================================================\n   %s\n=================================================================================================================\n", str)
}

func main() {

    password := []byte("abc")

    videoString := []string{
        "ffmpeg",
        "-init_hw_device", "d3d11va",
        "-filter_complex", "ddagrab=video_size=1920x1080:framerate=60",
        "-vcodec", "hevc_nvenc",
        "-tune", "ll",
        "-preset", "p7",
        "-spatial_aq", "1",
        "-temporal_aq", "1",
        "-forced-idr", "1",
        "-rc", "cbr",
        "-b:v", "500K",
        "-no-scenecut", "1",
        "-g", "216000",
        "-f", "hevc", "-",
    }

    audioString := []string{
        "ffmpeg",
        "-f", "dshow",
        "-channels", "2",
        "-sample_rate", "48000",
        "-sample_size", "16",
        "-audio_buffer_size", "15",
        "-i", "audio=RD Audio (High Definition Audio Device)",
        "-acodec", "libopus",
        "-mapping_family", "0",
        "-b:a", "25K",
        "-map", "0",
        "-f", "data", "-",
    }

    connected := false

    http.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
        session := request.Body.(*webtransport.Session)

        session.AcceptSession()
        fmt.Println("\nAccepted incoming WebTransport connection.")
        fmt.Println("Awaiting authentication...")

        authData, err := session.ReceiveMessage(session.Context()) // Waits here till first datagram
        if err != nil {                                            // if client closes connection before sending anything
            fmt.Println("\nConnection closed:", err)
            return
        }

        if len(authData) >= 2 && bytes.Equal(authData[2:], password) {
            if connected {
                session.CloseSession()
                warn("Client has authenticated, but a session is already taking place! Connection closed.")
                return
            } else {
                connected = true
                fmt.Println("Client has authenticated!\n")
            }
        } else {
            session.CloseSession()
            warn("Client has failed authentication! Connection closed. (" + string(authData[2:]) + ")")
            return
        }

        videoStream, _ := session.OpenUniStreamSync(session.Context())

        videoCmd := exec.Command(videoString[0], videoString[1:]...)
        go func() {
            videoOut, _ := videoCmd.StdoutPipe()
            videoCmd.Start()

            buffer := make([]byte, 15000)
            for {
                len, err := videoOut.Read(buffer)
                if err != nil {
                    break
                }
                if len > 0 {
                    videoStream.Write(buffer[:len])
                }
            }
        }()

        time.Sleep(50 * time.Millisecond)

        audioStream, err := session.OpenUniStreamSync(session.Context())

        audioCmd := exec.Command(audioString[0], audioString[1:]...)
        go func() {
            audioOut, _ := audioCmd.StdoutPipe()
            audioCmd.Start()

            buffer := make([]byte, 15000)
            for {
                len, err := audioOut.Read(buffer)
                if err != nil {
                    break
                }
                if len > 0 {
                    audioStream.Write(buffer[:len])
                }
            }
        }()

        for {
            data, err := session.ReceiveMessage(session.Context())
            if err != nil {
                videoCmd.Process.Kill()
                audioCmd.Process.Kill()

                connected = false

                fmt.Println("\nConnection closed:", err)
                break
            }

            if len(data) == 0 {

            } else if data[0] == byte(0) {
                fmt.Printf("Received mouse datagram: %s\n", data)
            }
        }

    })

    server := &webtransport.Server{
        ListenAddr: ":1024",
        TLSCert:    webtransport.CertFile{Path: "SSL/fullchain.pem"},
        TLSKey:     webtransport.CertFile{Path: "SSL/privkey.pem"},
        QuicConfig: &webtransport.QuicConfig{
            KeepAlive:      false,
            MaxIdleTimeout: 3 * time.Second,
        },
    }

    fmt.Println("Launching WebTransport server at", server.ListenAddr)
    ctx, cancel := context.WithCancel(context.Background())
    if err := server.Run(ctx); err != nil {
        log.Fatal(err)
        cancel()
    }

}