Recherche avancée

Sur d’autres sites (4846)

• Problems decoding a gstreamer pipeline into images using node.js

  18 septembre 2023, par JK2018

  I have this Gstreamer pipeline that launches a test video , encodes it in h264 and sends it via udp on localhost 5000.

  


  gst-launch-1.0 -v videotestsrc ! videoconvert ! x264enc tune=zerolatency ! h264parse ! queue ! rtph264pay ! udpsink host=127.0.0.1 port=5000


  


  I run this pipeline in my terminal.

  


  Then I have a minimalistic node.js server that is suposed to receive the udp stream coming from the gstreamer pipeline, decode the video into images.
Finally I emit each image (not a fragment of an image) over a socket.

  


  I have tried several approaches unsuccessfully.
My first attempt was using a node gstreamer library and use gstreamer to decode the udp and re encode as images.

  


  My second attempt was using ffmpeg library.
Here is the code below :

  


  const express = require("express");
const http = require("http");
const { Server } = require("socket.io");
const socketIO = require("socket.io");
const cors = require("cors");
const app = express();
const server = http.createServer(app);
const dgram = require("dgram");
const ffmpeg = require("fluent-ffmpeg");

const io = socketIO(server, {
  cors: {
    origin: "http://localhost:3001",
    methods: ["GET", "POST"],
  },
});

app.use(
  cors({
    origin: "http://localhost:3001",
    methods: ["GET", "POST"],
  })
);

const udpServer = dgram.createSocket("udp4");

io.on("connection", (socket) => {
  console.log("A client connected");

  socket.on("disconnect", () => {
    console.log("A client disconnected");
  });
});

// Function to decode an H.264 video frame
function decodeH264Video(inputData, callback) {
  const command = ffmpeg()
    .input(inputData)
    .inputFormat("h264")
    .inputOptions(["-c:v h264"])
    .toFormat("image2")
    .on("end", () => {
      console.log("Decoding complete");
    })
    .on("error", (err) => {
      console.error("Error decoding video:", err);
    })
    .pipe();

  callback(command);
}

// Function to convert a decoded video frame to an image (JPEG format)
function convertVideoFrameToImage(decodedData, callback) {
  const imageStream = decodedData.pipe();
  const imageBuffer = [];

  imageStream.on("data", (chunk) => {
    imageBuffer.push(chunk);
  });

  imageStream.on("end", () => {
    const imageData = Buffer.concat(imageBuffer);
    callback(imageData);
  });
}

udpServer.on("message", (message) => {
  // Decode the UDP packet containing H.264 encoded video
  decodeH264Video(message, (decodedVideo) => {
    // Process the decoded video frame and convert it to an image
    convertVideoFrameToImage(decodedVideo, (imageData) => {
      // Send the image data to connected clients
      io.sockets.emit("image", { imageData: imageData.toString("base64") });
    });
  });
});

udpServer.bind(5000);

server.listen(3000, () => {
  console.log("Server is running on port 3000");
});


  


  Any help is more than welcome

  


• ts video via UDP from ffmpeg to gstreamer [closed]

  23 janvier 2024, par aron.h

  Hardware : Jetson AGX ORIN
Software : Jetpack 5.0.2

  


  I have been attempting to send a video file locally via UDP using ffmpeg :

  


  ffmpeg -stream_loop -1 -re -i test.ts -map 0 -c copy -preset ultrafast -f mpegts "udp://127.0.0.1:5000"

  


  And receiving the same stream via UDP using gstreamer :

  


  gst-launch-1.0 udpsrc port=5000 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=H264 ! rtph264depay ! decodebin ! videoconvert ! aasink

  


  But I get an error on the receiving gstreamer end :

  


  /GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0: Could not decode stream.
Additional debug info:
gstrtpbasedepayload.c(505): gst_rtp_base_depayload_handle_buffer (): 
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0:
Received invalid RTP payload, dropping
ERROR: from element /GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0:
The stream is in the wrong format.
Additional debug info:
gstrtph264depay.c(1298): gst_rtp_h264_depay_process ():
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0:
NAL unit type 27 not supported yet


  


  More detailed information on the video file :

  


  Original ID: 1002
Codec: H264 - MPEG-4 AVC (part 10) (h264)
Type: Video
Video resolution: 1920x1080
Buffer dimensions: 1920x1088
Frame rate: 30
Decoded format: 
Orientation: Top left
Chroma location: left


  


  When I listen with the command gst-launch-1.0 -v udpsrc port=5000 ! fakesink dump=1, it is quite apparent that the packets from FFMPEG are being received.
I am not sure why gstreamer's rtph264depay says the stream is in the wrong format.

  


  Would I have to check some details on the FFMPEG side ?
This is what information FFMPEG shows by default while running.

  


  Input #0, mpegts, from 'test.ts':
  Duration: 00:00:57.36, start: 20902.827056, bitrate: 2504 kb/s
  Program 1
    Stream #0:0[0x3ea]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
Output #0, mpegts, to 'udp://127.0.0.1:5000':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 29.97 fps, 29.97 tbr, 90k tbn, 90k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame=  611 fps= 30 q=-1.0 Lsize=    5847kB time=00:00:20.62 bitrate=2323.0kbits/s speed=   1x
video:5350kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 9.301388%


  


  Any advice would be appreciated.

  


• FFmpeg - ducking music for voiceover (sidechain)

  7 décembre 2023, par kaushal

  I am trying to duck the background music from a voiceover input file using FFmpeg (I tried doing this using pydub with no success).

  


  The below command is predominantly working, except that it is truncating my background music file (main.mp3, over 3 min long) to the length of the voiceover file (taekwondo.mp3, about a min long). Any idea or suggestion on what can I change in below command so that the output file is full 3 min long, with background music ducked when the voiceover is playing. Once the voiceover completes, background music continues to play as normal.

  


  In below complex filter, this is what I'm doing :
adelay filter : to start voiceover playing after 5 seconds into background music
volume filter : Since my background music is too loud and voiceover is too low, I'm increasing voiceover volume by 1.5 times, and lowering background music volume to 0.7 times
sidechaincompress filter : Then I'm applying sidechain filter to duck background music whenever voiceover is playing, with usual threshold, ratio & release filters

  


  I've tried both amix & amerge, but results in same behaviour.

  


  ffmpeg -i main.mp3 -i taekwondo.mp3 -filter_complex "[1:a]adelay=5000|5000,volume=1.5[a];[0:a]volume=0.7[b];[a]asplit=2[sc][mix];[b][sc]sidechaincompress=threshold=0.05:ratio=20:level_sc=1:release=500:attack=1[compr];[compr][mix]amix" output.mp3