Recherche avancée
Autres articles (20)
-
Contribute to translation
13 avril 2011You can help us to improve the language used in the software interface to make MediaSPIP more accessible and user-friendly. You can also translate the interface into any language that allows it to spread to new linguistic communities.
To do this, we use the translation interface of SPIP where the all the language modules of MediaSPIP are available. Just subscribe to the mailing list and request further informantion on translation.
MediaSPIP is currently available in French and English (...) -
List of compatible distributions
26 avril 2011, parThe table below is the list of Linux distributions compatible with the automated installation script of MediaSPIP. Distribution nameVersion nameVersion number Debian Squeeze 6.x.x Debian Weezy 7.x.x Debian Jessie 8.x.x Ubuntu The Precise Pangolin 12.04 LTS Ubuntu The Trusty Tahr 14.04
If you want to help us improve this list, you can provide us access to a machine whose distribution is not mentioned above or send the necessary fixes to add (...) -
Submit bugs and patches
13 avril 2011Unfortunately a software is never perfect.
If you think you have found a bug, report it using our ticket system. Please to help us to fix it by providing the following information : the browser you are using, including the exact version as precise an explanation as possible of the problem if possible, the steps taken resulting in the problem a link to the site / page in question
If you think you have solved the bug, fill in a ticket and attach to it a corrective patch.
You may also (...)
Sur d’autres sites (5105)
-
Encoding/Decoding H264 using libav in C++ [closed]
20 mai, par gbock93I want to build an application to


- 

- capture frames in YUYV 4:2:2 format
- encode them to H264
- send over network
- decode the received data
- display the video stream












To do so I wrote 2 classes, H264Encoder and H264Decoder.


I post only the .cpp contents, the .h are trivial :


H264Encoder.cpp


#include 

#include <stdexcept>
#include <iostream>

H264Encoder::H264Encoder(unsigned int width_, unsigned int height_, unsigned int fps_):
 m_width(width_),
 m_height(height_),
 m_fps(fps_),
 m_frame_index(0),
 m_context(nullptr),
 m_frame(nullptr),
 m_packet(nullptr),
 m_sws_ctx(nullptr)
{
 // Find the video codec
 AVCodec* codec;
 codec = avcodec_find_encoder(AV_CODEC_ID_H264);
 if (!codec)
 throw std::runtime_error("[Encoder]: Error: Codec not found");

 // Allocate codec
 m_context = avcodec_alloc_context3(codec);
 if (!m_context)
 throw std::runtime_error("[Encoder]: Error: Could not allocate codec context");

 // Configure codec
 av_opt_set(m_context->priv_data, "preset", "ultrafast", 0);
 av_opt_set(m_context->priv_data, "tune", "zerolatency", 0);
 av_opt_set(m_context->priv_data, "crf", "35", 0); // Range: [0; 51], sane range: [18; 26], lower -> higher compression

 m_context->width = (int)width_;
 m_context->height = (int)height_;
 m_context->time_base = {1, (int)fps_};
 m_context->framerate = {(int)fps_, 1};
 m_context->codec_id = AV_CODEC_ID_H264;
 m_context->pix_fmt = AV_PIX_FMT_YUV420P; // H265|4 codec take as input only AV_PIX_FMT_YUV420P
 m_context->bit_rate = 400000;
 m_context->gop_size = 10;
 m_context->max_b_frames = 1;

 // Open codec
 if (avcodec_open2(m_context, codec, nullptr) < 0)
 throw std::runtime_error("[Encoder]: Error: Could not open codec");

 // Allocate frame and its buffer
 m_frame = av_frame_alloc();
 if (!m_frame) 
 throw std::runtime_error("[Encoder]: Error: Could not allocate frame");

 m_frame->format = m_context->pix_fmt;
 m_frame->width = m_context->width;
 m_frame->height = m_context->height;

 if (av_frame_get_buffer(m_frame, 0) < 0)
 throw std::runtime_error("[Encoder]: Error: Cannot allocate frame buffer");
 
 // Allocate packet
 m_packet = av_packet_alloc();
 if (!m_packet) 
 throw std::runtime_error("[Encoder]: Error: Could not allocate packet");

 // Convert from YUYV422 to YUV420P
 m_sws_ctx = sws_getContext(
 width_, height_, AV_PIX_FMT_YUYV422,
 width_, height_, AV_PIX_FMT_YUV420P,
 SWS_BILINEAR, nullptr, nullptr, nullptr
 );
 if (!m_sws_ctx) 
 throw std::runtime_error("[Encoder]: Error: Could not allocate sws context");

 //
 printf("[Encoder]: H264Encoder ready.\n");
}

H264Encoder::~H264Encoder()
{
 sws_freeContext(m_sws_ctx);
 av_packet_free(&m_packet);
 av_frame_free(&m_frame);
 avcodec_free_context(&m_context);

 printf("[Encoder]: H264Encoder destroyed.\n");
}

std::vector H264Encoder::encode(const cv::Mat& img_)
{
 /*
 - YUYV422 is a packed format. It has 3 components (av_pix_fmt_desc_get((AVPixelFormat)AV_PIX_FMT_YUYV422)->nb_components == 3) but
 data is stored in a single plane (av_pix_fmt_count_planes((AVPixelFormat)AV_PIX_FMT_YUYV422) == 1).
 - YUV420P is a planar format. It has 3 components (av_pix_fmt_desc_get((AVPixelFormat)AV_PIX_FMT_YUV420P)->nb_components == 3) and
 each component is stored in a separate plane (av_pix_fmt_count_planes((AVPixelFormat)AV_PIX_FMT_YUV420P) == 3) with its
 own stride.
 */
 std::cout << "[Encoder]" << std::endl;
 std::cout << "[Encoder]: Encoding img " << img_.cols << "x" << img_.rows << " | element size " << img_.elemSize() << std::endl;
 assert(img_.elemSize() == 2);

 uint8_t* input_data[1] = {(uint8_t*)img_.data};
 int input_linesize[1] = {2 * (int)m_width};
 
 if (av_frame_make_writable(m_frame) < 0)
 throw std::runtime_error("[Encoder]: Error: Cannot make frame data writable");

 // Convert from YUV422 image to YUV420 frame. Apply scaling if necessary
 sws_scale(
 m_sws_ctx,
 input_data, input_linesize, 0, m_height,
 m_frame->data, m_frame->linesize
 );
 m_frame->pts = m_frame_index;

 int n_planes = av_pix_fmt_count_planes((AVPixelFormat)m_frame->format);
 std::cout << "[Encoder]: Sending Frame " << m_frame_index << " with dimensions " << m_frame->width << "x" << m_frame->height << "x" << n_planes << std::endl;
 for (int i=0; iframerate.num) + 1;
 break;
 case AVERROR(EAGAIN):
 throw std::runtime_error("[Encoder]: avcodec_send_frame: EAGAIN");
 case AVERROR_EOF:
 throw std::runtime_error("[Encoder]: avcodec_send_frame: EOF");
 case AVERROR(EINVAL):
 throw std::runtime_error("[Encoder]: avcodec_send_frame: EINVAL");
 case AVERROR(ENOMEM):
 throw std::runtime_error("[Encoder]: avcodec_send_frame: ENOMEM");
 default:
 throw std::runtime_error("[Encoder]: avcodec_send_frame: UNKNOWN");
 }

 // Receive packet from codec
 std::vector result;
 while(ret >= 0)
 {
 ret = avcodec_receive_packet(m_context, m_packet);

 switch (ret)
 {
 case 0:
 std::cout << "[Encoder]: Received packet from codec of size " << m_packet->size << " bytes " << std::endl;
 result.insert(result.end(), m_packet->data, m_packet->data + m_packet->size);
 av_packet_unref(m_packet);
 break;

 case AVERROR(EAGAIN):
 std::cout << "[Encoder]: avcodec_receive_packet: EAGAIN" << std::endl;
 break;
 case AVERROR_EOF:
 std::cout << "[Encoder]: avcodec_receive_packet: EOF" << std::endl;
 break;
 case AVERROR(EINVAL):
 throw std::runtime_error("[Encoder]: avcodec_receive_packet: EINVAL");
 default:
 throw std::runtime_error("[Encoder]: avcodec_receive_packet: UNKNOWN");
 }
 }

 std::cout << "[Encoder]: Encoding complete" << std::endl;
 return result;
}
</iostream></stdexcept>

H264Decoder.cpp


#include 

#include <iostream>
#include <stdexcept>

H264Decoder::H264Decoder():
 m_context(nullptr),
 m_frame(nullptr),
 m_packet(nullptr)
{
 // Find the video codec
 AVCodec* codec;
 codec = avcodec_find_decoder(AV_CODEC_ID_H264);
 if (!codec)
 throw std::runtime_error("[Decoder]: Error: Codec not found");

 // Allocate codec
 m_context = avcodec_alloc_context3(codec);
 if (!m_context)
 throw std::runtime_error("[Decoder]: Error: Could not allocate codec context");

 // Open codec
 if (avcodec_open2(m_context, codec, nullptr) < 0)
 throw std::runtime_error("[Decoder]: Error: Could not open codec");

 // Allocate frame
 m_frame = av_frame_alloc();
 if (!m_frame)
 throw std::runtime_error("[Decoder]: Error: Could not allocate frame");

 // Allocate packet
 m_packet = av_packet_alloc();
 if (!m_packet) 
 throw std::runtime_error("[Decoder]: Error: Could not allocate packet");

 //
 printf("[Decoder]: H264Decoder ready.\n");
}

H264Decoder::~H264Decoder()
{
 av_packet_free(&m_packet);
 av_frame_free(&m_frame);
 avcodec_free_context(&m_context);

 printf("[Decoder]: H264Decoder destroyed.\n");
}

bool H264Decoder::decode(uint8_t* data_, size_t size_, cv::Mat& img_)
{
 std::cout << "[Decoder]" << std::endl;
 std::cout << "[Decoder]: decoding " << size_ << " bytes of data" << std::endl;

 // Fill packet
 m_packet->data = data_;
 m_packet->size = size_;

 if (size_ == 0)
 return false;

 // Send packet to codec
 int send_result = avcodec_send_packet(m_context, m_packet);

 switch (send_result)
 {
 case 0:
 std::cout << "[Decoder]: Sent packet to codec" << std::endl;
 break;
 case AVERROR(EAGAIN):
 throw std::runtime_error("[Decoder]: avcodec_send_packet: EAGAIN");
 case AVERROR_EOF:
 throw std::runtime_error("[Decoder]: avcodec_send_packet: EOF");
 case AVERROR(EINVAL):
 throw std::runtime_error("[Decoder]: avcodec_send_packet: EINVAL");
 case AVERROR(ENOMEM):
 throw std::runtime_error("[Decoder]: avcodec_send_packet: ENOMEM");
 default:
 throw std::runtime_error("[Decoder]: avcodec_send_packet: UNKNOWN");
 }

 // Receive frame from codec
 int n_planes;
 uint8_t* output_data[1];
 int output_line_size[1];

 int receive_result = avcodec_receive_frame(m_context, m_frame);

 switch (receive_result)
 {
 case 0:
 n_planes = av_pix_fmt_count_planes((AVPixelFormat)m_frame->format);
 std::cout << "[Decoder]: Received Frame with dimensions " << m_frame->width << "x" << m_frame->height << "x" << n_planes << std::endl;
 for (int i=0; i/
 std::cout << "[Decoder]: Decoding complete" << std::endl;
 return true;
}
</stdexcept></iostream>

To test the two classes I put together a main.cpp to grab a frame, encode/decode and display the decoded frame (no network transmission in place) :


main.cpp


while(...)
{
 // get frame from custom camera class. Format is YUYV 4:2:2
 camera.getFrame(camera_frame);
 // Construct a cv::Mat to represent the grabbed frame
 cv::Mat camera_frame_yuyv = cv::Mat(camera_frame.height, camera_frame.width, CV_8UC2, camera_frame.data.data());
 // Encode image
 std::vector encoded_data = encoder.encode(camera_frame_yuyv);
 if (!encoded_data.empty())
 {
 // Decode image
 cv::Mat decoded_frame;
 if (decoder.decode(encoded_data.data(), encoded_data.size(), decoded_frame))
 {
 // Display image
 cv::imshow("Camera", decoded_frame);
 cv::waitKey(1);
 }
 }
}


Compiling and executing the code I get random results between subsequent executions :


- 

- Sometimes the whole loop runs without problems and I see the decoded image.
- Sometimes the program crashes at the
sws_scale(...)call in the decoder with"Assertion desc failed at src/libswscale/swscale_internal.h:757". - Sometimes the loop runs but I see a black image and the message
Slice parameters 0, 720 are invalidis displayed when executing thesws_scale(...)call in the decoder.








Why is the behaviour so random ? What am I doing wrong with the libav API ?


Some resources I found useful :


- 

- This article on encoding
- This article on decoding






-
ffmpeg pipe process ends right after writing first buffer data to input stream and does not keep running
6 mai, par Taketo MatsunagaI have been trying to convert 16bit PCM (s16le) audio data to webm using ffmpeg in C#.
But the process ends right after the writing the first buffer data to standard input.
I has exited with the status 0, meaning success. But do not know why....
Could anyone tell me why ?


I apprecite it if you could support me.


public class SpeechService : ISpeechService
 {
 
 /// <summary>
 /// Defines the _audioInputStream
 /// </summary>
 private readonly MemoryStream _audioInputStream = new MemoryStream();

 public async Task SendPcmAsWebmViaWebSocketAsync(
 MemoryStream pcmAudioStream,
 int sampleRate,
 int channels) 
 {
 string inputFormat = "s16le";

 var ffmpegProcessInfo = new ProcessStartInfo
 {
 FileName = _ffmpegPath,
 Arguments =
 $"-f {inputFormat} -ar {sampleRate} -ac {channels} -i pipe:0 " +
 $"-f webm pipe:1",
 RedirectStandardInput = true,
 RedirectStandardOutput = true,
 RedirectStandardError = true,
 UseShellExecute = false,
 CreateNoWindow = true,
 };

 _ffmpegProcess = new Process { StartInfo = ffmpegProcessInfo };

 Console.WriteLine("Starting FFmpeg process...");
 try
 {

 if (!await Task.Run(() => _ffmpegProcess.Start()))
 {
 Console.Error.WriteLine("Failed to start FFmpeg process.");
 return;
 }
 Console.WriteLine("FFmpeg process started.");

 }
 catch (Exception ex)
 {
 Console.Error.WriteLine($"Error starting FFmpeg process: {ex.Message}");
 throw;
 }

 var encodeAndSendTask = Task.Run(async () =>
 {
 try
 {
 using var ffmpegOutputStream = _ffmpegProcess.StandardOutput.BaseStream;
 byte[] buffer = new byte[8192]; // Temporary buffer to read data
 byte[] sendBuffer = new byte[8192]; // Buffer to accumulate data for sending
 int sendBufferIndex = 0; // Tracks the current size of sendBuffer
 int bytesRead;

 Console.WriteLine("Reading WebM output from FFmpeg and sending via WebSocket...");
 while (true)
 {
 if ((bytesRead = await ffmpegOutputStream.ReadAsync(buffer, 0, buffer.Length)) > 0)
 {
 // Copy data to sendBuffer
 Array.Copy(buffer, 0, sendBuffer, sendBufferIndex, bytesRead);
 sendBufferIndex += bytesRead;

 // If sendBuffer is full, send it via WebSocket
 if (sendBufferIndex >= sendBuffer.Length)
 {
 var segment = new ArraySegment<byte>(sendBuffer, 0, sendBuffer.Length);
 _ws.SendMessage(segment);
 sendBufferIndex = 0; // Reset the index after sending
 }
 }
 }
 }
 catch (OperationCanceledException)
 {
 Console.WriteLine("Encode/Send operation cancelled.");
 }
 catch (IOException ex) when (ex.InnerException is ObjectDisposedException)
 {
 Console.WriteLine("Stream was closed, likely due to process exit or cancellation.");
 }
 catch (Exception ex)
 {
 Console.Error.WriteLine($"Error during encoding/sending: {ex}");
 }
 });

 var errorReadTask = Task.Run(async () =>
 {
 Console.WriteLine("Starting to read FFmpeg stderr...");
 using var errorReader = _ffmpegProcess.StandardError;
 try
 {
 string? line;
 while ((line = await errorReader.ReadLineAsync()) != null) 
 {
 Console.WriteLine($"[FFmpeg stderr] {line}");
 }
 }
 catch (OperationCanceledException) { Console.WriteLine("FFmpeg stderr reading cancelled."); }
 catch (TimeoutException) { Console.WriteLine("FFmpeg stderr reading timed out (due to cancellation)."); }
 catch (Exception ex) { Console.Error.WriteLine($"Error reading FFmpeg stderr: {ex.Message}"); }
 Console.WriteLine("Finished reading FFmpeg stderr.");
 });

 }

 public async Task AppendAudioBuffer(AudioMediaBuffer audioBuffer)
 {
 try
 {
 // audio for a 1:1 call
 var bufferLength = audioBuffer.Length;
 if (bufferLength > 0)
 {
 var buffer = new byte[bufferLength];
 Marshal.Copy(audioBuffer.Data, buffer, 0, (int)bufferLength);

 _logger.Info("_ffmpegProcess.HasExited:" + _ffmpegProcess.HasExited);
 using var ffmpegInputStream = _ffmpegProcess.StandardInput.BaseStream;
 await ffmpegInputStream.WriteAsync(buffer, 0, buffer.Length);
 await ffmpegInputStream.FlushAsync(); // バッファをフラッシュ
 _logger.Info("Wrote buffer data.");

 }
 }
 catch (Exception e)
 {
 _logger.Error(e, "Exception happend writing to input stream");
 }
 }

</byte>

Starting FFmpeg process...
FFmpeg process started.
Starting to read FFmpeg stderr...
Reading WebM output from FFmpeg and sending via WebSocket...
[FFmpeg stderr] ffmpeg version 7.1.1-essentials_build-www.gyan.dev Copyright (c) 2000-2025 the FFmpeg developers
[FFmpeg stderr] built with gcc 14.2.0 (Rev1, Built by MSYS2 project)
[FFmpeg stderr] configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-zlib --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-sdl2 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-dxva2 --enable-d3d11va --enable-d3d12va --enable-ffnvcodec --enable-libvpl --enable-nvdec --enable-nvenc --enable-vaapi --enable-libgme --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libtheora --enable-libvo-amrwbenc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-librubberband
[FFmpeg stderr] libavutil 59. 39.100 / 59. 39.100
[FFmpeg stderr] libavcodec 61. 19.101 / 61. 19.101
[FFmpeg stderr] libavformat 61. 7.100 / 61. 7.100
[FFmpeg stderr] libavdevice 61. 3.100 / 61. 3.100
[FFmpeg stderr] libavfilter 10. 4.100 / 10. 4.100
[FFmpeg stderr] libswscale 8. 3.100 / 8. 3.100
[FFmpeg stderr] libswresample 5. 3.100 / 5. 3.100
[FFmpeg stderr] libpostproc 58. 3.100 / 58. 3.100

[2025-05-06 15:44:43,598][INFO][XbLogger.cs:85] _ffmpegProcess.HasExited:False
[2025-05-06 15:44:43,613][INFO][XbLogger.cs:85] Wrote buffer data.
[2025-05-06 15:44:43,613][INFO][XbLogger.cs:85] Wrote buffer data.
[FFmpeg stderr] [aist#0:0/pcm_s16le @ 0000025ec8d36040] Guessed Channel Layout: mono
[FFmpeg stderr] Input #0, s16le, from 'pipe:0':
[FFmpeg stderr] Duration: N/A, bitrate: 256 kb/s
[FFmpeg stderr] Stream #0:0: Audio: pcm_s16le, 16000 Hz, mono, s16, 256 kb/s
[FFmpeg stderr] Stream mapping:
[FFmpeg stderr] Stream #0:0 -> #0:0 (pcm_s16le (native) -> opus (libopus))
[FFmpeg stderr] [libopus @ 0000025ec8d317c0] No bit rate set. Defaulting to 64000 bps.
[FFmpeg stderr] Output #0, webm, to 'pipe:1':
[FFmpeg stderr] Metadata:
[FFmpeg stderr] encoder : Lavf61.7.100
[FFmpeg stderr] Stream #0:0: Audio: opus, 16000 Hz, mono, s16, 64 kb/s
[FFmpeg stderr] Metadata:
[FFmpeg stderr] encoder : Lavc61.19.101 libopus
[FFmpeg stderr] [out#0/webm @ 0000025ec8d36200] video:0KiB audio:1KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 67.493113%
[FFmpeg stderr] size= 1KiB time=00:00:00.04 bitrate= 243.2kbits/s speed=2.81x
Finished reading FFmpeg stderr.
[2025-05-06 15:44:44,101][INFO][XbLogger.cs:85] _ffmpegProcess.HasExited:True
[2025-05-06 15:44:44,132][ERROR][XbLogger.cs:67] Exception happend writing to input stream
System.ObjectDisposedException: Cannot access a closed file.
 at System.IO.FileStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
 at System.IO.Stream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count)
 at EchoBot.Media.SpeechService.AppendAudioBuffer(AudioMediaBuffer audioBuffer) in C:\Users\tm068\Documents\workspace\myprj\xbridge-teams-bot\src\EchoBot\Media\SpeechService.cs:line 242


I am expecting the ffmpeg process keep running.


-
Send image and audio data to FFmpeg via named pipes
5 mai, par Nicke ManarinI'm able to send frames one by one to FFmpeg via a name pipe to create a video out of them, but if I try sending audio to a second named pipe, FFmpeg only accepts 1 frame in the frame pipe and starts reading from the audio pipe soon after it.


ffmpeg.exe -loglevel debug -hwaccel auto 
-f:v rawvideo -r 25 -pix_fmt bgra -video_size 782x601 -i \\.\pipe\video_to_ffmpeg 
-f:a s16le -ac 2 -ar 48000 -i \\.\pipe\audio_to_ffmpeg 
-c:v libx264 -preset fast -pix_fmt yuv420p 
-vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" -crf 23 -f:v mp4 -vsync vfr 
-c:a aac -b:a 128k -ar 48000 -ac 2 
-y "C:\Users\user\Desktop\video.mp4"


I start both pipes like so :


_imagePipeServer = new NamedPipeServerStream(ImagePipeName, PipeDirection.Out, 1, PipeTransmissionMode.Byte, PipeOptions.Asynchronous);
_imagePipeStreamWriter = new StreamWriter(_imagePipeServer);
_imagePipeServer.BeginWaitForConnection(null, null);

_audioPipeServer = new NamedPipeServerStream(AudioPipeName, PipeDirection.Out, 1, PipeTransmissionMode.Byte, PipeOptions.Asynchronous);
_audioPipeStreamWriter = new StreamWriter(_audioPipeServer);
_audioPipeServer.BeginWaitForConnection(null, null);


And send the data to the pipes using these methods :


public void EncodeFrame(byte[] data)
{
 if (_imagePipeServer?.IsConnected != true)
 throw new FFmpegException("Pipe not connected", Arguments, Output);

 _imagePipeStreamWriter?.BaseStream.Write(data, 0, data.Length);
}


public void EncodeAudio(ISampleProvider provider, long length)
{
 if (_audioPipeServer?.IsConnected != true)
 throw new FFmpegException("Pipe not connected", Arguments, Output);

 var buffer = new byte[provider.WaveFormat.AverageBytesPerSecond * length / TimeSpan.TicksPerSecond];
 var bytesRead = provider.ToWaveProvider().Read(buffer, 0, buffer.Length);

 if (bytesRead < 1)
 return;

 _audioPipeStreamWriter?.BaseStream.Write(buffer, 0, bytesRead);
 _audioPipeStreamWriter?.BaseStream.Flush();
}


Not sending the audio (and thus not creating the audio pipe) works, with FFmpeg taking one frame at time and creating the video normally.


But if I try sending the audio via a secondary pipe, I can only send one frame. This is the output when that happens (Btw, FFmpeg v7.1) :


Splitting the commandline.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'.
Reading option '-hwaccel' ... matched as option 'hwaccel' (use HW accelerated decoding) with argument 'auto'.
Reading option '-f:v' ... matched as option 'f' (force container format (auto-detected otherwise)) with argument 'rawvideo'.
Reading option '-r' ... matched as option 'r' (override input framerate/convert to given output framerate (Hz value, fraction or abbreviation)) with argument '25'.
Reading option '-pix_fmt' ... matched as option 'pix_fmt' (set pixel format) with argument 'bgra'.
Reading option '-video_size' ... matched as AVOption 'video_size' with argument '782x601'.
Reading option '-i' ... matched as input url with argument '\\.\pipe\video_to_ffmpeg'.
Reading option '-f:a' ... matched as option 'f' (force container format (auto-detected otherwise)) with argument 's16le'.
Reading option '-ac' ... matched as option 'ac' (set number of audio channels) with argument '2'.
Reading option '-ar' ... matched as option 'ar' (set audio sampling rate (in Hz)) with argument '48000'.
Reading option '-i' ... matched as input url with argument '\\.\pipe\audio_to_ffmpeg'.
Reading option '-c:v' ... matched as option 'c' (select encoder/decoder ('copy' to copy stream without reencoding)) with argument 'libx264'.
Reading option '-preset' ... matched as AVOption 'preset' with argument 'fast'.
Reading option '-pix_fmt' ... matched as option 'pix_fmt' (set pixel format) with argument 'yuv420p'.
Reading option '-vf' ... matched as option 'vf' (alias for -filter:v (apply filters to video streams)) with argument 'scale=trunc(iw/2)*2:trunc(ih/2)*2'.
Reading option '-crf' ... matched as AVOption 'crf' with argument '23'.
Reading option '-f:v' ... matched as option 'f' (force container format (auto-detected otherwise)) with argument 'mp4'.
Reading option '-fps_mode' ... matched as option 'fps_mode' (set framerate mode for matching video streams; overrides vsync) with argument 'vfr'.
Reading option '-c:a' ... matched as option 'c' (select encoder/decoder ('copy' to copy stream without reencoding)) with argument 'aac'.
Reading option '-b:a' ... matched as option 'b' (video bitrate (please use -b:v)) with argument '128k'.
Reading option '-ar' ... matched as option 'ar' (set audio sampling rate (in Hz)) with argument '48000'.
Reading option '-ac' ... matched as option 'ac' (set number of audio channels) with argument '2'.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option 'C:\Users\user\Desktop\video.mp4' ... matched as output url.
Finished splitting the commandline.

Parsing a group of options: global.
Applying option loglevel (set logging level) with argument debug.
Applying option y (overwrite output files) with argument 1.
Successfully parsed a group of options.

Parsing a group of options: input url \\.\pipe\video_to_ffmpeg.
Applying option hwaccel (use HW accelerated decoding) with argument auto.
Applying option f:v (force container format (auto-detected otherwise)) with argument rawvideo.
Applying option r (override input framerate/convert to given output framerate (Hz value, fraction or abbreviation)) with argument 25.
Applying option pix_fmt (set pixel format) with argument bgra.
Successfully parsed a group of options.

Opening an input file: \\.\pipe\video_to_ffmpeg.
[rawvideo @ 000001c302ee08c0] Opening '\\.\pipe\video_to_ffmpeg' for reading
[file @ 000001c302ee1000] Setting default whitelist 'file,crypto,data'
[rawvideo @ 000001c302ee08c0] Before avformat_find_stream_info() pos: 0 bytes read:65536 seeks:0 nb_streams:1
[rawvideo @ 000001c302ee08c0] All info found
[rawvideo @ 000001c302ee08c0] After avformat_find_stream_info() pos: 1879928 bytes read:1879928 seeks:0 frames:1
Input #0, rawvideo, from '\\.\pipe\video_to_ffmpeg':
 Duration: N/A, start: 0.000000, bitrate: 375985 kb/s
 Stream #0:0, 1, 1/25: Video: rawvideo, 1 reference frame (BGRA / 0x41524742), bgra, 782x601, 0/1, 375985 kb/s, 25 tbr, 25 tbn
Successfully opened the file.

Parsing a group of options: input url \\.\pipe\audio_to_ffmpeg.
Applying option f:a (force container format (auto-detected otherwise)) with argument s16le.
Applying option ac (set number of audio channels) with argument 2.
Applying option ar (set audio sampling rate (in Hz)) with argument 48000.
Successfully parsed a group of options.

Opening an input file: \\.\pipe\audio_to_ffmpeg.
[s16le @ 000001c302ef5380] Opening '\\.\pipe\audio_to_ffmpeg' for reading
[file @ 000001c302ef58c0] Setting default whitelist 'file,crypto,data'


The difference if I try sending 1 frame then some bytes (arbitrary length based on fps) of audio is that I get this extra comment at the end :


[s16le @ 0000025948c96d00] Before avformat_find_stream_info() pos: 0 bytes read:15360 seeks:0 nb_streams:1


Extra calls to
EncodeFrame()hang forever at theBaseStream.Write(frameBytes, 0, frameBytes.Length)call, suggesting that FFmpeg is no longer reading the data.

Something is causing FFmpeg to close or stop reading the first pipe and only accept data from the second one.


Perhaps the command is missing something ?



🏆 Working solution


I started using two
BlockingCollectionobjects, with the consumers running in separate tasks.

Start the process, setting up the pipes :


private Process? _process;
private NamedPipeServerStream? _imagePipeServer;
private NamedPipeServerStream? _audioPipeServer;
private StreamWriter? _imagePipeStreamWriter;
private StreamWriter? _audioPipeStreamWriter;
private readonly BlockingCollection _videoCollection = new();
private readonly BlockingCollection _audioCollection = new();

private const string ImagePipeName = "video_to_ffmpeg";
private const string AudioPipeName = "audio_to_ffmpeg";
private const string PipeStructure = @"\\.\pipe\"; //This part is only sent to FFmpeg, not to the .NET pipe creation.

public void StartEncoding(string arguments)
{
 _process = new Process
 {
 StartInfo = new ProcessStartInfo
 {
 FileName = "path to ffmpeg",
 Arguments = arguments.Replace("{image}", PipeStructure + ImagePipeName).Replace("{audio}", PipeStructure + AudioPipeName),
 RedirectStandardInput = false,
 RedirectStandardOutput = true,
 RedirectStandardError = true,
 UseShellExecute = false,
 CreateNoWindow = true
 }
 };

 StartFramePipeConnection();
 StartAudioPipeConnection();

 _process. Start();
 _process.BeginErrorReadLine();
 _process.BeginOutputReadLine();
}

private void StartFramePipeConnection()
{
 if (_imagePipeServer != null)
 {
 if (_imagePipeServer.IsConnected)
 _imagePipeServer.Disconnect();

 _imagePipeServer.Dispose();
 }

 _imagePipeServer = new NamedPipeServerStream(ImagePipeName, PipeDirection.Out, 1, PipeTransmissionMode.Byte, PipeOptions.Asynchronous);
 _imagePipeStreamWriter = new StreamWriter(_imagePipeServer);
 _imagePipeServer.BeginWaitForConnection(VideoPipe_Connected, null);
}

private void StartAudioPipeConnection()
{
 if (_audioPipeServer != null)
 {
 if (_audioPipeServer.IsConnected)
 _audioPipeServer.Disconnect();

 _audioPipeServer.Dispose();
 }

 _audioPipeServer = new NamedPipeServerStream(AudioPipeName, PipeDirection.Out, 1, PipeTransmissionMode.Byte, PipeOptions.Asynchronous);
 _audioPipeStreamWriter = new StreamWriter(_audioPipeServer);
 _audioPipeServer.BeginWaitForConnection(AudioPipe_Connected, null);
}


Start sending the data as soon as the pipe gets connected. Once the
BlockingCollectiongets its signal that no more data is going to be sent, it will leave theforeachblock and it will wait for the pipe to drain its data.

private void VideoPipe_Connected(IAsyncResult ar)
{
 Task.Run(() =>
 {
 try
 {
 foreach (var frameBytes in _videoCollection.GetConsumingEnumerable())
 { 
 _imagePipeStreamWriter?.BaseStream.Write(frameBytes, 0, frameBytes.Length);
 }

 _imagePipeServer?.WaitForPipeDrain();
 _imagePipeStreamWriter?.Close();
 }
 catch (Exception e)
 {
 //Logging
 throw;
 }
 });
}

private void AudioPipe_Connected(IAsyncResult ar)
{
 Task.Run(() =>
 {
 try
 {
 foreach (var audioChunk in _audioCollection.GetConsumingEnumerable())
 {
 _audioPipeStreamWriter?.BaseStream.Write(audioChunk, 0, audioChunk.Length);
 }

 _audioPipeServer?.WaitForPipeDrain();
 _audioPipeStreamWriter?.Close();
 }
 catch (Exception e)
 {
 //Logging
 throw;
 }
 });
}


You can send the image and audio data as soon as the
BlockingCollectionsare initiated, no need to wait for the pipes to connect.

public void EncodeImage(byte[] data)
{
 _videoCollection.Add(data);
}

public void EncodeAudio(ISampleProvider provider, long length)
{
 var sampleCount = (int)(provider.WaveFormat.SampleRate * ((double)length / TimeSpan.TicksPerSecond) * provider.WaveFormat.Channels);
 var floatBuffer = new float[sampleCount];

 var samplesRead = provider.Read(floatBuffer, 0, sampleCount);

 if (samplesRead < 1)
 return 0;

 var byteBuffer = new byte[samplesRead * 4]; //4 bytes per float, f32le.
 Buffer.BlockCopy(floatBuffer, 0, byteBuffer, 0, byteBuffer.Length);

 
 _audioCollection.Add(byteBuffer);
}


Once you finished producing data, make sure to signal the
BlockingCollections:

public void FinishEncoding()
{
 //Signal the end of video/audio producer.
 _videoCollection.CompleteAdding();
 _audioCollection.CompleteAdding();

 //Waits for 20 seconds for encoding to finish.
 _process?.WaitForExit(20_000);
}


The FFmpeg arguments were changed slightly :


-loglevel trace -hwaccel auto 
-f:v rawvideo -probesize 32 -r 25 -pix_fmt bgra -video_size 1109x627 -i {image} 
-f:a f32le -ac 2 -ar 48000 -probesize 32 -i {audio} 
-c:v libx264 -preset fast -pix_fmt yuv420p 
-vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" -crf 23 -f:v mp4 -fps_mode vfr 
-c:a aac -b:a 128k -ar 48000 -ac 2 
-y "C:\Users\user\Desktop\Video.mp4"


Syndiquer tout le site
