Recherche avancée
Médias (1)
-
Publier une image simplement
13 avril 2011, par ,
Mis à jour : Février 2012
Langue : français
Type : Video
Autres articles (106)
-
Support de tous types de médias
10 avril 2011Contrairement à beaucoup de logiciels et autres plate-formes modernes de partage de documents, MediaSPIP a l’ambition de gérer un maximum de formats de documents différents qu’ils soient de type : images (png, gif, jpg, bmp et autres...) ; audio (MP3, Ogg, Wav et autres...) ; vidéo (Avi, MP4, Ogv, mpg, mov, wmv et autres...) ; contenu textuel, code ou autres (open office, microsoft office (tableur, présentation), web (html, css), LaTeX, Google Earth) (...)
-
Diogene : création de masques spécifiques de formulaires d’édition de contenus
26 octobre 2010, parDiogene est un des plugins ? SPIP activé par défaut (extension) lors de l’initialisation de MediaSPIP.
A quoi sert ce plugin
Création de masques de formulaires
Le plugin Diogène permet de créer des masques de formulaires spécifiques par secteur sur les trois objets spécifiques SPIP que sont : les articles ; les rubriques ; les sites
Il permet ainsi de définir en fonction d’un secteur particulier, un masque de formulaire par objet, ajoutant ou enlevant ainsi des champs afin de rendre le formulaire (...) -
Librairies et binaires spécifiques au traitement vidéo et sonore
31 janvier 2010, parLes logiciels et librairies suivantes sont utilisées par SPIPmotion d’une manière ou d’une autre.
Binaires obligatoires FFMpeg : encodeur principal, permet de transcoder presque tous les types de fichiers vidéo et sonores dans les formats lisibles sur Internet. CF ce tutoriel pour son installation ; Oggz-tools : outils d’inspection de fichiers ogg ; Mediainfo : récupération d’informations depuis la plupart des formats vidéos et sonores ;
Binaires complémentaires et facultatifs flvtool2 : (...)
Sur d’autres sites (8168)
-
Encoding YUV420p uncompressed video that comes from custom IO instead of file. (C++ code)
24 avril 2023, par devprogI am trying to encode YUV420p raw data that being streamed live from a private memory implementation. It didn't work. As a preliminary stage to private memory reading, and after some searches, I tried to read Custom IO input according ffmpeg example. This didn't work either. So, for my tests I created YUV file using


ffmpeg -f lavfi -i testsrc=duration=10:size=1920x1080:rate=30 1920x1080p30_10s.mp4


create a mp4 file. As a note, it turns out that it creates mp4 file with yuv444 color system. I changed it to yuv420p with


ffmpeg -i 1920x1080p30_10s.mp4 -c:v libx264 -pix_fmt yuv420p 1920x1080p30_10s_420p.mp4


then create a yuv file with


ffmpeg -i 1920x1080p30_10s_420p.mp4 -c:v rawvideo -pixel_format yuv420p output_1920x1080_420p.yuv


Test the output_1920x1080_420p.yuv with ffplay, looks good. This file will be my input to custom IO operation.


Then wrote a sample code as follows (based on several links) :
avio_reading ffmpeg example.
Creating Custom FFmpeg IO-Context


Code fail with the function


if ((ret = avformat_open_input(&pAVFormatContext, "IO", NULL, &options_in) < 0) )
 {
 printf ("err=%d\n", ret);
 av_strerror(ret, errbuf, sizeof(errbuf));
 fprintf(stderr, "Unable to open err=%s\n", errbuf);
 return false;
 }


Some notes :


- 

- I added the code, the compile command & the output
- I tried use with option parameters to avformat_open_input() & without, it failed.
- I tried to use with seek callback to avio_alloc_context() & without, it failed.
- I tries to use instead of YUV some mp4 file and looks as running (didn't fail) - BUT it is NOT what I need. I Need a YUV input










Can anyone help ? Is it even possible ?


My following code compiles with :


g++ ffmpeg_custom_io.cpp -lavformat -lavdevice -lavcodec -lavutil -lswscale -lswresample -lpthread -pthread -o ffmpeg_custom_io


and whole code is as follows :


using namespace std;

#include 
#include 
#include 
#include <iostream>

extern "C" {
 #include <libavformat></libavformat>avformat.h>
 #include <libavcodec></libavcodec>avcodec.h>
 
 #include <libavutil></libavutil>opt.h>
 #include <libavutil></libavutil>imgutils.h>
 #include <libavutil></libavutil>file.h>
}


struct buffer_data {
 uint8_t * bd_ptr;
 size_t bd_size; ///< size left in the buffer
};

int readFileBuffer(void *opaque, uint8_t *buf, int buf_size)
{

 printf("Start reading custom IO\n");
 struct buffer_data *bd = (struct buffer_data *)opaque;
 buf_size = FFMIN(buf_size, bd->bd_size);

 printf("min buf size:%d\n", buf_size);

 if (!buf_size) {
 printf("return -1, size = %d\n", buf_size);
 return -1;
 }
 printf("ptr:%p size:%zu\n", bd->bd_ptr, bd->bd_size);

 /* copy internal buffer data to buf */
 memcpy(buf, bd->bd_ptr, buf_size);
 bd->bd_ptr += buf_size;
 bd->bd_size -= buf_size;
 printf("End reading custom IO\n");
 return buf_size;

}

int64_t seekFunction(void* opaque, int64_t offset, int whence)
{
 std::cout << "SEEK " << std::endl;
 if (whence == AVSEEK_SIZE)
 return -1; // I don't know "size of my handle in bytes"
}

static void encode(AVCodecContext *enc_ctx, AVFrame *frame, AVPacket *pkt,
 FILE *outfile)
{
 int ret;

 /* send the frame to the encoder */
 if (frame)
 printf("Send frame %3" PRId64 "\n", frame->pts);

 ret = avcodec_send_frame(enc_ctx, frame);
 if (ret < 0) {
 fprintf(stderr, "Error sending a frame for encoding\n");
 exit(1);
 }

 while (ret >= 0) {
 ret = avcodec_receive_packet(enc_ctx, pkt);
 if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
 return;
 else if (ret < 0) {
 fprintf(stderr, "Error during encoding\n");
 exit(1);
 }

 printf("Write packet %3" PRId64 " (size=%5d)\n", pkt->pts, pkt->size);
 fwrite(pkt->data, 1, pkt->size, outfile);
 av_packet_unref(pkt);
 }
}

int main(int argc, char **argv)
{

 av_log_set_level(AV_LOG_MAX_OFFSET);

 AVFormatContext* pAVFormatContext;
 uint8_t *buffer = NULL, *avio_ctx_buffer = NULL;
 size_t buffer_size, avio_ctx_buffer_size = 32768;

 int ret = 0;
 char errbuf[100];

 // Alloc format context
 //--------------------
 if ( !(pAVFormatContext = avformat_alloc_context()) )
 {
 ret = AVERROR(ENOMEM);
 std::cout << "error=" << ret << " Fail to allocate avformat_alloc_context()\n";
 return false;
 }

 // Copy from ffmpeg example - bd buffer
 //--------------------
 struct buffer_data bd = { 0, 0 };

 // slurp file content into buffer
 ret = av_file_map("output_1920x1080_420p.yuv", &buffer, &buffer_size, 0, NULL);
 if (ret < 0)
 {
 std::cout << "error=" << ret << " Fail to allocate avformat_alloc_context()\n";
 return false;
 }

 // fill opaque structure used by the AVIOContext read callback
 bd.bd_ptr = buffer;
 bd.bd_size = buffer_size;

 // Prepare AVIO context
 //--------------------
 avio_ctx_buffer = static_cast(av_malloc(avio_ctx_buffer_size + AV_INPUT_BUFFER_PADDING_SIZE));

 if (!avio_ctx_buffer) {
 ret = AVERROR(ENOMEM);
 std::cout << "error=" << ret << " Fail to allocate av_malloc()\n";
 return false;
 }

 AVIOContext* pAVIOContext = avio_alloc_context(avio_ctx_buffer,
 avio_ctx_buffer_size,
 0,
 &bd,
 readFileBuffer,
 NULL,
 seekFunction);

 if (!pAVIOContext) {
 ret = AVERROR(ENOMEM);
 std::cout << "error=" << ret << " Fail to allocate avio_alloc_context()\n";
 return false;
 }

 pAVFormatContext->pb = pAVIOContext;
 pAVFormatContext->flags = AVFMT_FLAG_CUSTOM_IO; // - NOT SURE ABOUT THIS FLAG

 // Handle ffmpeg input
 //--------------------
 AVDictionary* options_in = NULL;
 //av_dict_set(&options_in, "framerate", "30", 0);
 av_dict_set(&options_in, "video_size", "1920x1080", 0);
 av_dict_set(&options_in, "pixel_format", "yuv420p", 0);
 av_dict_set(&options_in, "vcodec", "rawvideo", 0);

 if ((ret = avformat_open_input(&pAVFormatContext, "IO", NULL, &options_in) < 0) )
 {
 printf ("err=%d\n", ret);
 av_strerror(ret, errbuf, sizeof(errbuf));
 fprintf(stderr, "Unable to open err=%s\n", errbuf);
 return false;
 }

 //Raw video doesn't contain any stream information.
 //if ((ret = avformat_find_stream_info(pAVFormatContext, 0)) < 0) {
 // fprintf(stderr, "Failed to retrieve input stream information");
 //}

 AVCodec* pAVCodec = avcodec_find_decoder(AV_CODEC_ID_RAWVIDEO); //Get pointer to rawvideo codec.

 AVCodecContext* pAVCodecContext = avcodec_alloc_context3(pAVCodec); //Allocate codec context.

 //Fill the codec context based on the values from the codec parameters.
 AVStream *vid_stream = pAVFormatContext->streams[0];
 avcodec_parameters_to_context(pAVCodecContext, vid_stream->codecpar);

 avcodec_open2(pAVCodecContext, pAVCodec, NULL); //Open the codec

 //Allocate memory for packet and frame
 AVPacket* pAVPacketIn = av_packet_alloc();
 AVFrame* pAVFrameOut = av_frame_alloc();

 /* find the mpeg1video encoder */
 AVCodec* pAVCodecOut = avcodec_find_encoder_by_name("libx264");
 if (!pAVCodecOut) {
 fprintf(stderr, "Codec '%s' not found\n", "libx264");
 exit(1);
 }

 AVCodecContext* pAVCodecContextOut = avcodec_alloc_context3(pAVCodecOut);
 if (!pAVCodecContextOut) {
 fprintf(stderr, "Could not allocate video codec context\n");
 exit(1);
 }

 AVPacket* pAVPacketOut = av_packet_alloc();
 if (!pAVPacketOut)
 exit(1);

 /* put sample parameters */
 pAVCodecContextOut->bit_rate = 2000000;
 /* resolution must be a multiple of two */
 pAVCodecContextOut->width = 1280;
 pAVCodecContextOut->height = 720;
 /* frames per second */
 pAVCodecContextOut->time_base = (AVRational){1, 30};
 pAVCodecContextOut->framerate = (AVRational){30, 1};

 /* emit one intra frame every ten frames
 * check frame pict_type before passing frame
 * to encoder, if frame->pict_type is AV_PICTURE_TYPE_I
 * then gop_size is ignored and the output of encoder
 * will always be I frame irrespective to gop_size
 */
 pAVCodecContextOut->gop_size = 10;
 pAVCodecContextOut->max_b_frames = 0;
 pAVCodecContextOut->pix_fmt = AV_PIX_FMT_YUV420P;

 if (pAVCodecOut->id == AV_CODEC_ID_H264)
 av_opt_set(pAVCodecContextOut->priv_data, "preset", "slow", 0);

 /* open it */
 ret = avcodec_open2(pAVCodecContextOut, pAVCodecOut, NULL);
 if (ret < 0) {
 //fprintf(stderr, "Could not open codec: %s\n", av_err2str(ret));
 exit(1);
 }

 FILE* filename_out = fopen("out_vid_h264.mp4", "wb");
 if (!filename_out) {
 fprintf(stderr, "Could not open %s\n", "out_vid_h264.mp4");
 exit(1);
 }

 pAVFrameOut->format = pAVCodecContextOut->pix_fmt;
 pAVFrameOut->width = pAVCodecContextOut->width;
 pAVFrameOut->height = pAVCodecContextOut->height;


 uint8_t endcode_suf[] = { 0, 0, 1, 0xb7 };
 int i = 0;
 //Read video frames and pass through the decoder.
 //Note: Since the video is rawvideo, we don't really have to pass it through the decoder.
 while (av_read_frame(pAVFormatContext, pAVPacketIn) >= 0)
 {
 //The video is not encoded - passing through the decoder is simply copy the data.
 avcodec_send_packet(pAVCodecContext, pAVPacketIn); //Supply raw packet data as input to the "decoder".
 avcodec_receive_frame(pAVCodecContext, pAVFrameOut); //Return decoded output data from the "decoder".

 fflush(stdout);

 pAVFrameOut->pts = i++;

 /* encode the image */
 encode(pAVCodecContextOut, pAVFrameOut, pAVPacketOut, filename_out);
 }

 /* flush the encoder */
 encode(pAVCodecContextOut, NULL, pAVPacketOut, filename_out);

 /* Add sequence end code to have a real MPEG file.
 It makes only sense because this tiny examples writes packets
 directly. This is called "elementary stream" and only works for some
 codecs. To create a valid file, you usually need to write packets
 into a proper file format or protocol; see mux.c.
 */
 if (pAVCodecOut->id == AV_CODEC_ID_MPEG1VIDEO || pAVCodecOut->id == AV_CODEC_ID_MPEG2VIDEO)
 fwrite(endcode_suf, 1, sizeof(endcode_suf), filename_out);
 fclose(filename_out);


 return 0;
}
</iostream>

The output


./ffmpeg_custom_io 
Start reading custom IO
min buf size:32768
ptr:0x7f236a200000 size:933120000
End reading custom IO
Probing adp score:25 size:2048
Probing adp score:25 size:4096
Probing adp score:25 size:8192
Probing adp score:25 size:16384
Start reading custom IO
min buf size:32768
ptr:0x7f236a208000 size:933087232
End reading custom IO
Start reading custom IO
min buf size:65536
ptr:0x7f236a210000 size:933054464
End reading custom IO
Start reading custom IO
min buf size:131072
ptr:0x7f236a220000 size:932988928
End reading custom IO
Start reading custom IO
min buf size:262144
ptr:0x7f236a240000 size:932857856
End reading custom IO
Start reading custom IO
min buf size:524288
ptr:0x7f236a280000 size:932595712
End reading custom IO
err=1
Unable to open err=Error number 1 occurred



I tried all as noted above expecting avformat_open_input() will succeed, but it failed.


-
Ffmpeg fail on avformat_open_input() while try to encode YUV420p uncompressed video, input from custom IO instead of file. (C++ code)
25 avril 2023, par devprogI am trying to encode YUV420p raw data that being streamed live from a private memory implementation. It didn't work. As a preliminary stage to private memory reading, and after some searches, I tried to read Custom IO input according ffmpeg example. This didn't work either. So, for my tests I created YUV file using


ffmpeg -f lavfi -i testsrc=duration=10:size=1920x1080:rate=30 1920x1080p30_10s.mp4


create a mp4 file. As a note, it turns out that it creates mp4 file with yuv444 color system. I changed it to yuv420p with


ffmpeg -i 1920x1080p30_10s.mp4 -c:v libx264 -pix_fmt yuv420p 1920x1080p30_10s_420p.mp4


then create a yuv file with


ffmpeg -i 1920x1080p30_10s_420p.mp4 -c:v rawvideo -pixel_format yuv420p output_1920x1080_420p.yuv


Test the output_1920x1080_420p.yuv with ffplay, looks good. This file will be my input to custom IO operation.


Then wrote a sample code as follows (based on several links) :
avio_reading ffmpeg example.
Creating Custom FFmpeg IO-Context


Code fail with the function


if ((ret = avformat_open_input(&pAVFormatContext, "IO", NULL, &options_in) < 0) )
 {
 printf ("err=%d\n", ret);
 av_strerror(ret, errbuf, sizeof(errbuf));
 fprintf(stderr, "Unable to open err=%s\n", errbuf);
 return false;
 }


Some notes :


- 

- I added the code, the compile command & the output
- I tried use with option parameters to avformat_open_input() & without, it failed.
- I tried to use with seek callback to avio_alloc_context() & without, it failed.
- I tries to use instead of YUV some mp4 file and looks as running (didn't fail) - BUT it is NOT what I need. I Need a YUV input










Can anyone help ? Is it even possible ?


My following code compiles with :


g++ ffmpeg_custom_io.cpp -lavformat -lavdevice -lavcodec -lavutil -lswscale -lswresample -lpthread -pthread -o ffmpeg_custom_io


and whole code is as follows :


using namespace std;

#include 
#include 
#include 
#include <iostream>

extern "C" {
 #include <libavformat></libavformat>avformat.h>
 #include <libavcodec></libavcodec>avcodec.h>
 
 #include <libavutil></libavutil>opt.h>
 #include <libavutil></libavutil>imgutils.h>
 #include <libavutil></libavutil>file.h>
}


struct buffer_data {
 uint8_t * bd_ptr;
 size_t bd_size; ///< size left in the buffer
};

int readFileBuffer(void *opaque, uint8_t *buf, int buf_size)
{

 printf("Start reading custom IO\n");
 struct buffer_data *bd = (struct buffer_data *)opaque;
 buf_size = FFMIN(buf_size, bd->bd_size);

 printf("min buf size:%d\n", buf_size);

 if (!buf_size) {
 printf("return -1, size = %d\n", buf_size);
 return -1;
 }
 printf("ptr:%p size:%zu\n", bd->bd_ptr, bd->bd_size);

 /* copy internal buffer data to buf */
 memcpy(buf, bd->bd_ptr, buf_size);
 bd->bd_ptr += buf_size;
 bd->bd_size -= buf_size;
 printf("End reading custom IO\n");
 return buf_size;

}

int64_t seekFunction(void* opaque, int64_t offset, int whence)
{
 std::cout << "SEEK " << std::endl;
 if (whence == AVSEEK_SIZE)
 return -1; // I don't know "size of my handle in bytes"
}

static void encode(AVCodecContext *enc_ctx, AVFrame *frame, AVPacket *pkt,
 FILE *outfile)
{
 int ret;

 /* send the frame to the encoder */
 if (frame)
 printf("Send frame %3" PRId64 "\n", frame->pts);

 ret = avcodec_send_frame(enc_ctx, frame);
 if (ret < 0) {
 fprintf(stderr, "Error sending a frame for encoding\n");
 exit(1);
 }

 while (ret >= 0) {
 ret = avcodec_receive_packet(enc_ctx, pkt);
 if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
 return;
 else if (ret < 0) {
 fprintf(stderr, "Error during encoding\n");
 exit(1);
 }

 printf("Write packet %3" PRId64 " (size=%5d)\n", pkt->pts, pkt->size);
 fwrite(pkt->data, 1, pkt->size, outfile);
 av_packet_unref(pkt);
 }
}

int main(int argc, char **argv)
{

 av_log_set_level(AV_LOG_MAX_OFFSET);

 AVFormatContext* pAVFormatContext;
 uint8_t *buffer = NULL, *avio_ctx_buffer = NULL;
 size_t buffer_size, avio_ctx_buffer_size = 32768;

 int ret = 0;
 char errbuf[100];

 // Alloc format context
 //--------------------
 if ( !(pAVFormatContext = avformat_alloc_context()) )
 {
 ret = AVERROR(ENOMEM);
 std::cout << "error=" << ret << " Fail to allocate avformat_alloc_context()\n";
 return false;
 }

 // Copy from ffmpeg example - bd buffer
 //--------------------
 struct buffer_data bd = { 0, 0 };

 // slurp file content into buffer
 ret = av_file_map("output_1920x1080_420p.yuv", &buffer, &buffer_size, 0, NULL);
 if (ret < 0)
 {
 std::cout << "error=" << ret << " Fail to allocate avformat_alloc_context()\n";
 return false;
 }

 // fill opaque structure used by the AVIOContext read callback
 bd.bd_ptr = buffer;
 bd.bd_size = buffer_size;

 // Prepare AVIO context
 //--------------------
 avio_ctx_buffer = static_cast(av_malloc(avio_ctx_buffer_size + AV_INPUT_BUFFER_PADDING_SIZE));

 if (!avio_ctx_buffer) {
 ret = AVERROR(ENOMEM);
 std::cout << "error=" << ret << " Fail to allocate av_malloc()\n";
 return false;
 }

 AVIOContext* pAVIOContext = avio_alloc_context(avio_ctx_buffer,
 avio_ctx_buffer_size,
 0,
 &bd,
 readFileBuffer,
 NULL,
 seekFunction);

 if (!pAVIOContext) {
 ret = AVERROR(ENOMEM);
 std::cout << "error=" << ret << " Fail to allocate avio_alloc_context()\n";
 return false;
 }

 pAVFormatContext->pb = pAVIOContext;
 pAVFormatContext->flags = AVFMT_FLAG_CUSTOM_IO; // - NOT SURE ABOUT THIS FLAG

 // Handle ffmpeg input
 //--------------------
 AVDictionary* options_in = NULL;
 //av_dict_set(&options_in, "framerate", "30", 0);
 av_dict_set(&options_in, "video_size", "1920x1080", 0);
 av_dict_set(&options_in, "pixel_format", "yuv420p", 0);
 av_dict_set(&options_in, "vcodec", "rawvideo", 0);

 if ((ret = avformat_open_input(&pAVFormatContext, "IO", NULL, &options_in) < 0) )
 {
 printf ("err=%d\n", ret);
 av_strerror(ret, errbuf, sizeof(errbuf));
 fprintf(stderr, "Unable to open err=%s\n", errbuf);
 return false;
 }

 //Raw video doesn't contain any stream information.
 //if ((ret = avformat_find_stream_info(pAVFormatContext, 0)) < 0) {
 // fprintf(stderr, "Failed to retrieve input stream information");
 //}

 AVCodec* pAVCodec = avcodec_find_decoder(AV_CODEC_ID_RAWVIDEO); //Get pointer to rawvideo codec.

 AVCodecContext* pAVCodecContext = avcodec_alloc_context3(pAVCodec); //Allocate codec context.

 //Fill the codec context based on the values from the codec parameters.
 AVStream *vid_stream = pAVFormatContext->streams[0];
 avcodec_parameters_to_context(pAVCodecContext, vid_stream->codecpar);

 avcodec_open2(pAVCodecContext, pAVCodec, NULL); //Open the codec

 //Allocate memory for packet and frame
 AVPacket* pAVPacketIn = av_packet_alloc();
 AVFrame* pAVFrameOut = av_frame_alloc();

 /* find the mpeg1video encoder */
 AVCodec* pAVCodecOut = avcodec_find_encoder_by_name("libx264");
 if (!pAVCodecOut) {
 fprintf(stderr, "Codec '%s' not found\n", "libx264");
 exit(1);
 }

 AVCodecContext* pAVCodecContextOut = avcodec_alloc_context3(pAVCodecOut);
 if (!pAVCodecContextOut) {
 fprintf(stderr, "Could not allocate video codec context\n");
 exit(1);
 }

 AVPacket* pAVPacketOut = av_packet_alloc();
 if (!pAVPacketOut)
 exit(1);

 /* put sample parameters */
 pAVCodecContextOut->bit_rate = 2000000;
 /* resolution must be a multiple of two */
 pAVCodecContextOut->width = 1280;
 pAVCodecContextOut->height = 720;
 /* frames per second */
 pAVCodecContextOut->time_base = (AVRational){1, 30};
 pAVCodecContextOut->framerate = (AVRational){30, 1};

 /* emit one intra frame every ten frames
 * check frame pict_type before passing frame
 * to encoder, if frame->pict_type is AV_PICTURE_TYPE_I
 * then gop_size is ignored and the output of encoder
 * will always be I frame irrespective to gop_size
 */
 pAVCodecContextOut->gop_size = 10;
 pAVCodecContextOut->max_b_frames = 0;
 pAVCodecContextOut->pix_fmt = AV_PIX_FMT_YUV420P;

 if (pAVCodecOut->id == AV_CODEC_ID_H264)
 av_opt_set(pAVCodecContextOut->priv_data, "preset", "slow", 0);

 /* open it */
 ret = avcodec_open2(pAVCodecContextOut, pAVCodecOut, NULL);
 if (ret < 0) {
 //fprintf(stderr, "Could not open codec: %s\n", av_err2str(ret));
 exit(1);
 }

 FILE* filename_out = fopen("out_vid_h264.mp4", "wb");
 if (!filename_out) {
 fprintf(stderr, "Could not open %s\n", "out_vid_h264.mp4");
 exit(1);
 }

 pAVFrameOut->format = pAVCodecContextOut->pix_fmt;
 pAVFrameOut->width = pAVCodecContextOut->width;
 pAVFrameOut->height = pAVCodecContextOut->height;


 uint8_t endcode_suf[] = { 0, 0, 1, 0xb7 };
 int i = 0;
 //Read video frames and pass through the decoder.
 //Note: Since the video is rawvideo, we don't really have to pass it through the decoder.
 while (av_read_frame(pAVFormatContext, pAVPacketIn) >= 0)
 {
 //The video is not encoded - passing through the decoder is simply copy the data.
 avcodec_send_packet(pAVCodecContext, pAVPacketIn); //Supply raw packet data as input to the "decoder".
 avcodec_receive_frame(pAVCodecContext, pAVFrameOut); //Return decoded output data from the "decoder".

 fflush(stdout);

 pAVFrameOut->pts = i++;

 /* encode the image */
 encode(pAVCodecContextOut, pAVFrameOut, pAVPacketOut, filename_out);
 }

 /* flush the encoder */
 encode(pAVCodecContextOut, NULL, pAVPacketOut, filename_out);

 /* Add sequence end code to have a real MPEG file.
 It makes only sense because this tiny examples writes packets
 directly. This is called "elementary stream" and only works for some
 codecs. To create a valid file, you usually need to write packets
 into a proper file format or protocol; see mux.c.
 */
 if (pAVCodecOut->id == AV_CODEC_ID_MPEG1VIDEO || pAVCodecOut->id == AV_CODEC_ID_MPEG2VIDEO)
 fwrite(endcode_suf, 1, sizeof(endcode_suf), filename_out);
 fclose(filename_out);


 return 0;
}
</iostream>

The output


./ffmpeg_custom_io 
Start reading custom IO
min buf size:32768
ptr:0x7f236a200000 size:933120000
End reading custom IO
Probing adp score:25 size:2048
Probing adp score:25 size:4096
Probing adp score:25 size:8192
Probing adp score:25 size:16384
Start reading custom IO
min buf size:32768
ptr:0x7f236a208000 size:933087232
End reading custom IO
Start reading custom IO
min buf size:65536
ptr:0x7f236a210000 size:933054464
End reading custom IO
Start reading custom IO
min buf size:131072
ptr:0x7f236a220000 size:932988928
End reading custom IO
Start reading custom IO
min buf size:262144
ptr:0x7f236a240000 size:932857856
End reading custom IO
Start reading custom IO
min buf size:524288
ptr:0x7f236a280000 size:932595712
End reading custom IO
err=1
Unable to open err=Error number 1 occurred



I tried all as noted above expecting avformat_open_input() will succeed, but it failed.


-
FFmpeg invalid data found when processing input with D3D11VA and DXVA2 hw acceleration
6 mai 2023, par grill2010I'm currently porting my Android streaming app to Windows and for decoding the h264 video stream I use FFmpeg with possible hardware acceleration. Last two weeks I was reading a lot of documentation and studied a lot of examples on the internet. For my project I use JavaCV which is internally using FFmpeg 5.1.2. On Windows I support D3D11VA, DXVA2 and Cuvid for hardware acceleration (and software decoding as fallback). During testing I noticed that I get some strange artefacts when using D3D11VA or DXVA2 hw acceleration. Upon further investigation I saw that I receive a lot of


"Invalid data found when processing input"


errors when calling
avcodec_send_packet. It seems this error only occurs on certain key frames. The error is reproducable all the time. The software decoder or cuvid decoder has absolutely no problem to process and to decode such a frame, so not sure why there should be an invalid data in the frame ? I played around a lot with the decoder configuration but nothing seems to help and at that point I think this is definitely not normal behaviour.

I provided a reproducable example which can be downloaded from here. All the important part is in the App.java class. In addition an example of the code was posted below. The example is trying to decode a key frame. The keyframe data with sps and pps is read from a file in the resource folder of the project.


To run the project just perform a .\gradlew build and afterwards a .\gradlew run. If you run the example the last log message shown in the terminal should be "SUCESS with HW decoding". The hardware decoder can be changed via the HW_DEVICE_TYPE variable in the App.java class. To disable hw acceleration just set the USE_HW_ACCEL to false.


For me everything seems to be correct and I have no idea what could be wrong with the code. I looked a lot on the internet to find the root cause of the issue and I did not really found a solution but other sources which are related to (maybe) the same problem


https://www.mail-archive.com/libav-user@ffmpeg.org/...


https://stackoverflow.com/questions/67307397/ffmpeg-...


I also found another streaming app on Windows which can use D3D11VA and DXVA2 hardware acceleration called Chiaki (it requires a PS4 or a PS5) which seems to have the exact same problem. I used the build provided here. It will fail to decode certain key frames as well when hardware acceleration with D3D11VA or DXVA2 is selected (e.g. the first key frame received by the stream). Chiaki can output the seemingly faulty frame but this is also possible with my example by setting the USE_AV_EF_EXPLODE to false.


Are there any ffmpeg gurus around that can check what's the problem with D3D11VA or DXVA2 ? Anything else that needs to be done to make the D3D11VA and DXVA2 hardware decoder work ? I'm now completly out of ideas and I'm not even sure if this is fixable.


I have Windows 11 installed on my test machine, and I have the latest Nvidea drivers installed.


Edit : here is a shrinked complete example of my project (keyframe file which includes sps and pps can be downloaded from here. It's a hex string file and can be decoded with the provided HexUtil class)


import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.layout.Pane;
import javafx.stage.Stage;
import org.bytedeco.ffmpeg.avcodec.AVCodec;
import org.bytedeco.ffmpeg.avcodec.AVCodecContext;
import org.bytedeco.ffmpeg.avcodec.AVCodecHWConfig;
import org.bytedeco.ffmpeg.avcodec.AVPacket;
import org.bytedeco.ffmpeg.avutil.AVBufferRef;
import org.bytedeco.ffmpeg.avutil.AVDictionary;
import org.bytedeco.ffmpeg.avutil.AVFrame;
import org.bytedeco.javacpp.BytePointer;
import org.bytedeco.javacpp.IntPointer;
import org.bytedeco.javacv.FFmpegLogCallback;
import org.tinylog.Logger;

import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.Objects;
import java.util.function.Consumer;

import static org.bytedeco.ffmpeg.avcodec.AVCodecContext.AV_EF_EXPLODE;
import static org.bytedeco.ffmpeg.avcodec.AVCodecContext.FF_THREAD_SLICE;
import static org.bytedeco.ffmpeg.global.avcodec.*;
import static org.bytedeco.ffmpeg.global.avutil.*;

public class App extends Application {

 /**** decoder variables ****/

 private AVHWContextInfo hardwareContext;

 private AVCodec decoder;
 private AVCodecContext m_VideoDecoderCtx;

 private AVCodecContext.Get_format_AVCodecContext_IntPointer formatCallback;
 private final int streamResolutionX = 1920;
 private final int streamResolutionY = 1080;

 // AV_HWDEVICE_TYPE_CUDA // example works with cuda
 // AV_HWDEVICE_TYPE_DXVA2 // producing Invalid data found on keyframe
 // AV_HWDEVICE_TYPE_D3D11VA // producing Invalid data found on keyframe
 private static final int HW_DEVICE_TYPE = AV_HWDEVICE_TYPE_DXVA2;

 private static final boolean USE_HW_ACCEL = true;

 private static final boolean USE_AV_EF_EXPLODE = true;

 public static void main(final String[] args) {
 //System.setProperty("prism.order", "d3d,sw");
 System.setProperty("prism.vsync", "false");
 Application.launch(App.class);
 }

 @Override
 public void start(final Stage primaryStage) {
 final Pane dummyPane = new Pane();
 dummyPane.setStyle("-fx-background-color: black");
 final Scene scene = new Scene(dummyPane, this.streamResolutionX, this.streamResolutionY);
 primaryStage.setScene(scene);
 primaryStage.show();
 primaryStage.setMinWidth(480);
 primaryStage.setMinHeight(360);

 this.initializeFFmpeg(result -> {
 if (!result) {
 Logger.error("FFmpeg could not be initialized correctly, terminating program");
 System.exit(1);
 return;
 }
 this.performTestFramesFeeding();
 });
 }

 private void initializeFFmpeg(final Consumer<boolean> finishHandler) {
 FFmpegLogCallback.setLevel(AV_LOG_DEBUG); // Increase log level until the first frame is decoded
 //FFmpegLogCallback.setLevel(AV_LOG_QUIET);
 this.decoder = avcodec_find_decoder(AV_CODEC_ID_H264); // usually decoder name is h264 and without hardware support it's yuv420p otherwise nv12

 if (this.decoder == null) {
 Logger.error("Unable to find decoder for format {}", "h264");
 finishHandler.accept(false);
 return;
 }
 Logger.info("Current decoder name: {}, {}", this.decoder.name().getString(), this.decoder.long_name().getString());

 if (true) {
 for (; ; ) {
 this.m_VideoDecoderCtx = avcodec_alloc_context3(this.decoder);
 if (this.m_VideoDecoderCtx == null) {
 Logger.error("Unable to find decoder for format AV_CODEC_ID_H264");
 if (this.hardwareContext != null) {
 this.hardwareContext.free();
 this.hardwareContext = null;
 }
 continue;
 }

 if (App.USE_HW_ACCEL) {
 this.hardwareContext = this.createHardwareContext();
 if (this.hardwareContext != null) {
 Logger.info("Set hwaccel support");
 this.m_VideoDecoderCtx.hw_device_ctx(this.hardwareContext.hwContext()); // comment to disable hwaccel
 }
 } else {
 Logger.info("Hwaccel manually disabled");
 }


 // Always request low delay decoding
 this.m_VideoDecoderCtx.flags(this.m_VideoDecoderCtx.flags() | AV_CODEC_FLAG_LOW_DELAY);

 // Allow display of corrupt frames and frames missing references
 this.m_VideoDecoderCtx.flags(this.m_VideoDecoderCtx.flags() | AV_CODEC_FLAG_OUTPUT_CORRUPT);
 this.m_VideoDecoderCtx.flags2(this.m_VideoDecoderCtx.flags2() | AV_CODEC_FLAG2_SHOW_ALL);

 if (App.USE_AV_EF_EXPLODE) {
 // Report decoding errors to allow us to request a key frame
 this.m_VideoDecoderCtx.err_recognition(this.m_VideoDecoderCtx.err_recognition() | AV_EF_EXPLODE);
 }

 // Enable slice multi-threading for software decoding
 if (this.m_VideoDecoderCtx.hw_device_ctx() == null) { // if not hw accelerated
 this.m_VideoDecoderCtx.thread_type(this.m_VideoDecoderCtx.thread_type() | FF_THREAD_SLICE);
 this.m_VideoDecoderCtx.thread_count(2/*AppUtil.getCpuCount()*/);
 } else {
 // No threading for HW decode
 this.m_VideoDecoderCtx.thread_count(1);
 }

 this.m_VideoDecoderCtx.width(this.streamResolutionX);
 this.m_VideoDecoderCtx.height(this.streamResolutionY);
 this.m_VideoDecoderCtx.pix_fmt(this.getDefaultPixelFormat());

 this.formatCallback = new AVCodecContext.Get_format_AVCodecContext_IntPointer() {
 @Override
 public int call(final AVCodecContext context, final IntPointer pixelFormats) {
 final boolean hwDecodingSupported = context.hw_device_ctx() != null && App.this.hardwareContext != null;
 final int preferredPixelFormat = hwDecodingSupported ?
 App.this.hardwareContext.hwConfig().pix_fmt() :
 context.pix_fmt();
 int i = 0;
 while (true) {
 final int currentSupportedFormat = pixelFormats.get(i++);
 System.out.println("Supported pixel formats " + currentSupportedFormat);
 if (currentSupportedFormat == preferredPixelFormat) {
 Logger.info("[FFmpeg]: pixel format in format callback is {}", currentSupportedFormat);
 return currentSupportedFormat;
 }
 if (currentSupportedFormat == AV_PIX_FMT_NONE) {
 break;
 }
 }

 i = 0;
 while (true) { // try again and search for yuv
 final int currentSupportedFormat = pixelFormats.get(i++);
 if (currentSupportedFormat == AV_PIX_FMT_YUV420P) {
 Logger.info("[FFmpeg]: Not found in first match so use {}", AV_PIX_FMT_YUV420P);
 return currentSupportedFormat;
 }
 if (currentSupportedFormat == AV_PIX_FMT_NONE) {
 break;
 }
 }

 i = 0;
 while (true) { // try again and search for nv12
 final int currentSupportedFormat = pixelFormats.get(i++);
 if (currentSupportedFormat == AV_PIX_FMT_NV12) {
 Logger.info("[FFmpeg]: Not found in second match so use {}", AV_PIX_FMT_NV12);
 return currentSupportedFormat;
 }
 if (currentSupportedFormat == AV_PIX_FMT_NONE) {
 break;
 }
 }

 Logger.info("[FFmpeg]: pixel format in format callback is using fallback {}", AV_PIX_FMT_NONE);
 return AV_PIX_FMT_NONE;
 }
 };
 this.m_VideoDecoderCtx.get_format(this.formatCallback);

 final AVDictionary options = new AVDictionary(null);
 final int result = avcodec_open2(this.m_VideoDecoderCtx, this.decoder, options);
 if (result < 0) {
 Logger.error("avcodec_open2 was not successful");
 finishHandler.accept(false);
 return;
 }
 av_dict_free(options);
 break;
 }
 }

 if (this.decoder == null || this.m_VideoDecoderCtx == null) {
 finishHandler.accept(false);
 return;
 }
 finishHandler.accept(true);
 }

 private AVHWContextInfo createHardwareContext() {
 AVHWContextInfo result = null;
 for (int i = 0; ; i++) {
 final AVCodecHWConfig config = avcodec_get_hw_config(this.decoder, i);
 if (config == null) {
 break;
 }

 if ((config.methods() & AV_CODEC_HW_CONFIG_METHOD_HW_DEVICE_CTX) < 0) {
 continue;
 }
 final int device_type = config.device_type();
 if (device_type != App.HW_DEVICE_TYPE) {
 continue;
 }
 final AVBufferRef hw_context = av_hwdevice_ctx_alloc(device_type);
 if (hw_context == null || av_hwdevice_ctx_create(hw_context, device_type, (String) null, null, 0) < 0) {
 Logger.error("HW accel not supported for type {}", device_type);
 av_free(config);
 av_free(hw_context);
 } else {
 Logger.info("HW accel created for type {}", device_type);
 result = new AVHWContextInfo(config, hw_context);
 }
 break;
 }

 return result;
 }

 @Override
 public void stop() {
 this.releaseNativeResources();
 }

 /************************/
 /*** video processing ***/
 /************************/


 private void performTestFramesFeeding() {
 final AVPacket pkt = av_packet_alloc();
 if (pkt == null) {
 return;
 }
 try (final BytePointer bp = new BytePointer(65_535 * 4)) {
 final byte[] frameData = AVTestFrames.h264KeyTestFrame;


 bp.position(0);

 bp.put(frameData);
 bp.limit(frameData.length);

 pkt.data(bp);
 pkt.capacity(bp.capacity());
 pkt.size(frameData.length);
 pkt.position(0);
 pkt.limit(frameData.length);
 final AVFrame avFrame = av_frame_alloc();

 final int err = avcodec_send_packet(this.m_VideoDecoderCtx, pkt); // this will fail with D3D11VA and DXVA2
 if (err < 0) {
 final BytePointer buffer = new BytePointer(512);
 av_strerror(err, buffer, buffer.capacity());
 final String string = buffer.getString();
 System.out.println("Error on decoding test frame " + err + " message " + string);
 av_frame_free(avFrame);
 return;
 }

 final int result = avcodec_receive_frame(this.m_VideoDecoderCtx, avFrame);
 final AVFrame decodedFrame;
 if (result == 0) {
 if (this.m_VideoDecoderCtx.hw_device_ctx() == null) {
 decodedFrame = avFrame;
 av_frame_unref(decodedFrame);
 System.out.println("SUCESS with SW decoding");
 } else {
 final AVFrame hwAvFrame = av_frame_alloc();
 if (av_hwframe_transfer_data(hwAvFrame, avFrame, 0) < 0) {
 System.out.println("Failed to transfer frame from hardware");
 av_frame_unref(hwAvFrame);
 decodedFrame = avFrame;
 } else {
 av_frame_unref(avFrame);
 decodedFrame = hwAvFrame;
 System.out.println("SUCESS with HW decoding");
 }
 av_frame_unref(decodedFrame);
 }
 } else {
 final BytePointer buffer = new BytePointer(512);
 av_strerror(result, buffer, buffer.capacity());
 final String string = buffer.getString();
 System.out.println("error " + result + " message " + string);
 av_frame_free(avFrame);
 }
 } finally {
 if (pkt.stream_index() != -1) {
 av_packet_unref(pkt);
 }
 pkt.releaseReference();
 }
 }

 final Object releaseLock = new Object();
 private volatile boolean released = false;

 private void releaseNativeResources() {
 if (this.released) {
 return;
 }
 this.released = true;
 synchronized (this.releaseLock) {
 // Close the video codec
 if (this.m_VideoDecoderCtx != null) {
 avcodec_free_context(this.m_VideoDecoderCtx);
 this.m_VideoDecoderCtx = null;
 }

 // close the format callback
 if (this.formatCallback != null) {
 this.formatCallback.close();
 this.formatCallback = null;
 }

 // close hw context
 if (this.hardwareContext != null) {
 this.hardwareContext.free();
 }
 }
 }

 private int getDefaultPixelFormat() {
 return AV_PIX_FMT_YUV420P; // Always return yuv420p here
 }

 public static final class HexUtil {
 private static final char[] hexArray = "0123456789ABCDEF".toCharArray();

 private HexUtil() {
 }

 public static String hexlify(final byte[] bytes) {
 final char[] hexChars = new char[bytes.length * 2];

 for (int j = 0; j < bytes.length; ++j) {
 final int v = bytes[j] & 255;
 hexChars[j * 2] = HexUtil.hexArray[v >>> 4];
 hexChars[j * 2 + 1] = HexUtil.hexArray[v & 15];
 }

 return new String(hexChars);
 }

 public static byte[] unhexlify(final String argbuf) {
 final int arglen = argbuf.length();
 if (arglen % 2 != 0) {
 throw new RuntimeException("Odd-length string");
 } else {
 final byte[] retbuf = new byte[arglen / 2];

 for (int i = 0; i < arglen; i += 2) {
 final int top = Character.digit(argbuf.charAt(i), 16);
 final int bot = Character.digit(argbuf.charAt(i + 1), 16);
 if (top == -1 || bot == -1) {
 throw new RuntimeException("Non-hexadecimal digit found");
 }

 retbuf[i / 2] = (byte) ((top << 4) + bot);
 }

 return retbuf;
 }
 }
 }

 public static final class AVHWContextInfo {
 private final AVCodecHWConfig hwConfig;
 private final AVBufferRef hwContext;

 private volatile boolean freed = false;

 public AVHWContextInfo(final AVCodecHWConfig hwConfig, final AVBufferRef hwContext) {
 this.hwConfig = hwConfig;
 this.hwContext = hwContext;
 }

 public AVCodecHWConfig hwConfig() {
 return this.hwConfig;
 }

 public AVBufferRef hwContext() {
 return this.hwContext;
 }

 public void free() {
 if (this.freed) {
 return;
 }
 this.freed = true;
 av_free(this.hwConfig);
 av_free(this.hwContext);
 }


 @Override
 public boolean equals(Object o) {
 if (this == o) return true;
 if (o == null || getClass() != o.getClass()) return false;
 AVHWContextInfo that = (AVHWContextInfo) o;
 return freed == that.freed && Objects.equals(hwConfig, that.hwConfig) && Objects.equals(hwContext, that.hwContext);
 }

 @Override
 public int hashCode() {
 return Objects.hash(hwConfig, hwContext, freed);
 }

 @Override
 public String toString() {
 return "AVHWContextInfo[" +
 "hwConfig=" + this.hwConfig + ", " +
 "hwContext=" + this.hwContext + ']';
 }

 }

 public static final class AVTestFrames {

 private AVTestFrames() {

 }

 static {
 InputStream inputStream = null;
 try {
 inputStream = AVTestFrames.class.getClassLoader().getResourceAsStream("h264_test_key_frame.txt");
 final byte[] h264TestFrameBuffer = inputStream == null ? new byte[0] : inputStream.readAllBytes();
 final String h264TestFrame = new String(h264TestFrameBuffer, StandardCharsets.UTF_8);
 AVTestFrames.h264KeyTestFrame = HexUtil.unhexlify(h264TestFrame);
 } catch (final IOException e) {
 Logger.error(e, "Could not parse test frame");
 } finally {
 if (inputStream != null) {
 try {
 inputStream.close();
 inputStream = null;
 } catch (final IOException e) {
 Logger.error(e, "Could not close test frame input stream");
 }
 }
 }
 }

 public static byte[] h264KeyTestFrame;
 }
}
</boolean>

The build gradle of the project looks like this


plugins {
 id 'application'
 id 'org.openjfx.javafxplugin' version '0.0.13'
}

group 'com.test.example'
version '1.0.0'

repositories {
 mavenCentral()
 mavenLocal()
 maven { url 'https://jitpack.io' }
}

dependencies {
 implementation group: 'org.bytedeco', name: 'javacv-platform', version: '1.5.8'
 implementation group: 'com.github.oshi', name: 'oshi-core', version: '3.4.3'
 implementation 'org.tinylog:tinylog-api:2.1.0'
 implementation 'org.tinylog:tinylog-impl:2.1.0'
 implementation 'org.jcodec:jcodec:0.2.5'
}

test {
 useJUnitPlatform()
}

javafx {
 version = '17.0.6'
 modules = ['javafx.graphics', 'javafx.controls', 'javafx.fxml', 'javafx.base']
}

mainClassName = 'com.test.example.App'


Syndiquer tout le site
