Recherche avancée

Sur d’autres sites (9319)

• FFmpeg C audio video streams (mic, webcam) sync to mp4

  22 février 2020, par NIAZ

  I am trying to capture audio and video using a microphone and a webcam into an mp4 file. The recorded file is playable but over the time the audio starts drifting away from video and for a longer period of time the gap increases. Both the audio and video is handled in separate threads, for the audio I am using audiofifo adapted from the transcode_acc.c example https://github.com/FFmpeg/FFmpeg/blob/master/doc/examples/transcode_aac.c
  Here is how I am setting up the streams
  Video :

  video_output_codec_ctx = video_stream->codec;
  
  video_output_codec_ctx->bit_rate = 2000000;
  
  video_output_codec_ctx->codec_id = AV_CODEC_ID_MPEG4;
  
  video_output_codec_ctx->width = 640;
  
  video_output_codec_ctx->height = 480;
  
  video_stream->time_base = (AVRational){1, fps};
  
  video_output_codec_ctx->time_base = video_stream->time_base;
  
  video_output_codec_ctx->pix_fmt = AV_PIX_FMT_YUV420P;
  
  video_output_codec_ctx->codec_type = AVMEDIA_TYPE_VIDEO;

  Audio :

  audio_output_codec_ctx->channels       = OUTPUT_CHANNELS; // 2
  
  audio_output_codec_ctx->channel_layout = av_get_default_channel_layout(OUTPUT_CHANNELS);
  
  audio_output_codec_ctx->sample_rate    = audio_input_codec_ctx->sample_rate;
  
  audio_output_codec_ctx->sample_fmt     = audio_output_codec->sample_fmts[0];
  
  audio_output_codec_ctx->bit_rate       = OUTPUT_BIT_RATE; // 96000
  
  audio_output_codec_ctx->strict_std_compliance = FF_COMPLIANCE_EXPERIMENTAL;
  
  
  
  /* Set the sample rate for the container. */
  
  audio_stream->time_base.den = audio_input_codec_ctx->sample_rate;
  
  audio_stream->time_base.num = 1;

  For the video pts, I increment the index by 1 once the frame is encoded and before sending the frame to the encoder I use rescale and also after receiving the frame,afterwards the packets are written via av_interleaved_write_frame().

  output_frame->pts = av_rescale_q(video_frame_index, video_output_codec_ctx->time_base, video_input_format_ctx->streams[0]->time_base);
  
  
  
  error = avcodec_send_frame(video_output_codec_ctx, output_frame);
  
  
  
  error = avcodec_receive_packet(video_output_codec_ctx, &output_packet);
  
  
  
  output_packet.stream_index = video_index;
  
  
  
  output_packet.pts = av_rescale_q_rnd(output_packet.pts, video_input_format_ctx->streams[0]->time_base, output_format_context->streams[video_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.dts = av_rescale_q_rnd(output_packet.dts, video_input_format_ctx->streams[0]->time_base, output_format_context->streams[video_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.duration = ((output_format_context->streams[0]->time_base.den / output_format_context->streams[0]->time_base.num) / video_output_codec_ctx->time_base.den);
  
  
  
  output_packet.pos = -1;  
  
  
  
  video_frame_index++;

  For the audio pts, I increment by frame->nb_samples once the frame is encoded I use rescale, afterwards the packets are written via av_interleaved_write_frame().

  frame->pts = aud_pts;
  
  aud_pts += frame->nb_samples;
  
  
  
  error = avcodec_send_frame(audio_output_codec_ctx, frame);
  
  
  
  error = avcodec_receive_packet(audio_output_codec_ctx, &output_packet);
  
  
  
  output_packet.stream_index = audio_index;
  
  
  
  output_packet.pts = av_rescale_q_rnd(output_packet.pts, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.dts = av_rescale_q_rnd(output_packet.dts, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.duration = av_rescale_q(output_packet.duration, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base);

  I am new to the FFmpeg C API, have tried various resources / posts over the internet but still haven’t been able to sync the audio and video in a robust way. Here are few question that I want to understand which will help me to resolve this. Any thoughts are really appreciated on this.

  1. Can the FFmpeg C API handle sync internally or this is something which needs to be handled from the caller side ?

  2. Am I setting up the PTS correctly both for audio and video in the first place ? I have noticed that when I use fps lower than 20, I get Invalid pts (66667) <= last (66667) Operation not permitted from the encoder. This must be something wrong with the way I am currently setting the video PTS. How can I set up the video PTS to handle lower fps ?

  3. I am also trying to adopt the idea of clocks sync from dranger’s tutorial, not sure whether this would be suitable for my use case, things like where to setup the audio and video clocks as he used only decoders, for the audio I am using fifo and not sure how to adjust the samples based on the clocks sync, also the way to call and setup the refresh timer ?

  4. Is there a better mechanism for creating a robust sync for my use case which can handle both audio and video if they go out of sync, would be great to have an idea about the samples and frames adjustment based on that ?

• FFmpeg C/C++ audio video streams (mic, webcam) sync to mp4

  21 février 2020, par NIAZ

  I am trying to capture audio and video using a microphone and a webcam into an mp4 file. The recorded file is playable but over the time the audio starts drifting away from video and for a longer period of time the gap increases. Both the audio and video is handled in separate threads, for the audio I am using audiofifo adapted from the transcode_acc.c example https://github.com/FFmpeg/FFmpeg/blob/master/doc/examples/transcode_aac.c
  Here is how I am setting up the streams
  Video :

  video_output_codec_ctx = video_stream->codec;
  
  video_output_codec_ctx->bit_rate = 2000000;
  
  video_output_codec_ctx->codec_id = AV_CODEC_ID_MPEG4;
  
  video_output_codec_ctx->width = 640;
  
  video_output_codec_ctx->height = 480;
  
  video_stream->time_base = (AVRational){1, fps};
  
  video_output_codec_ctx->time_base = video_stream->time_base;
  
  video_output_codec_ctx->pix_fmt = AV_PIX_FMT_YUV420P;
  
  video_output_codec_ctx->codec_type = AVMEDIA_TYPE_VIDEO;

  Audio :

  audio_output_codec_ctx->channels       = OUTPUT_CHANNELS; // 2
  
  audio_output_codec_ctx->channel_layout = av_get_default_channel_layout(OUTPUT_CHANNELS);
  
  audio_output_codec_ctx->sample_rate    = audio_input_codec_ctx->sample_rate;
  
  audio_output_codec_ctx->sample_fmt     = audio_output_codec->sample_fmts[0];
  
  audio_output_codec_ctx->bit_rate       = OUTPUT_BIT_RATE; // 96000
  
  audio_output_codec_ctx->strict_std_compliance = FF_COMPLIANCE_EXPERIMENTAL;
  
  
  
  /* Set the sample rate for the container. */
  
  audio_stream->time_base.den = audio_input_codec_ctx->sample_rate;
  
  audio_stream->time_base.num = 1;

  For the video pts, I increment the index by 1 once the frame is encoded and before sending the frame to the encoder I use rescale and also after receiving the frame,afterwards the packets are written via av_interleaved_write_frame().

  output_frame->pts = av_rescale_q(video_frame_index, video_output_codec_ctx->time_base, video_input_format_ctx->streams[0]->time_base);
  
  
  
  error = avcodec_send_frame(video_output_codec_ctx, output_frame);
  
  
  
  error = avcodec_receive_packet(video_output_codec_ctx, &output_packet);
  
  
  
  output_packet.stream_index = video_index;
  
  
  
  output_packet.pts = av_rescale_q_rnd(output_packet.pts, video_input_format_ctx->streams[0]->time_base, output_format_context->streams[video_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.dts = av_rescale_q_rnd(output_packet.dts, video_input_format_ctx->streams[0]->time_base, output_format_context->streams[video_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.duration = ((output_format_context->streams[0]->time_base.den / output_format_context->streams[0]->time_base.num) / video_output_codec_ctx->time_base.den);
  
  
  
  output_packet.pos = -1;  
  
  
  
  video_frame_index++;

  For the audio pts, I increment by frame->nb_samples once the frame is encoded I use rescale, afterwards the packets are written via av_interleaved_write_frame().

  frame->pts = aud_pts;
  
  aud_pts += frame->nb_samples;
  
  
  
  error = avcodec_send_frame(audio_output_codec_ctx, frame);
  
  
  
  error = avcodec_receive_packet(audio_output_codec_ctx, &output_packet);
  
  
  
  output_packet.stream_index = audio_index;
  
  
  
  output_packet.pts = av_rescale_q_rnd(output_packet.pts, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.dts = av_rescale_q_rnd(output_packet.dts, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
  
  
  
  output_packet.duration = av_rescale_q(output_packet.duration, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base);

  I am new to the FFmpeg C API, have tried various resources / posts over the internet but still haven’t been able to sync the audio and video in a robust way. Here are few question that I want to understand which will help me to resolve this. Any thoughts are really appreciated on this.

  1. Can the FFmpeg C API handle sync internally or this is something which needs to be handled from the caller side ?

  2. Am I setting up the PTS correctly both for audio and video in the first place ? I have noticed that when I use fps lower than 20, I get Invalid pts (66667) <= last (66667) Operation not permitted from the encoder. This must be something wrong with the way I am currently setting the video PTS. How can I set up the video PTS to handle lower fps ?

  3. I am also trying to adopt the idea of clocks sync from dranger’s tutorial, not sure whether this would be suitable for my use case, things like where to setup the audio and video clocks as he used only decoders, for the audio I am using fifo and not sure how to adjust the samples based on the clocks sync, also the way to call and setup the refresh timer ?

  4. Is there a better mechanism for creating a robust sync for my use case which can handle both audio and video if they go out of sync, would be great to have an idea about the samples and frames adjustment based on that ?

• Revisiting Nosefart and Discovering GME

  30 mai 2011, par Multimedia Mike — Game Hacking

  I found the following screenshot buried deep in an old directory structure of mine :

  
  
  

  I tried to recall how this screenshot came to exist. Had I actually created a functional KDE frontend to Nosefart yet neglected to release it ? I think it’s more likely that I used some designer tool (possibly KDevelop) to prototype a frontend. This would have been sometime in 2000.

  However, this screenshot prompted me to revisit Nosefart.

  Nosefart Background
  Nosefart is a program that can play Nintendo Sound Format (NSF) files. NSF files are files containing components that were surgically separated from Nintendo Entertainment System (NES) ROM dumps. These components contain the music playback engines for various games. An NSF player is a stripped down emulation system that can simulate the NES6502 CPU along with the custom hardware (2 square waves, 1 triangle wave, 1 noise generator, and 1 limited digital channel).

  Nosefart was written by Matt Conte and eventually imported into a Sourceforge project, though it has not seen any development since then. The distribution contains standalone command line players for Linux and DOS, a GTK frontend for the Linux command line version, and plugins for Winamp, XMMS, and CL-Amp.

  The Sourceforge project page notes that Nosefart is also part of XBMC. Let the record show that Nosefart is also incorporated into xine (I did that in 2002, I think).

  Upgrading the API
  When I tried running the command line version of Nosefart under Linux, I hit hard against the legacy audio API : OSS. Remember that ?

  In fairly short order, I was able to upgrade the CL program to use PulseAudio. The program is not especially sophisticated. It’s a single-threaded affair which checks for a keypress, processes an audio frame, and sends the frame out to the OSS file interface. All that was needed was to rewrite open_hardware() and close_hardware() for PA and then replace the write statement in play(). The only quirk that stood out is that including <pulse/pulseaudio.h> is insufficient for programming PA’s simple API. <pulse/simple.h> must be included separately.

  For extra credit, I adapted the program to ALSA. The program uses the most simplistic audio output API possible — just keep filling a buffer and sending it out to the DAC.

  Discovering GME
  I’m not sure what to do with the the program now since, during my research to attempt to bring Nosefart up to date, I became aware of a software library named Game Music Emu, or GME. It’s a pure C++ library that can essentially play any classic video game format you can possible name. Wow. A lot can happen in 10 years when you’re not paying attention.

  It’s such a well-written library that I didn’t need any tutorial or documentation to come up to speed. Just a quick read of the main gme.h header library enabled me in short order to whip up a quick C program that could play NSF and SPC files. Path of least resistance : Client program asks library to open a hardcoded file, synthesize 10 seconds of audio, and dump it into a file ; ask the FLAC command line program to transcode raw data to .flac file ; use ffplay to verify the results.

  I might develop some other uses for this library.