Recherche avancée

Sur d’autres sites (5019)

• FFMPEG to create an MPEG-DASH stream with VP8

  24 avril 2017, par Kenneth Worden

  I’m trying to use FFMPEG to stream a live video feed from my webcam /dev/video0. Following scattered tutorials and scarce documentation (is this a known problem for the encoding community ?) I arrived at the following bash script :

  #!/bin/bash
  
  
  
  ffmpeg \
  
      -y \
  
      -f v4l2 \
  
          -i /dev/video0 \
  
          -s 640x480 \
  
          -input_format mjpeg \
  
          -r 24 \
  
      -map 0:0 \
  
      -pix_fmt yuv420p \
  
      -codec:v libvpx \
  
          -s 640x480 \
  
          -threads 4 \
  
          -b:v 50k \
  
          -tile-columns 4 \
  
          -frame-parallel 1 \
  
          -keyint_min 24 -g 24 \
  
      -f webm_chunk \
  
          -header "stream.hdr" \
  
          -chunk_start_index 1 \
  
      stream_%d.chk &
  
  
  
  sleep 2
  
  
  
  ffmpeg \
  
      -f webm_dash_manifest -live 1 \
  
      -i stream.hdr \
  
      -c copy \
  
      -map 0 \
  
      -f webm_dash_manifest -live 1 \
  
          -adaptation_sets "id=0,streams=0" \
  
          -chunk_start_index 1 \
  
          -chunk_duration_ms 1000 \
  
          -time_shift_buffer_depth 30000 \
  
          -minimum_update_period 60000 \
  
      stream_manifest.mpd

  When I run this script, my webcam light turns on, the stream.hdr and stream_manifest.mpd files are written, and chunks start to be created (i.e. stream_1.chk, stream_2.chk, etc...). However, FFMPEG throws the following error :

  Could not write header for output file #0 (incorrect codec parameters
   ?) : Invalid data found when processing input

  I will explain what I think I am doing with this script, and hopefully this will expose any errors in my thinking.

  First, we invoke FFMPEG to use Video for Linux 2 (v4l2) to read from my webcam (/dev/video0) of a resolution 640x480. The input format is mjpeg with a framerate of 24fps.

  I then declare that FFMPEG should "map" (copy) the video stream output by v4l2 to a file. I specify the pixel format (YUV420P) and use libvpx (VP8 encoding) to encode the video stream. I set the size to be 640x480, use 4 threads, set the bitrate to be 50kbps, do some magic with tile-columns and frame-parallel options, and set the I-frames to be 24 frames apart.

  I then create a stream.hdr file. The starting index is 1. This command continues to run infinitely until I kill it, grabbing new video from my webcam and outputting it into chunks.

  I then sleep for 2 seconds to give the previous command time to generate a header file.

  And that’s really it. The next invocation of FFMPEG simply creates the MPEG-DASH manifest file given the header generated in the previous step.

  So what’s going on ? Why can I not view the video in a web browser (I’m using Dash.js) ? I serve the manifest, header, and chunks on a Node.js server so that trivial issue is not the problem.

  ---

  Edit : Here is my full console output.

  ffmpeg version 3.0.7-0ubuntu0.16.10.1 Copyright (c) 2000-2017 the FFmpeg developers
  
    built with gcc 6.2.0 (Ubuntu 6.2.0-5ubuntu12) 20161005
  
    configuration: --prefix=/usr --extra-version=0ubuntu0.16.10.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librubberband --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-chromaprint --enable-libx264
  
    libavutil      55. 17.103 / 55. 17.103
  
    libavcodec     57. 24.102 / 57. 24.102
  
    libavformat    57. 25.100 / 57. 25.100
  
    libavdevice    57.  0.101 / 57.  0.101
  
    libavfilter     6. 31.100 /  6. 31.100
  
    libavresample   3.  0.  0 /  3.  0.  0
  
    libswscale      4.  0.100 /  4.  0.100
  
    libswresample   2.  0.101 /  2.  0.101
  
    libpostproc    54.  0.100 / 54.  0.100
  
  [video4linux2,v4l2 @ 0x55847e244ea0] The driver changed the time per frame from 1/24 to 1/30
  
  [mjpeg @ 0x55847e245c00] Changing bps to 8
  
  Input #0, video4linux2,v4l2, from '/dev/video0':
  
    Duration: N/A, start: 64305.102081, bitrate: N/A
  
      Stream #0:0: Video: mjpeg, yuvj422p(pc, bt470bg/unknown/unknown), 640x480, -5 kb/s, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
  
  Codec AVOption frame-parallel (Enable frame parallel decodability features) specified for output file #0 (stream_%d.chk) has not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video streams) or that it is a private option of some encoder which was not actually used for any stream.
  
  Codec AVOption tile-columns (Number of tile columns to use, log2) specified for output file #0 (stream_%d.chk) has not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video streams) or that it is a private option of some encoder which was not actually used for any stream.
  
  [swscaler @ 0x55847e24b720] deprecated pixel format used, make sure you did set range correctly
  
  [libvpx @ 0x55847e248a20] v1.5.0
  
  Output #0, webm_chunk, to 'stream_%d.chk':
  
    Metadata:
  
      encoder         : Lavf57.25.100
  
      Stream #0:0: Video: vp8 (libvpx), yuv420p, 640x480, q=-1--1, 50 kb/s, 30 fps, 30 tbn, 30 tbc
  
      Metadata:
  
        encoder         : Lavc57.24.102 libvpx
  
      Side data:
  
        unknown side data type 10 (24 bytes)
  
  Stream mapping:
  
    Stream #0:0 -> #0:0 (mjpeg (native) -> vp8 (libvpx))
  
  Press [q] to stop, [?] for help
  
  frame=   21 fps=0.0 q=0.0 size=N/A time=00:00:00.70 bitrate=N/A dup=5 drop=frame=   36 fps= 35 q=0.0 size=N/A time=00:00:01.20 bitrate=N/A dup=5 drop=frame=   51 fps= 33 q=0.0 size=N/A time=00:00:01.70 bitrate=N/A dup=5 drop=ffmpeg version 3.0.7-0ubuntu0.16.10.1 Copyright (c) 2000-2017 the FFmpeg developers
  
    built with gcc 6.2.0 (Ubuntu 6.2.0-5ubuntu12) 20161005
  
    configuration: --prefix=/usr --extra-version=0ubuntu0.16.10.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librubberband --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-chromaprint --enable-libx264
  
    libavutil      55. 17.103 / 55. 17.103
  
    libavcodec     57. 24.102 / 57. 24.102
  
    libavformat    57. 25.100 / 57. 25.100
  
    libavdevice    57.  0.101 / 57.  0.101
  
    libavfilter     6. 31.100 /  6. 31.100
  
    libavresample   3.  0.  0 /  3.  0.  0
  
    libswscale      4.  0.100 /  4.  0.100
  
    libswresample   2.  0.101 /  2.  0.101
  
    libpostproc    54.  0.100 / 54.  0.100
  
  Input #0, webm_dash_manifest, from 'stream.hdr':
  
    Metadata:
  
      encoder         : Lavf57.25.100
  
    Duration: N/A, bitrate: N/A
  
      Stream #0:0: Video: vp8, yuv420p, 640x480, SAR 1:1 DAR 4:3, 30 fps, 30 tbr, 1k tbn, 1k tbc (default)
  
      Metadata:
  
        webm_dash_manifest_file_name: stream.hdr
  
        webm_dash_manifest_track_number: 1
  
  Output #0, webm_dash_manifest, to 'stream_manifest.mpd':
  
    Metadata:
  
      encoder         : Lavf57.25.100
  
      Stream #0:0: Video: vp8, yuv420p, 640x480 [SAR 1:1 DAR 4:3], q=2-31, 30 fps, 30 tbr, 1k tbn, 1k tbc (default)
  
      Metadata:
  
        webm_dash_manifest_file_name: stream.hdr
  
        webm_dash_manifest_track_number: 1
  
  Stream mapping:
  
    Stream #0:0 -> #0:0 (copy)
  
  Could not write header for output file #0 (incorrect codec parameters ?): Invalid data found when processing input
  
  frame=   67 fps= 33 q=0.0 size
  
  frame=   82 fps= 32 q=0.0 size=N/A time=00:00:02.73 bitrate=N/A dup=5 drop=
  
  frame=   97 fps= 32 q=0.0 size=N/A time=00:00:03.23 bitrate=N/A dup=5 drop=
  
  frame=  112 fps= 32 q=0.0 size=N/A time=00:00:03.73 bitrate=N/A dup=5 ...

• Java.lang.UnsatisfiedLinkError : No implementation found for int [duplicate]

  28 mars 2017, par Muthukumar Subramaniam

  This question already has an answer here :

  • Android NDK C++ JNI (no implementation found for native…)
    
    6 answers
    

  I executed youtube watch me android application project. I just add some classes in my project and build with ndk. I got the error like

  java.lang.UnsatisfiedLinkError : No implementation found for int com.ephronsystem.mobilizerapp.Ffmpeg.encodeVideoFrame(byte[]) (tried Java_com_ephronsystem_mobilizerapp_Ffmpeg_encodeVideoFrame and Java_com_ephronsystem_mobilizerapp_Ffmpeg_encodeVideoFrame___3B).

  My code :

  package com.ephronsystem.mobilizerapp;
  
  
  
  public class Ffmpeg {
  
  
  
       static {
  
          System.loadLibrary("ffmpeg");
  
      }
  
  
  
      public static native boolean init(int width, int height, int audio_sample_rate, String rtmpUrl);
  
  
  
      public static native void shutdown();
  
  
  
      // Returns the size of the encoded frame.
  
      public static native int encodeVideoFrame(byte[] yuv_image);
  
  
  
      public static native int encodeAudioFrame(short[] audio_data, int length);
  
  }

  This is ffmpeg-jni.c

   #include <android></android>log.h>
  
  #include 
  
  #include 
  
  #include "libavcodec/avcodec.h"
  
  #include "libavformat/avformat.h"
  
  #include "libavutil/opt.h"
  
  
  
  #ifdef __cplusplus
  
  extern "C" {
  
  #endif
  
  
  
  JNIEXPORT jboolean JNICALL Java_com_ephronsystem_mobilizerapp_Ffmpeg_init(JNIEnv *env, jobject thiz,
  
                                                                   jint width, jint height,
  
                                                                   jint audio_sample_rate,
  
                                                                   jstring rtmp_url);
  
  JNIEXPORT void JNICALL Java_com_ephronsystem_mobilizerapp_Ffmpeg_shutdown(JNIEnv *env,
  
  jobject thiz
  
  );
  
  JNIEXPORT jint JNICALL Java_com_ephronsystem_mobilizerapp_Ffmpeg_encodeVideoFrame(JNIEnv
  
  *env,
  
  jobject thiz,
  
          jbyteArray
  
  yuv_image);
  
  JNIEXPORT jint JNICALL Java_com_ephronsystem_mobilizerapp_Ffmpeg_encodeAudioFrame(JNIEnv *env,
  
                                                                               jobject thiz,
  
                                                                               jshortArray audio_data,
  
                                                                               jint length);
  
  
  
  #ifdef __cplusplus
  
  }
  
  #endif
  
  
  
  #define LOGI(...) __android_log_print(ANDROID_LOG_INFO, "ffmpeg-jni", __VA_ARGS__)
  
  #define URL_WRONLY 2
  
          static AVFormatContext *fmt_context;
  
          static AVStream *video_stream;
  
          static AVStream *audio_stream;
  
  
  
          static int pts
  
  = 0;
  
  static int last_audio_pts = 0;
  
  
  
  // Buffers for UV format conversion
  
  static unsigned char *u_buf;
  
  static unsigned char *v_buf;
  
  
  
  static int enable_audio = 1;
  
  static int64_t audio_samples_written = 0;
  
  static int audio_sample_rate = 0;
  
  
  
  // Stupid buffer for audio samples. Not even a proper ring buffer
  
  #define AUDIO_MAX_BUF_SIZE 16384  // 2x what we get from Java
  
  static short audio_buf[AUDIO_MAX_BUF_SIZE];
  
  static int audio_buf_size = 0;
  
  
  
  void AudioBuffer_Push(const short *audio, int num_samples) {
  
      if (audio_buf_size >= AUDIO_MAX_BUF_SIZE - num_samples) {
  
          LOGI("AUDIO BUFFER OVERFLOW: %i + %i > %i", audio_buf_size, num_samples,
  
               AUDIO_MAX_BUF_SIZE);
  
          return;
  
      }
  
      for (int i = 0; i &lt; num_samples; i++) {
  
          audio_buf[audio_buf_size++] = audio[i];
  
      }
  
  }
  
  
  
  int AudioBuffer_Size() { return audio_buf_size; }
  
  
  
  short *AudioBuffer_Get() { return audio_buf; }
  
  
  
  void AudioBuffer_Pop(int num_samples) {
  
      if (num_samples > audio_buf_size) {
  
          LOGI("Audio buffer Pop WTF: %i vs %i", num_samples, audio_buf_size);
  
          return;
  
      }
  
      memmove(audio_buf, audio_buf + num_samples, num_samples * sizeof(short));
  
      audio_buf_size -= num_samples;
  
  }
  
  
  
  void AudioBuffer_Clear() {
  
      memset(audio_buf, 0, sizeof(audio_buf));
  
      audio_buf_size = 0;
  
  }
  
  
  
  static void log_callback(void *ptr, int level, const char *fmt, va_list vl) {
  
      char x[2048];
  
      vsnprintf(x, 2048, fmt, vl);
  
      LOGI(x);
  
  }
  
  
  
  JNIEXPORT jboolean JNICALL Java_com_ephronsystem_mobilizerapp_Ffmpeg_init(JNIEnv *env, jobject thiz,
  
                                                                   jint width, jint height,
  
                                                                   jint audio_sample_rate_param,
  
                                                                   jstring rtmp_url) {
  
      avcodec_register_all();
  
      av_register_all();
  
      av_log_set_callback(log_callback);
  
  
  
      fmt_context = avformat_alloc_context();
  
      AVOutputFormat *ofmt = av_guess_format("flv", NULL, NULL);
  
      if (ofmt) {
  
          LOGI("av_guess_format returned %s", ofmt->long_name);
  
      } else {
  
          LOGI("av_guess_format fail");
  
          return JNI_FALSE;
  
      }
  
  
  
      fmt_context->oformat = ofmt;
  
      LOGI("creating video stream");
  
      video_stream = av_new_stream(fmt_context, 0);
  
  
  
      if (enable_audio) {
  
          LOGI("creating audio stream");
  
          audio_stream = av_new_stream(fmt_context, 1);
  
      }
  
  
  
      // Open Video Codec.
  
      // ======================
  
      AVCodec *video_codec = avcodec_find_encoder(AV_CODEC_ID_H264);
  
      if (!video_codec) {
  
          LOGI("Did not find the video codec");
  
          return JNI_FALSE;  // leak!
  
      } else {
  
          LOGI("Video codec found!");
  
      }
  
      AVCodecContext *video_codec_ctx = video_stream->codec;
  
      video_codec_ctx->codec_id = video_codec->id;
  
      video_codec_ctx->codec_type = AVMEDIA_TYPE_VIDEO;
  
      video_codec_ctx->level = 31;
  
  
  
      video_codec_ctx->width = width;
  
      video_codec_ctx->height = height;
  
      video_codec_ctx->pix_fmt = PIX_FMT_YUV420P;
  
      video_codec_ctx->rc_max_rate = 0;
  
      video_codec_ctx->rc_buffer_size = 0;
  
      video_codec_ctx->gop_size = 12;
  
      video_codec_ctx->max_b_frames = 0;
  
      video_codec_ctx->slices = 8;
  
      video_codec_ctx->b_frame_strategy = 1;
  
      video_codec_ctx->coder_type = 0;
  
      video_codec_ctx->me_cmp = 1;
  
      video_codec_ctx->me_range = 16;
  
      video_codec_ctx->qmin = 10;
  
      video_codec_ctx->qmax = 51;
  
      video_codec_ctx->keyint_min = 25;
  
      video_codec_ctx->refs = 3;
  
      video_codec_ctx->trellis = 0;
  
      video_codec_ctx->scenechange_threshold = 40;
  
      video_codec_ctx->flags |= CODEC_FLAG_LOOP_FILTER;
  
      video_codec_ctx->me_method = ME_HEX;
  
      video_codec_ctx->me_subpel_quality = 6;
  
      video_codec_ctx->i_quant_factor = 0.71;
  
      video_codec_ctx->qcompress = 0.6;
  
      video_codec_ctx->max_qdiff = 4;
  
      video_codec_ctx->time_base.den = 10;
  
      video_codec_ctx->time_base.num = 1;
  
      video_codec_ctx->bit_rate = 3200 * 1000;
  
      video_codec_ctx->bit_rate_tolerance = 0;
  
      video_codec_ctx->flags2 |= 0x00000100;
  
  
  
      fmt_context->bit_rate = 4000 * 1000;
  
  
  
      av_opt_set(video_codec_ctx, "partitions", "i8x8,i4x4,p8x8,b8x8", 0);
  
      av_opt_set_int(video_codec_ctx, "direct-pred", 1, 0);
  
      av_opt_set_int(video_codec_ctx, "rc-lookahead", 0, 0);
  
      av_opt_set_int(video_codec_ctx, "fast-pskip", 1, 0);
  
      av_opt_set_int(video_codec_ctx, "mixed-refs", 1, 0);
  
      av_opt_set_int(video_codec_ctx, "8x8dct", 0, 0);
  
      av_opt_set_int(video_codec_ctx, "weightb", 0, 0);
  
  
  
      if (fmt_context->oformat->flags &amp; AVFMT_GLOBALHEADER)
  
          video_codec_ctx->flags |= CODEC_FLAG_GLOBAL_HEADER;
  
  
  
      LOGI("Opening video codec");
  
      AVDictionary *vopts = NULL;
  
      av_dict_set(&amp;vopts, "profile", "main", 0);
  
      //av_dict_set(&amp;vopts, "vprofile", "main", 0);
  
      av_dict_set(&amp;vopts, "rc-lookahead", 0, 0);
  
      av_dict_set(&amp;vopts, "tune", "film", 0);
  
      av_dict_set(&amp;vopts, "preset", "ultrafast", 0);
  
      av_opt_set(video_codec_ctx->priv_data, "tune", "film", 0);
  
      av_opt_set(video_codec_ctx->priv_data, "preset", "ultrafast", 0);
  
      av_opt_set(video_codec_ctx->priv_data, "tune", "film", 0);
  
      int open_res = avcodec_open2(video_codec_ctx, video_codec, &amp;vopts);
  
      if (open_res &lt; 0) {
  
          LOGI("Error opening video codec: %i", open_res);
  
          return JNI_FALSE;   // leak!
  
      }
  
  
  
      // Open Audio Codec.
  
      // ======================
  
  
  
      if (enable_audio) {
  
          AudioBuffer_Clear();
  
          audio_sample_rate = audio_sample_rate_param;
  
          AVCodec *audio_codec = avcodec_find_encoder(AV_CODEC_ID_AAC);
  
          if (!audio_codec) {
  
              LOGI("Did not find the audio codec");
  
              return JNI_FALSE;  // leak!
  
          } else {
  
              LOGI("Audio codec found!");
  
          }
  
          AVCodecContext *audio_codec_ctx = audio_stream->codec;
  
          audio_codec_ctx->codec_id = audio_codec->id;
  
          audio_codec_ctx->codec_type = AVMEDIA_TYPE_AUDIO;
  
          audio_codec_ctx->bit_rate = 128000;
  
          audio_codec_ctx->bit_rate_tolerance = 16000;
  
          audio_codec_ctx->channels = 1;
  
          audio_codec_ctx->profile = FF_PROFILE_AAC_LOW;
  
          audio_codec_ctx->sample_fmt = AV_SAMPLE_FMT_FLT;
  
          audio_codec_ctx->sample_rate = 44100;
  
  
  
          LOGI("Opening audio codec");
  
          AVDictionary *opts = NULL;
  
          av_dict_set(&amp;opts, "strict", "experimental", 0);
  
          open_res = avcodec_open2(audio_codec_ctx, audio_codec, &amp;opts);
  
          LOGI("audio frame size: %i", audio_codec_ctx->frame_size);
  
  
  
          if (open_res &lt; 0) {
  
              LOGI("Error opening audio codec: %i", open_res);
  
              return JNI_FALSE;   // leak!
  
          }
  
      }
  
  
  
      const jbyte *url = (*env)->GetStringUTFChars(env, rtmp_url, NULL);
  
  
  
      // Point to an output file
  
      if (!(ofmt->flags &amp; AVFMT_NOFILE)) {
  
          if (avio_open(&amp;fmt_context->pb, url, URL_WRONLY) &lt; 0) {
  
              LOGI("ERROR: Could not open file %s", url);
  
              return JNI_FALSE;  // leak!
  
          }
  
      }
  
      (*env)->ReleaseStringUTFChars(env, rtmp_url, url);
  
  
  
      LOGI("Writing output header.");
  
      // Write file header
  
      if (avformat_write_header(fmt_context, NULL) != 0) {
  
          LOGI("ERROR: av_write_header failed");
  
          return JNI_FALSE;
  
      }
  
  
  
      pts = 0;
  
      last_audio_pts = 0;
  
      audio_samples_written = 0;
  
  
  
      // Initialize buffers for UV format conversion
  
      int frame_size = video_codec_ctx->width * video_codec_ctx->height;
  
      u_buf = (unsigned char *) av_malloc(frame_size / 4);
  
      v_buf = (unsigned char *) av_malloc(frame_size / 4);
  
  
  
      LOGI("ffmpeg encoding init done");
  
      return JNI_TRUE;
  
  }
  
  
  
  JNIEXPORT void JNICALL Java_com_ephronsystem_mobilizerapp_Ffmpeg_shutdown(JNIEnv
  
  *env,
  
  jobject thiz
  
  ) {
  
  av_write_trailer(fmt_context);
  
  avio_close(fmt_context
  
  ->pb);
  
  avcodec_close(video_stream
  
  ->codec);
  
  if (enable_audio) {
  
  avcodec_close(audio_stream
  
  ->codec);
  
  }
  
  av_free(fmt_context);
  
  av_free(u_buf);
  
  av_free(v_buf);
  
  
  
  fmt_context = NULL;
  
  u_buf = NULL;
  
  v_buf = NULL;
  
  }
  
  
  
  JNIEXPORT jint JNICALL Java_com_ephronsystem_mobilizerapp_Ffmpeg_encodeVideoFrame(JNIEnv
  
  *env,
  
  jobject thiz,
  
          jbyteArray
  
  yuv_image) {
  
  int yuv_length = (*env)->GetArrayLength(env, yuv_image);
  
  unsigned char *yuv_data = (*env)->GetByteArrayElements(env, yuv_image, 0);
  
  
  
  AVCodecContext *video_codec_ctx = video_stream->codec;
  
  //LOGI("Yuv size: %i w: %i h: %i", yuv_length, video_codec_ctx->width, video_codec_ctx->height);
  
  
  
  int frame_size = video_codec_ctx->width * video_codec_ctx->height;
  
  
  
  const unsigned char *uv = yuv_data + frame_size;
  
  
  
  // Convert YUV from NV12 to I420. Y channel is the same so we don't touch it,
  
  // we just have to deinterleave UV.
  
  for (
  
  int i = 0;
  
  i &lt; frame_size / 4; i++) {
  
  v_buf[i] = uv[i * 2];
  
  u_buf[i] = uv[i * 2 + 1];
  
  }
  
  
  
  AVFrame source;
  
  memset(&amp;source, 0, sizeof(AVFrame));
  
  source.data[0] =
  
  yuv_data;
  
  source.data[1] =
  
  u_buf;
  
  source.data[2] =
  
  v_buf;
  
  source.linesize[0] = video_codec_ctx->
  
  width;
  
  source.linesize[1] = video_codec_ctx->width / 2;
  
  source.linesize[2] = video_codec_ctx->width / 2;
  
  
  
  // only for bitrate regulation. irrelevant for sync.
  
  source.
  
  pts = pts;
  
  pts++;
  
  
  
  int out_length = frame_size + (frame_size / 2);
  
  unsigned char *out = (unsigned char *) av_malloc(out_length);
  
  int compressed_length = avcodec_encode_video(video_codec_ctx, out, out_length, &amp;source);
  
  
  
  (*env)->
  
  ReleaseByteArrayElements(env, yuv_image, yuv_data,
  
  0);
  
  
  
  // Write to file too
  
  if (compressed_length > 0) {
  
  AVPacket pkt;
  
  av_init_packet(&amp;pkt);
  
  pkt.
  
  pts = last_audio_pts;
  
  if (video_codec_ctx->coded_frame &amp;&amp; video_codec_ctx->coded_frame->key_frame) {
  
  pkt.flags |= 0x0001;
  
  }
  
  pkt.
  
  stream_index = video_stream->index;
  
  pkt.
  
  data = out;
  
  pkt.
  
  size = compressed_length;
  
  if (
  
  av_interleaved_write_frame(fmt_context,
  
  &amp;pkt) != 0) {
  
  LOGI("Error writing video frame");
  
  }
  
  } else {
  
  LOGI("??? compressed_length &lt;= 0");
  
  }
  
  
  
  last_audio_pts++;
  
  
  
  av_free(out);
  
  return
  
  compressed_length;
  
  }
  
  
  
  JNIEXPORT jint JNICALL Java_com_ephronsystem_mobilizerapp_Ffmpeg_encodeAudioFrame(JNIEnv
  
  *env,
  
  jobject thiz,
  
          jshortArray
  
  audio_data,
  
  jint length
  
  ) {
  
  if (!enable_audio) {
  
  return 0;
  
  }
  
  
  
  short *audio = (*env)->GetShortArrayElements(env, audio_data, 0);
  
  //LOGI("java audio buffer size: %i", length);
  
  
  
  AVCodecContext *audio_codec_ctx = audio_stream->codec;
  
  
  
  unsigned char *out = av_malloc(128000);
  
  
  
  AudioBuffer_Push(audio, length
  
  );
  
  
  
  int total_compressed = 0;
  
  while (
  
  
  
  AudioBuffer_Size()
  
  
  
  >= audio_codec_ctx->frame_size) {
  
  AVPacket pkt;
  
  av_init_packet(&amp;pkt);
  
  
  
  int compressed_length = avcodec_encode_audio(audio_codec_ctx, out, 128000,
  
                                               AudioBuffer_Get());
  
  
  
  total_compressed +=
  
  compressed_length;
  
  audio_samples_written += audio_codec_ctx->
  
  frame_size;
  
  
  
  int new_pts = (audio_samples_written * 1000) / audio_sample_rate;
  
  if (compressed_length > 0) {
  
  pkt.
  
  size = compressed_length;
  
  pkt.
  
  pts = new_pts;
  
  last_audio_pts = new_pts;
  
  //LOGI("audio_samples_written: %i  comp_length: %i   pts: %i", (int)audio_samples_written, (int)compressed_length, (int)new_pts);
  
  pkt.flags |= 0x0001;
  
  pkt.
  
  stream_index = audio_stream->index;
  
  pkt.
  
  data = out;
  
  if (
  
  av_interleaved_write_frame(fmt_context,
  
  &amp;pkt) != 0) {
  
  LOGI("Error writing audio frame");
  
  }
  
  }
  
  AudioBuffer_Pop(audio_codec_ctx
  
  ->frame_size);
  
  }
  
  
  
  (*env)->
  
  ReleaseShortArrayElements(env, audio_data, audio,
  
  0);
  
  
  
  av_free(out);
  
  return
  
  total_compressed;
  
  }

• Audio/Video encoding with ffmpeg

  16 mars 2017, par Vroz

  Audio/Video encoding with ffmpeg :

  I am trying to create an avi file with encoded video and audio, using ffmpeg.

  First, I create the file :

              //define BITRATE 10000000
  
              //define GOP 300
  
              //define FPS 60
  
              //define VIDEOTYPE "avi"
  
  
  
              if (!encoder_->createFile(QFileInfo(*(videoFile_.data())).absoluteFilePath(), targetRect.width(), targetRect.height(), BITRATE*(1000 / FPS), GOP, 1000))    

  The buffers are initialized as :

              audio_outbuf_size = 44100 * 0.005 * 16; //5ms of audio should be encoded, each time this function is called
  
              audio_outbuf = new uint8_t[audio_outbuf_size];
  
  
  
              outbuf_size = getWidth()*getHeight() * 3;        
  
              outbuf = new uint8_t[outbuf_size];

  Then add audio and video streams (audio : CODEC_ID_PCM_S16LE, 16000 kb/s and 44100 Hz, video : PIX_FMT_YUV420P)

              void MediaMuxer::addAudioStream(QString fileName, ffmpeg::CodecID codec_id)
  
              {
  
                  // Add the audio stream
  
                  ffmpeg::AVCodec *encoder = avcodec_find_encoder(codec_id);
  
                  pAudioStream_ = ffmpeg::av_new_stream(pOutputFormatCtx_, 0);
  
                  if (!pAudioStream_) {
  
                      printf("Could not allocate stream\n");
  
                      return;
  
                  }
  
  
  
                  pAudioCodecCtx_ = pAudioStream_->codec;
  
                  pAudioCodecCtx_->codec_id = codec_id;
  
                  pAudioCodecCtx_->codec_type = ffmpeg::AVMEDIA_TYPE_AUDIO;
  
                  pAudioCodecCtx_->sample_fmt = ffmpeg::AV_SAMPLE_FMT_S16;
  
                  pAudioCodecCtx_->sample_fmt = encoder->sample_fmts[0];
  
                  pAudioCodecCtx_->bit_rate = 16000;
  
                  //pAudioCodecCtx_->bit_rate = 64000;
  
                  pAudioCodecCtx_->sample_rate = N;
  
                  pAudioCodecCtx_->channels = 1;
  
  
  
                  pAudioCodecCtx_->time_base.den = FPS;
  
                  pAudioCodecCtx_->time_base.num = 1;
  
  
  
                  avcodec_thread_init(pAudioCodecCtx_, 10);
  
  
  
                  // some formats want stream headers to be separate
  
                  if (pOutputFormatCtx_->oformat->flags & AVFMT_GLOBALHEADER)
  
                      pAudioCodecCtx_->flags |= CODEC_FLAG_GLOBAL_HEADER;
  
  
  
                  if (av_set_parameters(pOutputFormatCtx_, NULL) < 0)
  
                  {
  
                      printf("Invalid output format parameters\n");
  
                      return;
  
                  }
  
  
  
                  //ffmpeg::dump_format(pOutputFormatCtx_, 0, fileName.toStdString().c_str(), 1);
  
  
  
                  // open_video
  
                  // find the audio encoder
  
                  pAudioCodec_ = avcodec_find_encoder(pAudioCodecCtx_->codec_id);
  
                  if (!pAudioCodec_)
  
                  {
  
                      printf("codec not found\n");
  
                      return;
  
                  }
  
                  // open the codec
  
                  if (avcodec_open(pAudioCodecCtx_, pAudioCodec_) < 0)
  
                  {
  
                      printf("could not open codec\n");
  
                      return;
  
                  }
  
  
  
                  // Allocate memory for output
  
                  if (!initAudioOutputBuf())
  
                  {
  
                      printf("Can't allocate memory for audio output bitstream\n");
  
                      return;
  
                  }
  
  
  
                  // Allocate the audio frame
  
                  if (!initAudioFrame())
  
                  {
  
                      printf("Can't init audio frame\n");
  
                      return;
  
                  }
  
  
  
                  if (url_fopen(&pOutputFormatCtx_->pb, fileName.toStdString().c_str(), URL_WRONLY) < 0)
  
                  {
  
                      printf("Could not open '%s'\n", fileName.toStdString().c_str());
  
                      return;
  
                  }
  
                  av_write_header(pOutputFormatCtx_);
  
              }
  
  
  
              void MediaMuxer::addVideoStream(QString fileName)
  
              {
  
                  // Add the video stream
  
                  pVideoStream_ = ffmpeg::av_new_stream(pOutputFormatCtx_, 0);
  
                  if (!pVideoStream_)
  
                  {
  
                      printf("Could not allocate stream\n");
  
                      return;
  
                  }
  
  
  
                  pVideoCodecCtx_ = pVideoStream_->codec;
  
                  pVideoCodecCtx_->codec_id = pOutputFormat_->video_codec;
  
                  pVideoCodecCtx_->codec_type = ffmpeg::AVMEDIA_TYPE_VIDEO;
  
  
  
                  pVideoCodecCtx_->bit_rate = Bitrate;
  
                  pVideoCodecCtx_->width = getWidth();
  
                  pVideoCodecCtx_->height = getHeight();
  
                  pVideoCodecCtx_->time_base.den = FPS;
  
                  pVideoCodecCtx_->time_base.num = 1;
  
                  pVideoCodecCtx_->gop_size = Gop;
  
                  pVideoCodecCtx_->pix_fmt = ffmpeg::PIX_FMT_YUV420P;
  
  
  
                  avcodec_thread_init(pVideoCodecCtx_, 10);
  
  
  
                  // some formats want stream headers to be separate
  
                  if (pOutputFormatCtx_->oformat->flags & AVFMT_GLOBALHEADER)
  
                      pVideoCodecCtx_->flags |= CODEC_FLAG_GLOBAL_HEADER;
  
  
  
  
  
                  if (av_set_parameters(pOutputFormatCtx_, NULL) < 0)
  
                  {
  
                      printf("Invalid output format parameters\n");
  
                      return;
  
                  }
  
  
  
                  //ffmpeg::dump_format(pOutputFormatCtx_, 0, fileName.toStdString().c_str(), 1);
  
  
  
                  // open_video
  
                  // find the video encoder
  
                  pVideoCodec_ = avcodec_find_encoder(pVideoCodecCtx_->codec_id);
  
                  if (!pVideoCodec_)
  
                  {
  
                      printf("codec not found\n");
  
                      return;
  
                  }
  
                  // open the codec
  
                  if (avcodec_open(pVideoCodecCtx_, pVideoCodec_) < 0)
  
                  {
  
                      printf("could not open codec\n");
  
                      return;
  
                  }
  
  
  
                  // Allocate memory for output
  
                  if (!initOutputBuf())
  
                  {
  
                      printf("Can't allocate memory for output bitstream\n");
  
                      return;
  
                  }
  
  
  
                  // Allocate the YUV frame
  
                  if (!initFrame())
  
                  {
  
                      printf("Can't init frame\n");
  
                      return;
  
                  }
  
  
  
                  if (url_fopen(&pOutputFormatCtx_->pb, fileName.toStdString().c_str(), URL_WRONLY) < 0)
  
                  {
  
                      printf("Could not open '%s'\n", fileName.toStdString().c_str());
  
                      return;
  
                  }
  
                  av_write_header(pOutputFormatCtx_);
  
              }

  Finally, I call alternatively encodeVideo/encodeAudio to encode video and PCM audio frames at specific recording times(pts) :

              int MediaMuxer::encodeVideo(const QImage &img, unsigned pts)
  
              {
  
                  convertImage_sws(img);     // SWS conversion
  
                  pVideoCodecCtx_->coded_frame->pts = pts;  // Set the time stamp
  
                  int out_size = ffmpeg::avcodec_encode_video(pVideoCodecCtx_, outbuf, outbuf_size, ppicture);        
  
                  pVideoCodecCtx_->coded_frame->pts = pts;  // Set the time stamp
  
  
  
                  if (out_size > 0)
  
                  {
  
                      ffmpeg::av_init_packet(&pkt);       
  
                      if (pVideoCodecCtx_->coded_frame->pts != (0x8000000000000000LL))
  
                          pkt.pts = av_rescale_q(pVideoCodecCtx_->coded_frame->pts, pVideoCodecCtx_->time_base, pVideoStream_->time_base);
  
                      if (pVideoCodecCtx_->coded_frame->key_frame)
  
                          pkt.flags |= AV_PKT_FLAG_KEY;
  
  
  
                      pkt.stream_index = pVideoStream_->index;
  
                      pkt.data = outbuf;
  
                      pkt.size = out_size;
  
                      int ret = ffmpeg::av_interleaved_write_frame(pOutputFormatCtx_, &pkt);      
  
                      if (ret<0)
  
                          return -1;
  
  
  
                  }
  
                  return out_size;
  
              }
  
  
  
              int MediaMuxer::encodeAudio(unsigned pts)
  
              {
  
                  pAudioCodecCtx_->coded_frame->pts = pts;  // Set the time stamp
  
  
  
                  // simple sound encoding    
  
                  int16_t samples[220] = { 0 }; // buffer
  
                  int n;                // buffer index
  
                  double Fs = 44100.0;  // sampling frequency
  
  
  
                  // Generate audio data
  
                  for (n = 0; n < 220; ++n)   //220 samples (44100*.005sec as the interval between 2 video frames is 10ms)
  
                      samples[n] = 16383.0 * sin(n*1000.0*2.0*M_PI / Fs); //sine wav
  
  
  
                  int  out_size = ffmpeg::avcodec_encode_audio(pAudioCodecCtx_, audio_outbuf, audio_outbuf_size, (const short*)samples);
  
  
  
                  pAudioCodecCtx_->coded_frame->pts = pts;  // Set the time stamp
  
  
  
                  if (out_size>0)
  
                  {
  
                      // Packet
  
                      ffmpeg::AVPacket pkt = { 0 };
  
                      av_init_packet(&pkt);
  
                      pkt.data = NULL; // packet data will be allocated by the encoder
  
                      pkt.size = 0;
  
                      if (pAudioCodecCtx_->coded_frame->pts != (0x8000000000000000LL))
  
                          pkt.pts = av_rescale_q(pAudioCodecCtx_->coded_frame->pts, pAudioCodecCtx_->time_base, pAudioStream_->time_base);
  
                      if (pAudioCodecCtx_->coded_frame->key_frame)
  
                          pkt.flags |= AV_PKT_FLAG_KEY;
  
  
  
                      pkt.stream_index = pAudioStream_->index;
  
                      pkt.data = audio_outbuf;
  
  
  
                      pkt.size = out_size;
  
                      int ret = av_interleaved_write_frame(pOutputFormatCtx_, &pkt);
  
                      if (ret<0)
  
                          return -1;
  
                      av_free_packet(&pkt);
  
                  }       
  
                  //end simple sound encoding
  
  
  
                  return pkt.size;
  
              }

  The result is a nice video with some audio behind (either a regular beeping sound at regular intervals but ending way earlier than the video or a continuous longer sound that also last shorter than the video).

  I want to generate a beeping sound each time the function encodeAudio() is called - at non-regular intervals. I have tried to modify the sampling rate, the buffer size, the pkt size and the number of samples but without any success. I also tried to set the pts at different times but it did not get me where I want to be. Could someone please help ?