Recherche avancée

Sur d’autres sites (9754)

• Workflow for creating animated hand-drawn videos - encoding difficulties

  8 décembre 2017, par Mircode

  I want to create YouTube videos, kind of in the style of a white-board animation.

  Tldr question : How can I encode into a lossless rgb video format with ffmpeg, including alpha channel ?

  More detailed :
  My current workflow looks like this :

  I draw the slides in Inkscape, I group all paths that are to be drawn in one go (one scene so to say) and store the slide as svg. Then I run a custom python script over that. It animates the slides as described here https://codepen.io/MyXoToD/post/howto-self-drawing-svg-animation. Each frame is exported as svg, converted to png and fed to ffmpeg for making a video from it.

  For every scene (a couple of paths being drawn, there are several scenes per slide) I create an own video file and then I also store a png file that contains the last frame of that video.

  I then use kdenlive to join it all together : A video containing the drawing of the first scene, then a png which holds the last image of the video while I talk about the drawing, then the next animated drawing, then the next still image where I continue talking and so on. I use these intermediate images because freezing the last frame is tedious in kdenlive and I have around 600 scenes. Here I do the editing, adjust the duration of the still images and render the final video.

  The background of the video is a photo of a blackboard which never changes, the strokes are paths with a filter to make it look like chalk.

  So far so good, everything almost works.

  My problem is : Whenever there is a transition between an animation and a still image, it is visible in the final result. I have tried several approaches to make this work but nothing is without flaws.

  My first approach was to encode the animations as mp4 like this :

  p = Popen(['ffmpeg', '-y', '-f', 'image2pipe', '-vcodec', 'png', '-r', str(fps), '-i', '-', '-vcodec', 'libx264', '-crf', '21', '-bf', '2', '-flags', '+cgop', '-pix_fmt', 'yuv420p', '-movflags', 'faststart', '-r', str(fps), videofile], stdin=PIPE)

  which is recommended for YouTube. But then there is a little brightness difference between video and still image.

  Then I tried mov with png codec :

  p = Popen(['ffmpeg', '-y', '-f', 'image2pipe', '-vcodec', 'png', '-r', str(fps), '-i', '-', '-vcodec', 'png', '-r', str(fps), videofile], stdin=PIPE)

  I think this encodes every frame as png in the video. It creates way bigger files since every frame is encoded separately. But it’s ok since I can use transparency for the background and just store the chalk strokes. However, sometimes I want to swipe parts of the chalk on a slide away, which I do by drawing background over it. Which would work if those overlaid, animated background chunks which are stored in the video looked exactly like the underlying png in the background. But it doesn’t. It’s slightly more blurry and I believe the color changes a tiny bit as well. Which I don’t understand since I thought the video just stores a sequence of pngs... Is there some quality setting that I miss here ?

  Then I read about ProRes4444 and tried that :

  p = Popen(['ffmpeg', '-y', '-f', 'image2pipe', '-vcodec', 'png', '-r', str(fps), '-i', '-', '-c:v', 'prores_ks', '-pix_fmt', 'yuva444p10le', '-alpha_bits', '8', '-profile:v', '4444', '-r', str(fps), videofile], stdin=PIPE)

  and this actually seems to work. However, the animation files become larger than the bunch of png files they contain, probably because this format stores 12 bit per channel. This is not thaat horrible since only intermediate videos grow big, the final result is still ok.

  But ideally there would be a lossless codec which stores in rgb colorspace with 8 bit per channel, 8 bit for alpha and considers only the difference to the previous frame (because all that changes from frame to frame is a tiny bit of chalk drawing). Is there such a thing ? Alternatively, I’d also be ok without transparency but then I have to store the background in every scene. But if only changes are stored from frame to frame within one scene, that should be manageable.

  Or should I make some fundamental changes in my workflow altogether ?

  Sorry that this is rather lengthy, I appreciate any help.

  Cheers !

• Live555 : X264 Stream Live source based on "testOnDemandRTSPServer"

  26 octobre 2017, par user2660369

  I am trying to create a rtsp Server that streams the OpenGL output of my program. I had a look at How to write a Live555 FramedSource to allow me to stream H.264 live, but I need the stream to be unicast. So I had a look at testOnDemandRTSPServer. Using the same Code fails. To my understanding I need to provide memory in which I store my h264 frames so the OnDemandServer can read them on Demand.

  H264VideoStreamServerMediaSubsession.cpp

  H264VideoStreamServerMediaSubsession*
  
  H264VideoStreamServerMediaSubsession::createNew(UsageEnvironment& env,
  
                            Boolean reuseFirstSource) {
  
    return new H264VideoStreamServerMediaSubsession(env, reuseFirstSource);
  
  }
  
  
  
  H264VideoStreamServerMediaSubsession::H264VideoStreamServerMediaSubsession(UsageEnvironment& env, Boolean reuseFirstSource)
  
    : OnDemandServerMediaSubsession(env, reuseFirstSource), fAuxSDPLine(NULL), fDoneFlag(0), fDummyRTPSink(NULL) {
  
  }
  
  
  
  H264VideoStreamServerMediaSubsession::~H264VideoStreamServerMediaSubsession() {
  
    delete[] fAuxSDPLine;
  
  }
  
  
  
  static void afterPlayingDummy(void* clientData) {
  
    H264VideoStreamServerMediaSubsession* subsess = (H264VideoStreamServerMediaSubsession*)clientData;
  
    subsess->afterPlayingDummy1();
  
  }
  
  
  
  void H264VideoStreamServerMediaSubsession::afterPlayingDummy1() {
  
    // Unschedule any pending 'checking' task:
  
    envir().taskScheduler().unscheduleDelayedTask(nextTask());
  
    // Signal the event loop that we're done:
  
    setDoneFlag();
  
  }
  
  
  
  static void checkForAuxSDPLine(void* clientData) {
  
    H264VideoStreamServerMediaSubsession* subsess = (H264VideoStreamServerMediaSubsession*)clientData;
  
    subsess->checkForAuxSDPLine1();
  
  }
  
  
  
  void H264VideoStreamServerMediaSubsession::checkForAuxSDPLine1() {
  
    char const* dasl;
  
  
  
    if (fAuxSDPLine != NULL) {
  
      // Signal the event loop that we're done:
  
      setDoneFlag();
  
    } else if (fDummyRTPSink != NULL && (dasl = fDummyRTPSink->auxSDPLine()) != NULL) {
  
      fAuxSDPLine = strDup(dasl);
  
      fDummyRTPSink = NULL;
  
  
  
      // Signal the event loop that we're done:
  
      setDoneFlag();
  
    } else {
  
      // try again after a brief delay:
  
      int uSecsToDelay = 100000; // 100 ms
  
      nextTask() = envir().taskScheduler().scheduleDelayedTask(uSecsToDelay,
  
                    (TaskFunc*)checkForAuxSDPLine, this);
  
    }
  
  }
  
  
  
  char const* H264VideoStreamServerMediaSubsession::getAuxSDPLine(RTPSink* rtpSink, FramedSource* inputSource) {
  
    if (fAuxSDPLine != NULL) return fAuxSDPLine; // it's already been set up (for a previous client)
  
  
  
    if (fDummyRTPSink == NULL) { // we're not already setting it up for another, concurrent stream
  
      // Note: For H264 video files, the 'config' information ("profile-level-id" and "sprop-parameter-sets") isn't known
  
      // until we start reading the file.  This means that "rtpSink"s "auxSDPLine()" will be NULL initially,
  
      // and we need to start reading data from our file until this changes.
  
      fDummyRTPSink = rtpSink;
  
  
  
      // Start reading the file:
  
      fDummyRTPSink->startPlaying(*inputSource, afterPlayingDummy, this);
  
  
  
      // Check whether the sink's 'auxSDPLine()' is ready:
  
      checkForAuxSDPLine(this);
  
    }
  
  
  
    envir().taskScheduler().doEventLoop(&fDoneFlag);
  
  
  
    return fAuxSDPLine;
  
  }
  
  
  
  FramedSource* H264VideoStreamServerMediaSubsession::createNewStreamSource(unsigned /*clientSessionId*/, unsigned& estBitrate) {
  
    estBitrate = 500; // kb
  
    megamol::remotecontrol::View3D_MRC *parent = (megamol::remotecontrol::View3D_MRC*)this->parent;
  
    return H264VideoStreamFramer::createNew(envir(), parent->h264FramedSource);
  
  }
  
  
  
  RTPSink* H264VideoStreamServerMediaSubsession::createNewRTPSink(Groupsock* rtpGroupsock, unsigned char rtpPayloadTypeIfDynamic, FramedSource* /*inputSource*/) {
  
    return H264VideoRTPSink::createNew(envir(), rtpGroupsock, rtpPayloadTypeIfDynamic);
  
  }

  FramedSource.cpp

  H264FramedSource* H264FramedSource::createNew(UsageEnvironment& env,
  
                                            unsigned preferredFrameSize,
  
                                            unsigned playTimePerFrame)
  
  {
  
      return new H264FramedSource(env, preferredFrameSize, playTimePerFrame);
  
  }
  
  
  
  H264FramedSource::H264FramedSource(UsageEnvironment& env,
  
                                 unsigned preferredFrameSize,
  
                                 unsigned playTimePerFrame)
  
      : FramedSource(env),
  
      fPreferredFrameSize(fMaxSize),
  
      fPlayTimePerFrame(playTimePerFrame),
  
      fLastPlayTime(0),
  
      fCurIndex(0)
  
  {
  
  
  
      x264_param_default_preset(&param, "veryfast", "zerolatency");
  
      param.i_threads = 1;
  
      param.i_width = 1024;
  
      param.i_height = 768;
  
      param.i_fps_num = 30;
  
      param.i_fps_den = 1;
  
      // Intra refres:
  
      param.i_keyint_max = 60;
  
      param.b_intra_refresh = 1;
  
      //Rate control:
  
      param.rc.i_rc_method = X264_RC_CRF;
  
      param.rc.f_rf_constant = 25;
  
      param.rc.f_rf_constant_max = 35;
  
      param.i_sps_id = 7;
  
      //For streaming:
  
      param.b_repeat_headers = 1;
  
      param.b_annexb = 1;
  
      x264_param_apply_profile(&param, "baseline");
  
  
  
      param.i_log_level = X264_LOG_ERROR;
  
  
  
      encoder = x264_encoder_open(&param);
  
      pic_in.i_type            = X264_TYPE_AUTO;
  
      pic_in.i_qpplus1         = 0;
  
      pic_in.img.i_csp         = X264_CSP_I420;
  
      pic_in.img.i_plane       = 3;
  
  
  
  
  
      x264_picture_alloc(&pic_in, X264_CSP_I420, 1024, 768);
  
  
  
      convertCtx = sws_getContext(1024, 768, PIX_FMT_RGBA, 1024, 768, PIX_FMT_YUV420P, SWS_FAST_BILINEAR, NULL, NULL, NULL);
  
      eventTriggerId = envir().taskScheduler().createEventTrigger(deliverFrame0);
  
  }
  
  
  
  H264FramedSource::~H264FramedSource()
  
  {
  
      envir().taskScheduler().deleteEventTrigger(eventTriggerId);
  
      eventTriggerId = 0;
  
  }
  
  
  
  void H264FramedSource::AddToBuffer(uint8_t* buf, int surfaceSizeInBytes)
  
  {
  
      uint8_t* surfaceData = (new uint8_t[surfaceSizeInBytes]);
  
  
  
      memcpy(surfaceData, buf, surfaceSizeInBytes);
  
  
  
      int srcstride = 1024*4;
  
      sws_scale(convertCtx, &surfaceData, &srcstride,0, 768, pic_in.img.plane, pic_in.img.i_stride);
  
      x264_nal_t* nals = NULL;
  
      int i_nals = 0;
  
      int frame_size = -1;
  
  
  
  
  
      frame_size = x264_encoder_encode(encoder, &nals, &i_nals, &pic_in, &pic_out);
  
  
  
      static bool finished = false;
  
  
  
      if (frame_size >= 0)
  
      {
  
      static bool alreadydone = false;
  
      if(!alreadydone)
  
      {
  
  
  
          x264_encoder_headers(encoder, &nals, &i_nals);
  
          alreadydone = true;
  
      }
  
      for(int i = 0; i < i_nals; ++i)
  
      {
  
          m_queue.push(nals[i]);
  
      }
  
      }
  
      delete [] surfaceData;
  
      surfaceData = nullptr;
  
  
  
      envir().taskScheduler().triggerEvent(eventTriggerId, this);
  
  }
  
  
  
  void H264FramedSource::doGetNextFrame()
  
  {
  
      deliverFrame();
  
  }
  
  
  
  void H264FramedSource::deliverFrame0(void* clientData)
  
  {
  
      ((H264FramedSource*)clientData)->deliverFrame();
  
  }
  
  
  
  void H264FramedSource::deliverFrame()
  
  {
  
      x264_nal_t nalToDeliver;
  
  
  
      if (fPlayTimePerFrame > 0 && fPreferredFrameSize > 0) {
  
      if (fPresentationTime.tv_sec == 0 && fPresentationTime.tv_usec == 0) {
  
          // This is the first frame, so use the current time:
  
          gettimeofday(&fPresentationTime, NULL);
  
      } else {
  
          // Increment by the play time of the previous data:
  
          unsigned uSeconds   = fPresentationTime.tv_usec + fLastPlayTime;
  
          fPresentationTime.tv_sec += uSeconds/1000000;
  
          fPresentationTime.tv_usec = uSeconds%1000000;
  
      }
  
  
  
      // Remember the play time of this data:
  
      fLastPlayTime = (fPlayTimePerFrame*fFrameSize)/fPreferredFrameSize;
  
      fDurationInMicroseconds = fLastPlayTime;
  
      } else {
  
      // We don't know a specific play time duration for this data,
  
      // so just record the current time as being the 'presentation time':
  
      gettimeofday(&fPresentationTime, NULL);
  
      }
  
  
  
      if(!m_queue.empty())
  
      {
  
      m_queue.wait_and_pop(nalToDeliver);
  
  
  
      uint8_t* newFrameDataStart = (uint8_t*)0xD15EA5E;
  
  
  
      newFrameDataStart = (uint8_t*)(nalToDeliver.p_payload);
  
      unsigned newFrameSize = nalToDeliver.i_payload;
  
  
  
      // Deliver the data here:
  
      if (newFrameSize > fMaxSize) {
  
          fFrameSize = fMaxSize;
  
          fNumTruncatedBytes = newFrameSize - fMaxSize;
  
      }
  
      else {
  
          fFrameSize = newFrameSize;
  
      }
  
  
  
      memcpy(fTo, nalToDeliver.p_payload, nalToDeliver.i_payload);
  
  
  
      FramedSource::afterGetting(this);
  
      }
  
  }

  Relevant part of the RTSP-Server Therad

    RTSPServer* rtspServer = RTSPServer::createNew(*(parent->env), 8554, NULL);
  
    if (rtspServer == NULL) {
  
      *(parent->env) << "Failed to create RTSP server: " << (parent->env)->getResultMsg() << "\n";
  
      exit(1);
  
    }
  
    char const* streamName = "Stream";
  
    parent->h264FramedSource = H264FramedSource::createNew(*(parent->env), 0, 0);
  
    H264VideoStreamServerMediaSubsession *h264VideoStreamServerMediaSubsession = H264VideoStreamServerMediaSubsession::createNew(*(parent->env), true);
  
    h264VideoStreamServerMediaSubsession->parent = parent;
  
    sms->addSubsession(h264VideoStreamServerMediaSubsession);
  
    rtspServer->addServerMediaSession(sms);
  
  
  
    parent->env->taskScheduler().doEventLoop(); // does not return

  Once a connection exists the render loop calls

  h264FramedSource->AddToBuffer(videoData, 1024*768*4);

• C++/C FFmpeg artifact build up across video frames

  6 mars 2017, par ChiragRaman

  Context :
  I am building a recorder for capturing video and audio in separate threads (using Boost thread groups) using FFmpeg 2.8.6 on Ubuntu 16.04. I followed the demuxing_decoding example here : https://www.ffmpeg.org/doxygen/2.8/demuxing_decoding_8c-example.html

  Video capture specifics :
  I am reading H264 off a Logitech C920 webcam and writing the video to a raw file. The issue I notice with the video is that there seems to be a build-up of artifacts across frames until a particular frame resets. Here is my frame grabbing, and decoding functions :

  // Used for injecting decoding functions for different media types, allowing
  
  // for a generic decode loop
  
  typedef std::function PacketDecoder;
  
  
  
  /**
  
   * Decodes a video packet.
  
   * If the decoding operation is successful, returns the number of bytes decoded,
  
   * else returns the result of the decoding process from ffmpeg
  
   */
  
  int decode_video_packet(AVPacket *packet,
  
                          int *got_frame,
  
                          int cached){
  
      int ret = 0;
  
      int decoded = packet->size;
  
  
  
      *got_frame = 0;
  
  
  
      //Decode video frame
  
      ret = avcodec_decode_video2(video_decode_context,
  
                                  video_frame, got_frame, packet);
  
      if (ret < 0) {
  
          //FFmpeg users should use av_err2str
  
          char errbuf[128];
  
          av_strerror(ret, errbuf, sizeof(errbuf));
  
          std::cerr << "Error decoding video frame " << errbuf << std::endl;
  
          decoded = ret;
  
      } else {
  
          if (*got_frame) {
  
              video_frame->pts = av_frame_get_best_effort_timestamp(video_frame);
  
  
  
              //Write to log file
  
              AVRational *time_base = &video_decode_context->time_base;
  
              log_frame(video_frame, time_base,
  
                        video_frame->coded_picture_number, video_log_stream);
  
  
  
  #if( DEBUG )
  
              std::cout << "Video frame " << ( cached ? "(cached)" : "" )
  
                        << " coded:" <<  video_frame->coded_picture_number
  
                        << " pts:" << pts << std::endl;
  
  #endif
  
  
  
              /*Copy decoded frame to destination buffer:
  
               *This is required since rawvideo expects non aligned data*/
  
              av_image_copy(video_dest_attr.video_destination_data,
  
                            video_dest_attr.video_destination_linesize,
  
                            (const uint8_t **)(video_frame->data),
  
                            video_frame->linesize,
  
                            video_decode_context->pix_fmt,
  
                            video_decode_context->width,
  
                            video_decode_context->height);
  
  
  
              //Write to rawvideo file
  
              fwrite(video_dest_attr.video_destination_data[0],
  
                     1,
  
                     video_dest_attr.video_destination_bufsize,
  
                     video_out_file);
  
  
  
              //Unref the refcounted frame
  
              av_frame_unref(video_frame);
  
          }
  
      }
  
  
  
      return decoded;
  
  }
  
  
  
  /**
  
   * Grabs frames in a loop and decodes them using the specified decoding function
  
   */
  
  int process_frames(AVFormatContext *context,
  
                     PacketDecoder packet_decoder) {
  
      int ret = 0;
  
      int got_frame;
  
      AVPacket packet;
  
  
  
      //Initialize packet, set data to NULL, let the demuxer fill it
  
      av_init_packet(&packet);
  
      packet.data = NULL;
  
      packet.size = 0;
  
  
  
      // read frames from the file
  
      for (;;) {
  
          ret = av_read_frame(context, &packet);
  
          if (ret < 0) {
  
              if  (ret == AVERROR(EAGAIN)) {
  
                  continue;
  
              } else {
  
                  break;
  
              }
  
          }
  
  
  
          //Convert timing fields to the decoder timebase
  
          unsigned int stream_index = packet.stream_index;
  
          av_packet_rescale_ts(&packet,
  
                               context->streams[stream_index]->time_base,
  
                               context->streams[stream_index]->codec->time_base);
  
  
  
          AVPacket orig_packet = packet;
  
          do {
  
              ret = packet_decoder(&packet, &got_frame, 0);
  
              if (ret < 0) {
  
                  break;
  
              }
  
              packet.data += ret;
  
              packet.size -= ret;
  
          } while (packet.size > 0);
  
          av_free_packet(&orig_packet);
  
  
  
          if(stop_recording == true) {
  
              break;
  
          }
  
      }
  
  
  
      //Flush cached frames
  
      std::cout << "Flushing frames" << std::endl;
  
      packet.data = NULL;
  
      packet.size = 0;
  
      do {
  
          packet_decoder(&packet, &got_frame, 1);
  
      } while (got_frame);
  
  
  
      av_log(0, AV_LOG_INFO, "Done processing frames\n");
  
      return ret;
  
  }

  Questions :
  

  1. How do I go about debugging the underlying issue ?
     
  2. Is it possible that running the decoding code in a thread other than the one in which the decoding context was opened is causing the problem ?
     
  3. Am I doing something wrong in the decoding code ?

  Things I have tried/found :
  

  1. I found this thread that is about the same problem here : FFMPEG decoding artifacts between keyframes
     (I cannot post samples of my corrupted frames due to privacy issues, but the image linked to in that question depicts the same issue I have)
     However, the answer to the question is posted by the OP without specific details about how the issue was fixed. The OP only mentions that he wasn’t ’preserving the packets correctly’, but nothing about what was wrong or how to fix it. I do not have enough reputation to post a comment seeking clarification.

  2. I was initially passing the packet into the decoding function by value, but switched to passing by pointer on the off chance that the packet freeing was being done incorrectly.

  3. I found another question about debugging decoding issues, but couldn’t find anything conclusive : How is video decoding corruption debugged ?

  I’d appreciate any insight. Thanks a lot !

  [EDIT] In response to Ronald’s answer, I am adding a little more information that wouldn’t fit in a comment :

  1. I am only calling decode_video_packet() from the thread processing video frames ; the other thread processing audio frames calls a similar decode_audio_packet() function. So only one thread calls the function. I should mention that I have set the thread_count in the decoding context to 1, failing which I would get a segfault in malloc.c while flushing the cached frames.

  2. I can see this being a problem if the process_frames and the frame decoder function were run on separate threads, which is not the case. Is there a specific reason why it would matter if the freeing is done within the function, or after it returns ? I believe the freeing function is passed a copy of the original packet because multiple decode calls would be required for audio packet in case the decoder doesnt decode the entire audio packet.

  3. A general problem is that the corruption does not occur all the time. I can debug better if it is deterministic. Otherwise, I can’t even say if a solution works or not.