Recherche avancée

Sur d’autres sites (9312)

• Capture from multiple streams concurrently, best way to do it and how to reduce CPU usage

  19 juin 2019, par DRONE_6969

  I am currently in the process of writing an application that will capture a lot of RTSP streams(in my case its 12) and display it on the QT widget. The problem arouses when I am going beyond around 6-7 streams, the CPU usage spikes and there is visible stutter.

  The reason why I think that it is not QT draw function is because I have done some checking to measure how much time it takes to draw an incoming image from camera and just sample images I had, it is always a lot less than 33 milliseconds(even if there are 12 widgets being updated).

  I also just ran opencv capture method without drawing and got pretty much the same CPU consumption as if I was drawing the frames (lost like 10% CPU at most and GPU usage went to zero).

  IMPORTANT : I am using RTSP stream which is a h264 stream.

  IF IT MATTERS MY SPECS :

  Intel Core i7-6700 @ 3.40GHZ(8 CPUS)
  Memory : 16gb
  GPU : Intel HD Graphics 530

  (Also I ran my code on a computer with dedicated Graphics card, it did eliminate some stutter but CPU usage is still pretty high)

  I am currently using OPENCV 4.1.0 with GSTREAMER enabled and built, I also have the OPENCV-WORLD version, there is no difference in performance.

  I have created a special class called Camera that holds its frame size constraints and various control functions as well stream function. The stream function is being ran on a separate thread, whenever stream() function is done with current frame it sends ready Mat via onNewFrame event I created which converts to QPixmap and updates widget’s lastImage variable. This way I can update image in a more thread safe way.

  I have tried to manipulate those VideoCapture.set() values, but it didn’t really help.

  This is my stream function (Ignore the bool return, it doesn’t do anything it is a remnant from couple of minutes ago when I was trying to use std::async) :

  bool Camera::stream() {
  
      /* This function is meant to run on a separate thread and fill up the buffer independantly of
  
      main stream thread */
  
      //cv::setNumThreads(100);
  
      /* Rules for these slightly changed! */
  
      Mat pre;  // Grab initial undoctored frame
  
      //pre = Mat::zeros(size, CV_8UC1);
  
      Mat frame; // Final modified frame
  
      frame = Mat::zeros(size, CV_8UC1);
  
      if (!pre.isContinuous()) pre = pre.clone();
  
  
  
      ipCam.open(streamUrl, CAP_FFMPEG);
  
  
  
  
  
      while (ipCam.isOpened() && capture) {
  
          // If camera is opened wel need to capture and process the frame
  
          try {
  
              auto start = std::chrono::system_clock::now();
  
  
  
              ipCam >> pre;
  
  
  
              if (pre.empty()) {
  
                  /* Check for blank frame, return error if there is a blank frame*/
  
                  cerr << id << ": ERROR! blank frame grabbed\n";
  
                  for (FrameListener* i : clients) {
  
                      i->onNotification(1); // Notify clients about this shit
  
                  }
  
                  break;
  
              }
  
  
  
              else {
  
                  // Only continue if frame not empty
  
  
  
                  if (pre.cols != size.width && pre.rows != size.height) {
  
                      resize(pre, frame, size);
  
                      pre.release();
  
                  }
  
                  else {
  
                      frame = pre;
  
                  }
  
  
  
                  dPacket* pack = new dPacket{id,&frame};
  
                  for (auto i : clients) {
  
                      i->onPNewFrame(pack);
  
                  }
  
                  frame.release();
  
                  delete pack;
  
              }
  
          }
  
  
  
          catch (int e) {
  
              cout << endl << "-----Exception during capture process! CODE " << e << endl;
  
          }
  
          // End camera manipulations
  
      }
  
  
  
      cout << "Camera timed out, or connection is closed..." << endl;
  
      if (tryResetConnection) {
  
          cout << "Reconnection flag is set, retrying after 3 seconds..." << endl;
  
          for (FrameListener* i : clients) {
  
              i->onNotification(-1); // Notify clients about this shit
  
          }
  
          this_thread::sleep_for(chrono::milliseconds(3000));
  
          stream();
  
      }
  
  
  
      return true;
  
  }

  This is my onPNewFrame function. The conversion is still being done on camera’s thread because it was called within stream() and therefore is within that scope(and I also checked) :

  void GLWidget::onPNewFrame(dPacket* inPack) {
  
      lastFlag = 0;
  
  
  
      if (bufferEnabled) {
  
          buffer.push(QPixmap::fromImage(toQImageFromPMat(inPack->frame)));
  
      }
  
      else {
  
          if (playing) {
  
              /* Only process if this widget is playing */
  
              frameProcessing = true;
  
              lastImage.convertFromImage(toQImageFromPMat(inPack->frame));
  
              frameProcessing = false;
  
          }
  
      }
  
  
  
      if (lastFlag != -1 && !lastImage.isNull()) {
  
          connecting = false;
  
      }
  
      else {
  
          connecting = true;
  
      }
  
  }

  This is my Mat to QImage :

  QImage GLWidget::toQImageFromPMat(cv::Mat* mat) {
  
  
  
  
  
  
  
      return QImage(mat->data, mat->cols, mat->rows, QImage::Format_RGB888).rgbSwapped();

  NOTE : not converting does not result in CPU boost (at least not a significant one).

  Minimal verifiable example

  This program is large. I am going to paste GLWidget.cpp and GLWidget.h as well as Camera.h and Camera.cpp. You can put GLWidget into anything just as long as you spawn more than 6 of it. Camera relies on the CamUtils, but it is possible to just paste url in videocapture

  I also supplied CamUtils, just in case

  Camera.h :

  #pragma once
  
  #include <iostream>
  
  #include <vector>
  
  #include <fstream>
  
  #include <map>
  
  #include <string>
  
  #include <sstream>
  
  #include <algorithm>
  
  #include "FrameListener.h"
  
  #include 
  
  #include <thread>
  
  #include "CamUtils.h"
  
  #include <ctime>
  
  #include "dPacket.h"
  
  
  
  using namespace std;
  
  using namespace cv;
  
  
  
  class Camera
  
  {
  
  
  
      /*
  
          CLEANED UP!
  
          Camera now is only responsible for streaming and echoing captured frames.
  
          Frames are now wrapped into dPacket struct.
  
      */
  
  
  
  
  
  private:
  
      string id;
  
      vector clients;
  
      VideoCapture ipCam;
  
      string streamUrl;
  
      Size size;
  
      bool tryResetConnection = false;
  
  
  
      //TODO: Remove these as they are not going to be used going on:
  
      bool isPlaying = true;
  
      bool capture = true;
  
  
  
      //SECRET FEATURES:
  
      bool detect = false;
  
  
  
  
  
  public:
  
      Camera(string url, int width = 480, int height = 240, bool detect_=false);
  
      bool stream();
  
      void setReconnectable(bool newReconStatus);
  
      void addListener(FrameListener* client);
  
      vector<bool> getState();    // Returns current state: vector[0] stream state; vector[1] stream state; TODO: Remove this as this is no longer should control behaviour
  
      void killStream();
  
      bool getReconnectable();
  
  };
  
  
  
  </bool></ctime></thread></algorithm></sstream></string></map></fstream></vector></iostream>

  Camera.cpp

  #include "Camera.h"
  
  
  
  
  
  Camera::Camera(string url, int width, int height, bool detect_) // Default 240p
  
  {
  
      streamUrl = url; // Prepare url
  
      size = Size(width, height);
  
      detect = detect_;
  
  
  
  }
  
  
  
  void Camera::addListener(FrameListener* client) {
  
      clients.push_back(client);
  
  }
  
  
  
  
  
  /*
  
                  TEST CAMERAS(Paste into cameras.dViewer):
  
                  {"id":"96a73796-c129-46fc-9c01-40acd8ed7122","ip":"176.57.73.231","password":"null","username":"null"},
  
                  {"id":"96a73796-c129-46fc-9c01-40acd8ed7122","ip":"176.57.73.231","password":"null","username":"null"},
  
                  {"id":"96a73796-c129-46fc-9c01-40acd8ed7144","ip":"172.20.101.13","password":"admin","username":"root"}
  
                  {"id":"96a73796-c129-46fc-9c01-40acd8ed7144","ip":"172.20.101.13","password":"admin","username":"root"}
  
  
  
  */
  
  
  
  
  
  
  
  bool Camera::stream() {
  
      /* This function is meant to run on a separate thread and fill up the buffer independantly of
  
      main stream thread */
  
      //cv::setNumThreads(100);
  
      /* Rules for these slightly changed! */
  
      Mat pre;  // Grab initial undoctored frame
  
      //pre = Mat::zeros(size, CV_8UC1);
  
      Mat frame; // Final modified frame
  
      frame = Mat::zeros(size, CV_8UC1);
  
      if (!pre.isContinuous()) pre = pre.clone();
  
  
  
      ipCam.open(streamUrl, CAP_FFMPEG);
  
  
  
      while (ipCam.isOpened() && capture) {
  
          // If camera is opened wel need to capture and process the frame
  
          try {
  
              auto start = std::chrono::system_clock::now();
  
  
  
              ipCam >> pre;
  
  
  
              if (pre.empty()) {
  
                  /* Check for blank frame, return error if there is a blank frame*/
  
                  cerr << id << ": ERROR! blank frame grabbed\n";
  
                  for (FrameListener* i : clients) {
  
                      i->onNotification(1); // Notify clients about this shit
  
                  }
  
                  break;
  
              }
  
  
  
              else {
  
                  // Only continue if frame not empty
  
  
  
                  if (pre.cols != size.width && pre.rows != size.height) {
  
                      resize(pre, frame, size);
  
                      pre.release();
  
                  }
  
                  else {
  
                      frame = pre;
  
                  }
  
  
  
                  auto end = std::chrono::system_clock::now();
  
                  std::time_t ts = std::chrono::system_clock::to_time_t(end);
  
                  dPacket* pack = new dPacket{ id,&frame};
  
                  for (auto i : clients) {
  
                      i->onPNewFrame(pack);
  
                  }
  
                  frame.release();
  
                  delete pack;
  
              }
  
          }
  
  
  
          catch (int e) {
  
              cout << endl << "-----Exception during capture process! CODE " << e << endl;
  
          }
  
          // End camera manipulations
  
      }
  
  
  
      cout << "Camera timed out, or connection is closed..." << endl;
  
      if (tryResetConnection) {
  
          cout << "Reconnection flag is set, retrying after 3 seconds..." << endl;
  
          for (FrameListener* i : clients) {
  
              i->onNotification(-1); // Notify clients about this shit
  
          }
  
          this_thread::sleep_for(chrono::milliseconds(3000));
  
          stream();
  
      }
  
  
  
      return true;
  
  }
  
  
  
  
  
  void Camera::killStream(){
  
      tryResetConnection = false;
  
      capture = false;
  
      ipCam.release();
  
  }
  
  
  
  void Camera::setReconnectable(bool reconFlag) {
  
      tryResetConnection = reconFlag;
  
  }
  
  
  
  bool Camera::getReconnectable() {
  
      return tryResetConnection;
  
  }
  
  
  
  vector<bool> Camera::getState() {
  
      vector<bool> states;
  
      states.push_back(isPlaying);
  
      states.push_back(ipCam.isOpened());
  
      return states;
  
  }
  
  
  
  
  
  
  
  </bool></bool>

  GLWidget.h :

  #ifndef GLWIDGET_H
  
  #define GLWIDGET_H
  
  
  
  #include <qopenglwidget>
  
  #include <qmouseevent>
  
  #include "FrameListener.h"
  
  #include "Camera.h"
  
  #include "FrameListener.h"
  
  #include 
  
  #include "Camera.h"
  
  #include "CamUtils.h"
  
  #include 
  
  #include "dPacket.h"
  
  #include <chrono>
  
  #include <ctime>
  
  #include 
  
  #include "FullScreenVideo.h"
  
  #include <qmovie>
  
  #include "helper.h"
  
  #include <iostream>
  
  #include <qpainter>
  
  #include <qtimer>
  
  
  
  class Helper;
  
  
  
  class GLWidget : public QOpenGLWidget, public FrameListener
  
  {
  
      Q_OBJECT
  
  
  
  public:
  
      GLWidget(std::string camId, CamUtils *cUtils, int width, int height, bool denyFullScreen_ = false, bool detectFlag_=false, QWidget* parent = nullptr);
  
      void killStream();
  
      ~GLWidget();
  
  
  
  public slots:
  
      void animate();
  
      void setBufferEnabled(bool setState);
  
      void setCameraRetryConnection(bool setState);
  
      void GLUpdate();            // Call to update the widget
  
      void onRightClickMenu(const QPoint&amp; point);
  
  
  
  protected:
  
      void paintEvent(QPaintEvent* event) override;
  
      void onPNewFrame(dPacket* frame);
  
      void onNotification(int alert_code);
  
  
  
  
  
  private:
  
      // Objects and resourses
  
      Helper* helper;
  
      Camera* cam;
  
      CamUtils* camUtils;
  
      QTimer* timer; // Keep track of update
  
      QPixmap lastImage;
  
      QMovie* connMov;
  
      QMovie* test;
  
  
  
      QPixmap logo;
  
  
  
      // Control fields
  
      int width;
  
      int height;
  
      int camUtilsAddr;
  
      int elapsed;
  
      std::thread* camThread;
  
      std::string camId;
  
      bool denyFullScreen = false;
  
      bool playing = true;
  
      bool streaming = true;
  
      bool debug = false;
  
      bool connecting = true;
  
      int lastFlag = 0;
  
  
  
  
  
      // Debug fields
  
      std::chrono::high_resolution_clock::time_point lastFrameAt;
  
      std::chrono::high_resolution_clock::time_point now;
  
      std::chrono::duration<double> painTime; // time took to draw last frame
  
  
  
      //Buffer stuff
  
      std::queue<qpixmap> buffer;
  
      bool bufferEnabled = false;
  
      bool initialBuffer = false;
  
      bool buffering = true;
  
      bool frameProcessing = false;
  
  
  
  
  
  
  
      //Functions
  
      QImage toQImageFromPMat(cv::Mat* inFrame);
  
      void mousePressEvent(QMouseEvent* event) override;
  
      void drawImageGLLatest(QPainter* painter, QPaintEvent* event, int elapsed);
  
      void drawOnPaused(QPainter* painter, QPaintEvent* event, int elapsed);
  
      void drawOnStatus(int statusFlag, QPainter* painter, QPaintEvent* event, int elapsed);
  
  };
  
  
  
  #endif
  
  
  
  </qpixmap></double></qtimer></qpainter></iostream></qmovie></ctime></chrono></qmouseevent></qopenglwidget>

  GLWidget.cpp :

  #include "glwidget.h"
  
  #include <future>
  
  
  
  
  
  FullScreenVideo* fullScreen;
  
  
  
  GLWidget::GLWidget(std::string camId_, CamUtils* cUtils, int width_, int height_,  bool denyFullScreen_, bool detectFlag_, QWidget* parent)
  
      : QOpenGLWidget(parent), helper(helper)
  
  {
  
      cout &lt;&lt; "Player for CAMERA " &lt;&lt; camId_ &lt;&lt; endl;
  
  
  
      /* Underlying properties */
  
      camUtils = cUtils;
  
      cout &lt;&lt; "GLWidget Incoming CamUtils addr " &lt;&lt; camUtils &lt;&lt; endl;
  
      cout &lt;&lt; "GLWidget Set CamUtils addr " &lt;&lt; camUtils &lt;&lt; endl;
  
      camId = camId_;
  
      elapsed = 0;
  
      width = width_ + 5;
  
      height = height_ + 5;
  
      helper = new Helper();
  
      setFixedSize(width, height);
  
      denyFullScreen = denyFullScreen_;
  
  
  
      /* Camera capture thread */
  
      cam = new Camera(camUtils->getCameraStreamURL(camId), width_, height_, detectFlag_);
  
      cam->addListener(this);
  
  
  
      /* Sync states */
  
      vector<bool> initState = cam->getState();
  
      playing = initState[0];
  
      streaming = initState[1];
  
      cout &lt;&lt; "Initial states: " &lt;&lt; playing &lt;&lt; " " &lt;&lt; streaming &lt;&lt; endl;
  
      camThread = new std::thread(&amp;Camera::stream, cam);
  
      cout &lt;&lt; "================================================" &lt;&lt; endl;
  
  
  
      // Right click set up
  
      setContextMenuPolicy(Qt::CustomContextMenu);
  
  
  
  
  
      /* Loading gif */
  
      connMov = new QMovie("establishingConnection.gif");
  
      connMov->start();
  
      QString url = R"(RLC-logo.png)";
  
      logo = QPixmap(url);
  
      QTimer* timer = new QTimer(this);
  
      connect(timer, SIGNAL(timeout()), this, SLOT(GLUpdate()));
  
      timer->start(1000/30);
  
      playing = true;
  
  
  
  }
  
  
  
  /* SYSTEM */
  
  void GLWidget::animate()
  
  {
  
      elapsed = (elapsed + qobject_cast(sender())->interval()) % 1000;
  
      std::cout &lt;&lt; elapsed &lt;&lt; "\n";
  
  }
  
  
  
  
  
  void GLWidget::GLUpdate() {
  
      /* Process descisions before update call */
  
      if (bufferEnabled) {
  
          /* Process buffer before update */
  
          now = chrono::high_resolution_clock::now();
  
          std::chrono::duration timeSinceLastUpdate = now - lastFrameAt;
  
          if (timeSinceLastUpdate.count() > 25) {
  
              if (buffer.size() > 1 &amp;&amp; playing) {
  
                  lastImage.swap(buffer.front());
  
                  buffer.pop();
  
                  lastFrameAt = chrono::high_resolution_clock::now();
  
              }
  
          }
  
          //update(); // Update
  
      }
  
      else {
  
          /* No buffer */
  
      }
  
      repaint();
  
  }
  
  
  
  
  
  /* EVENTS */
  
  void GLWidget::onRightClickMenu(const QPoint&amp; point) {
  
      cout &lt;&lt; "Right click request got" &lt;&lt; endl;
  
  
  
      QPoint globPos = this->mapToGlobal(point);
  
      QMenu myMenu;
  
  
  
      if (!denyFullScreen) {
  
          myMenu.addAction("Open Full Screen");
  
      }
  
      myMenu.addAction("Toggle Debug Info");
  
  
  
  
  
      QAction* selected = myMenu.exec(globPos);
  
  
  
      if (selected) {
  
          string optiontxt = selected->text().toStdString();
  
  
  
          if (optiontxt == "Open Full Screen") {
  
              cout &lt;&lt; "Chose to open full screen of " &lt;&lt; camId &lt;&lt; endl;
  
              fullScreen = new FullScreenVideo(bufferEnabled, this);
  
              fullScreen->setUpView(camUtils, camId);
  
              fullScreen->show();
  
              playing = false;
  
          }
  
  
  
          if (optiontxt == "Toggle Debug Info") {
  
              cout &lt;&lt; "Chose to toggle debug of " &lt;&lt; camId &lt;&lt; endl;
  
              debug = !debug;
  
          }
  
      }
  
      else {
  
          cout &lt;&lt; "Chose nothing!" &lt;&lt; endl;
  
      }
  
  
  
  
  
  }
  
  
  
  
  
  
  
  void GLWidget::onPNewFrame(dPacket* inPack) {
  
      lastFlag = 0;
  
  
  
      if (bufferEnabled) {
  
          buffer.push(QPixmap::fromImage(toQImageFromPMat(inPack->frame)));
  
      }
  
      else {
  
          if (playing) {
  
              /* Only process if this widget is playing */
  
              frameProcessing = true;
  
              lastImage.convertFromImage(toQImageFromPMat(inPack->frame));
  
              frameProcessing = false;
  
          }
  
      }
  
  
  
      if (lastFlag != -1 &amp;&amp; !lastImage.isNull()) {
  
          connecting = false;
  
      }
  
      else {
  
          connecting = true;
  
      }
  
  }
  
  
  
  
  
  void GLWidget::onNotification(int alert) {
  
      lastFlag = alert;   
  
  }
  
  
  
  
  
  /* Paint events*/
  
  
  
  
  
  void GLWidget::paintEvent(QPaintEvent* event)
  
  {
  
      QPainter painter(this);
  
  
  
          if (lastFlag != 0 || connecting) {
  
              drawOnStatus(lastFlag, &amp;painter, event, elapsed);
  
          }
  
          else {
  
  
  
              /* Actual frame drawing */
  
              if (playing) {
  
                  if (!frameProcessing) {
  
                      drawImageGLLatest(&amp;painter, event, elapsed);
  
                  }
  
              }
  
              else {
  
                  drawOnPaused(&amp;painter, event, elapsed);
  
              }
  
          }
  
      painter.end();
  
  
  
  }
  
  
  
  
  
  /* DRAWING STUFF */
  
  
  
  void GLWidget::drawOnStatus(int statusFlag, QPainter* bgPaint, QPaintEvent* event, int elapsed) {
  
  
  
      QString str;
  
      QFont font("times", 15);
  
      bgPaint->eraseRect(QRect(0, 0, width, height));
  
      if (!lastImage.isNull()) {
  
          bgPaint->drawPixmap(QRect(0, 0, width, height), lastImage);
  
      }
  
      /* Test background painting */
  
      if (connecting) {
  
          string k = "Connecting to " + camUtils->getIp(camId);
  
          str.append(k.c_str());
  
      }
  
      else {
  
          switch (statusFlag) {
  
          case 1:
  
              str = "Blank frame received...";
  
              break;
  
  
  
          case -1:
  
              if (cam->getReconnectable()) {
  
                  str = "Connection lost, will try to reconnect.";
  
                  bgPaint->setOpacity(0.3);
  
              }
  
              else {
  
                  str = "Connection lost...";
  
                  bgPaint->setOpacity(0.3);
  
              }
  
  
  
              break;
  
          }
  
      }
  
  
  
      bgPaint->drawPixmap(QRect(0, 0, width, height), QPixmap::fromImage(connMov->currentImage()));
  
      bgPaint->setPen(Qt::red);
  
      bgPaint->setFont(font);
  
      QFontMetrics fm(font);
  
      const QRect kek(0, 0, fm.width(str), fm.height());
  
      QRect bound;
  
      bgPaint->setOpacity(1);
  
      bgPaint->drawText(bgPaint->viewport().width()/2 - kek.width()/2, bgPaint->viewport().height()/2 - kek.height(), str);
  
  
  
      bgPaint->drawPixmap(bgPaint->viewport().width() / 2 - logo.width()/2, height - logo.width() - 15, logo);
  
  
  
  }
  
  
  
  
  
  
  
  void GLWidget::drawOnPaused(QPainter* painter, QPaintEvent* event, int elapsed) {
  
      painter->eraseRect(0, 0, width, height);
  
      QFont font = painter->font();
  
      font.setPointSize(18);
  
      painter->setPen(Qt::red);
  
      QFontMetrics fm(font);
  
      QString str("Paused");
  
      painter->drawPixmap(QRect(0, 0, width, height),lastImage);
  
      painter->drawText(QPoint(painter->viewport().width() - fm.width(str), 50), str);
  
  
  
      if (debug) {
  
          QFont font = painter->font();
  
          font.setPointSize(25);
  
          painter->setPen(Qt::red);
  
          string camMess = "CAMID: " + camId;
  
          QString mess(camMess.c_str());
  
          string camIp = "IP: " + camUtils->getIp(camId);
  
          QString ipMess(camIp.c_str());
  
          QString bufferSize("Buffer size: " + QString::number(buffer.size()));
  
          QString lastFrameText("Last frame draw time: " + QString::number(painTime.count()) + "s");
  
          painter->drawText(QPoint(10, 50), mess);
  
          painter->drawText(QPoint(10, 60), ipMess);
  
          QString bufferState;
  
          if (bufferEnabled) {
  
              bufferState = QString("Experimental BUFFER is enabled!");
  
              QString currentBufferSize("Current buffer load: " + QString::number(buffer.size()));
  
              painter->drawText(QPoint(10, 80), currentBufferSize);
  
          }
  
          else {
  
              bufferState = QString("Experimental BUFFER is disabled!");
  
          }
  
          painter->drawText(QPoint(10, 70), bufferState);
  
          painter->drawText(QPoint(10, height - 25), lastFrameText);
  
      }
  
  }
  
  
  
  
  
  void GLWidget::drawImageGLLatest(QPainter* painter, QPaintEvent* event, int elapsed) {
  
      auto start = chrono::high_resolution_clock::now();
  
      painter->drawPixmap(QRect(0, 0, width, height), lastImage);
  
      if (debug) {
  
          QFont font = painter->font();
  
          font.setPointSize(25);
  
          painter->setPen(Qt::red);
  
          string camMess = "CAMID: " + camId;
  
          QString mess(camMess.c_str());
  
          string camIp = "IP: " + camUtils->getIp(camId);
  
          QString ipMess(camIp.c_str());
  
          QString bufferSize("Buffer size: " + QString::number(buffer.size()));
  
          QString lastFrameText("Last frame draw time: " + QString::number(painTime.count()) + "s");
  
          painter->drawText(QPoint(10, 50), mess);
  
          painter->drawText(QPoint(10, 60), ipMess);
  
          QString bufferState;
  
          if(bufferEnabled){
  
              bufferState = QString("Experimental BUFFER is enabled!");
  
              QString currentBufferSize("Current buffer load: " + QString::number(buffer.size()));
  
              painter->drawText(QPoint(10,80), currentBufferSize);
  
          }
  
          else {
  
              bufferState = QString("Experimental BUFFER is disabled!");
  
              QString currentBufferSize("Current buffer load: " + QString::number(buffer.size()));
  
              painter->drawText(QPoint(10, 80), currentBufferSize);
  
          }
  
          painter->drawText(QPoint(10, 70), bufferState);
  
          painter->drawText(QPoint(10, height - 25), lastFrameText);
  
  
  
      }
  
      auto end = chrono::high_resolution_clock::now();
  
      painTime = end - start;
  
  }
  
  
  
  
  
  
  
  /* END DRAWING STUFF */
  
  
  
  
  
  
  
  /* UI EVENTS */
  
  
  
  void GLWidget::mousePressEvent(QMouseEvent* e) {
  
  
  
      if (e->button() == Qt::LeftButton) {
  
          if (fullScreen == nullptr || !fullScreen->isVisible()) { // Do not unpause if window is opened
  
              playing = !playing;
  
          }
  
      }
  
  
  
      if (e->button() == Qt::RightButton) {
  
          onRightClickMenu(e->pos());
  
      }
  
  }
  
  
  
  
  
  
  
  /* Utilities */
  
  QImage GLWidget::toQImageFromPMat(cv::Mat* mat) {
  
  
  
  
  
  
  
      return QImage(mat->data, mat->cols, mat->rows, QImage::Format_RGB888).rgbSwapped();
  
  
  
  
  
  
  
  }
  
  
  
  /* State control */
  
  
  
  void GLWidget::killStream() {
  
      cam->killStream();
  
      camThread->join();
  
  }
  
  
  
  void GLWidget::setBufferEnabled(bool newBufferState) {
  
      cout &lt;&lt; "Player: " &lt;&lt; camId &lt;&lt; ", buffer state updated: " &lt;&lt; newBufferState &lt;&lt; endl;
  
      bufferEnabled = newBufferState;
  
      buffer.empty();
  
  }
  
  
  
  void GLWidget::setCameraRetryConnection(bool newState) {
  
      cam->setReconnectable(newState);
  
  }
  
  
  
  /* Destruction */
  
  GLWidget::~GLWidget() {
  
      cam->killStream();
  
      camThread->join();
  
  }
  
  </bool></future>

  CamUtils.h :

  #pragma once
  
  #include <iostream>
  
  #include <vector>
  
  #include <fstream>
  
  #include <map>
  
  #include <string>
  
  #include <sstream>
  
  #include <algorithm>
  
  #include <nlohmann></nlohmann>json.hpp>
  
  
  
  using namespace std;
  
  using json = nlohmann::json;
  
  
  
  class CamUtils
  
  {
  
  private:
  
  
  
      string camDb = "cameras.dViewer";
  
      map> cameraList; // Legacy
  
      json cameras;
  
      ofstream dbFile;
  
      bool dbExists(); // Always hard coded
  
  
  
      /* Old IMPLEMENTATION */
  
      void writeLineToDb_(const string&amp; content, bool append = false);
  
      void loadCameras_();
  
  
  
      /* JSON based */
  
      void loadCameras();
  
  
  
  public:
  
      CamUtils();
  
      string generateRandomString(size_t length);
  
      string getCameraStreamURL(string cameraId) const;
  
      string saveCamera(string ip, string username, string pass); // Return generated id
  
      vector<string> listAllCameraIds();
  
      string getIp(string cameraId);
  
  };
  
  
  
  
  
  </string></algorithm></sstream></string></map></fstream></vector></iostream>

  CamUtils.cpp :

  #include "CamUtils.h"
  
  #pragma comment(lib, "rpcrt4.lib")  // UuidCreate - Minimum supported OS Win 2000
  
  #include 
  
  #include <iostream>
  
  
  
  CamUtils::CamUtils()
  
  {
  
      if (!dbExists()) {
  
          ofstream dbFile;
  
          dbFile.open(camDb);
  
          cameras["cameras"] = json::array();
  
          dbFile &lt;&lt; cameras &lt;&lt; std::endl;
  
          dbFile.close();
  
  
  
      }
  
      else {
  
          loadCameras();
  
      }
  
  }
  
  
  
  
  
  
  
  
  
  vector<string> CamUtils::listAllCameraIds() {
  
      vector<string> ids;
  
      cout &lt;&lt; "IN LIST " &lt;&lt; endl;
  
      for (auto&amp; cam : cameras["cameras"]) {
  
          ids.push_back(cam["id"].get<string>());
  
          //cout &lt;&lt; cam["id"].get<string>() &lt;&lt; std::endl;
  
      }
  
      return ids;
  
  }
  
  
  
  string CamUtils::getIp(string id) {
  
      vector<string> camDetails = cameraList[id];
  
      string ip = "NO IP WILL DISPLAYED UNTIL I FIGURE OUT A BUG";
  
      for (auto&amp; cam : cameras["cameras"]) {
  
          if (id == cam["id"]) {
  
              ip = cam["ip"].get<string>();
  
          }
  
      }
  
  
  
      return ip;
  
  }
  
  
  
  string CamUtils::getCameraStreamURL(string id) const {
  
      string url = "err"; // err is the default, it will be overwritten in case id is found, dont forget to check for it
  
  
  
      for (auto&amp; cam : cameras["cameras"]) {
  
          if (id == cam["id"]) {
  
              if (cam["username"].get<string>() == "null") {
  
                  url = "rtsp://" + cam["ip"].get<string>() + ":554/axis-media/media.amp?tcp";
  
              }
  
              else {
  
                  url = "rtsp://" + cam["username"].get<string>() + ":" + cam["password"].get<string>() + "@" + cam["ip"].get<string>() + ":554/axis-media/media.amp?streamprofile=720_30";
  
              }
  
          }
  
      }
  
  
  
      return url;  // Dont forget to check for err when using this shit
  
  }
  
  
  
  
  
  string CamUtils::saveCamera(string ip, string username, string password) {
  
      UUID uid;
  
      UuidCreate(&amp;uid);
  
      char* str;
  
      UuidToStringA(&amp;uid, (RPC_CSTR*)&amp;str);
  
      string id = str;
  
      cout &lt;&lt; "GEN: " &lt;&lt; id &lt;&lt; endl;
  
      json cam = json({}); //Create emtpy object
  
      cam["id"] = id;
  
      cam["ip"] = ip;
  
      cam["username"] = username;
  
      cam["password"] = password;
  
      cameras["cameras"].push_back(cam);
  
      std::ofstream out(camDb);
  
      out &lt;&lt; cameras &lt;&lt; std::endl;
  
      cout &lt;&lt; cameras["cameras"] &lt;&lt; endl;
  
  
  
      cout &lt;&lt; "Saved camera as " &lt;&lt; id &lt;&lt; endl;
  
      return id;
  
  }
  
  
  
  
  
  bool CamUtils::dbExists() {
  
      ifstream dbFile(camDb);
  
      return (bool)dbFile;
  
  }
  
  
  
  
  
  
  
  
  
  
  
  void CamUtils::loadCameras() {
  
      cout &lt;&lt; "Load call" &lt;&lt; endl;
  
      ifstream dbFile(camDb);
  
      string line;
  
      string wholeFile;
  
  
  
      while (std::getline(dbFile, line)) {
  
          cout &lt;&lt; line &lt;&lt; endl;
  
          wholeFile += line;
  
      }
  
      try {
  
          cameras = json::parse(wholeFile);
  
          //cout &lt;&lt; cameras["cameras"] &lt;&lt; endl;
  
  
  
      }
  
      catch (exception e) {
  
          cout &lt;&lt; e.what() &lt;&lt; endl;
  
      }
  
      dbFile.close();
  
  }
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  /*
  
      LEGACY CODE, TO BE REMOVED!
  
  
  
  */
  
  
  
  
  
  
  
  void CamUtils::loadCameras_() {
  
      /* 
  
          LEGACY CODE:
  
          This used to be the way to load cameras, but I moved on to JSON based configuration so this is no longer needed and will be removed soon
  
      */
  
  
  
      ifstream dbFile(camDb);
  
      string line;
  
      while (std::getline(dbFile, line)) {
  
          /*
  
              This function load camera data to the map:
  
              The order MUST be the following: 0:ID, 1:IP, 2:USERNAME, 3:PASSWORD.
  
              Always delimited with | no spaces between!
  
          */
  
          if (!line.empty()) {
  
              stringstream ss(line);
  
              string item;
  
              vector<string> splitString;
  
  
  
              while (std::getline(ss, item, '|')) {
  
                  splitString.push_back(item);
  
              }
  
              if (splitString.size() > 0) {
  
                  /* Dont even parse if the program didnt split right*/
  
                  //cout &lt;&lt; "Split string: " &lt;&lt; splitString.size() &lt;&lt; "\n";
  
                  for (int i = 0; i &lt; (splitString.size()); i++) cameraList[splitString[0]].push_back(splitString[i]);
  
              }
  
          }
  
      }
  
  }
  
  
  
  
  
  
  
  void CamUtils::writeLineToDb_(const string &amp; content, bool append) {
  
      ofstream dbFile;
  
      cout &lt;&lt; "Creating?";
  
      if (append) {
  
          dbFile.open(camDb, ios_base::app);
  
      }
  
      else {
  
          dbFile.open(camDb);
  
      }
  
  
  
      dbFile &lt;&lt; content.c_str() &lt;&lt; "\r\n";
  
      dbFile.flush();
  
  }
  
  
  
  /* JSON Reworx */
  
  
  
  
  
  
  
  
  
  string CamUtils::generateRandomString(size_t length)
  
  {
  
      const char* charmap = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
  
      const size_t charmapLength = strlen(charmap);
  
      auto generator = [&amp;]() { return charmap[rand() % charmapLength]; };
  
      string result;
  
      result.reserve(length);
  
      generate_n(back_inserter(result), length, generator);
  
      return result;
  
  }
  
  </string></string></string></string></string></string></string></string></string></string></string></string></iostream>

  End of example

  How would I go about decreasing CPU usage when dealing with large amount of streams ?

• FFmpeg smearing rtp missed packet

  3 janvier 2018, par Akim Benchiha

  I’m using ffmpeg to combine streams. But during the combinaison I have some errors/Warnings. I don’t know why. I can see the frames are dropping. And the output video is poor quality

  Here the situation :
  First : Try to combine local video and a rtsp stream from data center.

  ffmpeg -i rtsp://cloudzensg.loginto.me:554/livecast -i Xmen2trailer.mov -filter_complex "[0]scale=-1:-1[b];[1]scale=128:128[w];[b][w] overlay=10:10" -vcodec libx264 -preset ultrafast -f flv out.mp4
  
  ffmpeg version N-89672-g41e51fbcd9 Copyright (c) 2000-2018 the FFmpeg developers
  
    built with gcc 7.2.0 (GCC)
  
  [udp @ 000001e07f58ccc0] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
  
  [udp @ 000001e07f5a2b80] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
  
  [udp @ 000001e07f5b4040] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
  
  [udp @ 000001e07f5c4300] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
  
  Input #0, rtsp, from 'rtsp://cloudzensg.loginto.me:554/livecast':
  
    Metadata:
  
      title           : session
  
    Duration: N/A, start: 0.086367, bitrate: N/A
  
      Stream #0:0: Audio: aac, 44100 Hz, stereo, fltp
  
      Stream #0:1: Video: h264 (Constrained Baseline), yuv420p(progressive), 1280x720, 30 fps, 30 tbr, 90k tbn, 60 tbc
  
  Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'Xmen2trailer.mov':
  
    Metadata:
  
      creation_time   : 2003-02-17T16:23:00.000000Z
  
      title           : X2
  
      title-eng       : X2
  
      copyright-eng   : ┬⌐2003 20th Century Fox
  
      comment         : QuickTime 5 version encoded and delivered by www.apple.com/trailers/
  
      copyright       : ┬⌐2003 20th Century Fox
  
      comment-eng     : QuickTime 5 version encoded and delivered by www.apple.com/trailers/
  
    Duration: 00:02:29.08, start: 0.000000, bitrate: 1283 kb/s
  
      Stream #1:0(eng): Video: svq3 (SVQ3 / 0x33515653), yuvj420p(pc), 480x272, 1153 kb/s, 24 fps, 24 tbr, 600 tbn, 600 tbc (default)
  
      Metadata:
  
        creation_time   : 2003-02-17T16:23:00.000000Z
  
        handler_name    : Apple Alias Data Handler
  
        encoder         : Sorenson Video 3
  
      Stream #1:1(eng): Audio: qdm2 (QDM2 / 0x324D4451), 44100 Hz, stereo, s16, 128 kb/s (default)
  
      Metadata:
  
        creation_time   : 2003-02-17T16:23:00.000000Z
  
        handler_name    : Apple Alias Data Handler
  
  File 'out.mp4' already exists. Overwrite ? [y/N] y
  
  Stream mapping:
  
    Stream #0:1 (h264) -> scale (graph 0)
  
    Stream #1:0 (svq3) -> scale (graph 0)
  
    overlay (graph 0) -> Stream #0:0 (libx264)
  
    Stream #1:1 -> #0:1 (qdm2 (native) -> mp3 (libmp3lame))
  
  Press [q] to stop, [?] for help
  
  [swscaler @ 000001e002b3d980] deprecated pixel format used, make sure you did set range correctly
  
  [libx264 @ 000001e07f5edd00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
  
  [libx264 @ 000001e07f5edd00] profile Constrained Baseline, level 3.1
  
  [libx264 @ 000001e07f5edd00] 264 - core 152 r2851 ba24899 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
  
  Output #0, flv, to 'out.mp4':
  
    Metadata:
  
      title           : session
  
      encoder         : Lavf58.3.100
  
      Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuv420p, 1280x720, q=-1--1, 30 fps, 1k tbn, 30 tbc (default)
  
      Metadata:
  
        encoder         : Lavc58.9.100 libx264
  
      Side data:
  
        cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
  
      Stream #0:1(eng): Audio: mp3 (libmp3lame) ([2][0][0][0] / 0x0002), 44100 Hz, stereo, s16p (default)
  
      Metadata:
  
        creation_time   : 2003-02-17T16:23:00.000000Z
  
        handler_name    : Apple Alias Data Handler
  
        encoder         : Lavc58.9.100 libmp3lame
  
  Past duration 0.889320 too large
  
  Past duration 0.629326 too large
  
  Past duration 0.979652 too large
  
  Past duration 0.909660 too large
  
  Past duration 0.646996 too large
  
  Past duration 0.881325 too large
  
  Past duration 0.728661 too large
  
  Past duration 0.970665 too large
  
  Past duration 0.610664 too large
  
  Past duration 0.851997 too large
  
  Past duration 0.779991 too large
  
  Past duration 0.870995 too large
  
  Past duration 0.752327 too large
  
  Past duration 0.970665 too large
  
  Past duration 0.679665 too large
  
  Past duration 0.951988 too large
  
  Past duration 0.913994 too large
  
  Past duration 0.641655 too large
  
  [rtsp @ 000001e07f58a700] max delay reached. need to consume packet
  
  [rtsp @ 000001e07f58a700] RTP: missed 747 packets
  
  [h264 @ 000001e07f63a140] corrupted macroblock 30 15 (total_coeff=-1)063.3kbits/s dup=0 drop=66 speed=6.52x
  
  [h264 @ 000001e07f63a140] error while decoding MB 30 15
  
  [h264 @ 000001e07f63a140] concealing 2380 DC, 2380 AC, 2380 MV errors in P frame
  
  Past duration 0.971657 too large
  
  Past duration 0.867989 too large    1280kB time=00:00:04.44 bitrate=2360.6kbits/s dup=0 drop=70 speed=1.44x
  
  [rtsp @ 000001e07f58a700] max delay reached. need to consume packet
  
  [rtsp @ 000001e07f58a700] RTP: missed 43 packets
  
  [h264 @ 000001e07f639380] corrupted macroblock 11 18 (total_coeff=-1)
  
  [h264 @ 000001e07f639380] error while decoding MB 11 18
  
  [h264 @ 000001e07f639380] concealing 2156 DC, 2156 AC, 2156 MV errors in P frame
  
  Past duration 0.895988 too large
  
  Past duration 0.708656 too large
  
  frame=  105 fps= 29 q=-1.0 Lsize=    2169kB time=00:00:08.98 bitrate=1977.1kbits/s dup=0 drop=76 speed=2.52x
  
  video:2020kB audio:141kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.363079%
  
  [libx264 @ 000001e07f5edd00] frame I:1     Avg QP:20.00  size: 84510
  
  [libx264 @ 000001e07f5edd00] frame P:104   Avg QP:18.44  size: 19074
  
  [libx264 @ 000001e07f5edd00] mb I  I16..4: 100.0%  0.0%  0.0%
  
  [libx264 @ 000001e07f5edd00] mb P  I16..4: 15.4%  0.0%  0.0%  P16..4: 23.1%  0.0%  0.0%  0.0%  0.0%    skip:61.5%
  
  [libx264 @ 000001e07f5edd00] coded y,uvDC,uvAC intra: 7.4% 7.3% 4.5% inter: 14.3% 12.2% 3.4%
  
  [libx264 @ 000001e07f5edd00] i16 v,h,dc,p: 71% 27%  1%  0%
  
  [libx264 @ 000001e07f5edd00] i8c dc,h,v,p: 58% 37%  4%  0%
  
  [libx264 @ 000001e07f5edd00] kb/s:1969.74
  
  Exiting normally, received signal 2

  second :
  Try to combine local rtsp and local video

  ffmpeg -i rtsp://192.168.1.203:554/livecast -i Xmen2trailer.mov -filter_complex "[0]scale=-1:-1[b];[1]scale=128:128[w];[b][w] overlay=10:10" -vcodec libx264 -preset ultrafast -f flv out.mp4
  
  ffmpeg version N-89672-g41e51fbcd9 Copyright (c) 2000-2018 the FFmpeg developers
  
    built with gcc 7.2.0 (GCC)
  
  [udp @ 000002aa1fe4ca00] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
  
  [udp @ 000002aa1fe62900] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
  
  [udp @ 000002aa1fe73d80] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
  
  [udp @ 000002aa1fe84040] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
  
  Input #0, rtsp, from 'rtsp://192.168.1.203:554/livecast':
  
    Metadata:
  
      title           : session
  
    Duration: N/A, start: 0.000000, bitrate: N/A
  
      Stream #0:0: Audio: aac (LC), 48000 Hz, stereo, fltp
  
      Stream #0:1: Video: h264 (Constrained Baseline), yuv420p(progressive), 1920x1080, 30 fps, 30 tbr, 90k tbn, 60 tbc
  
  Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'Xmen2trailer.mov':
  
    Metadata:
  
      creation_time   : 2003-02-17T16:23:00.000000Z
  
      title           : X2
  
      title-eng       : X2
  
      copyright-eng   : ┬⌐2003 20th Century Fox
  
      comment         : QuickTime 5 version encoded and delivered by www.apple.com/trailers/
  
      copyright       : ┬⌐2003 20th Century Fox
  
      comment-eng     : QuickTime 5 version encoded and delivered by www.apple.com/trailers/
  
    Duration: 00:02:29.08, start: 0.000000, bitrate: 1283 kb/s
  
      Stream #1:0(eng): Video: svq3 (SVQ3 / 0x33515653), yuvj420p(pc), 480x272, 1153 kb/s, 24 fps, 24 tbr, 600 tbn, 600 tbc (default)
  
      Metadata:
  
        creation_time   : 2003-02-17T16:23:00.000000Z
  
        handler_name    : Apple Alias Data Handler
  
        encoder         : Sorenson Video 3
  
      Stream #1:1(eng): Audio: qdm2 (QDM2 / 0x324D4451), 44100 Hz, stereo, s16, 128 kb/s (default)
  
      Metadata:
  
        creation_time   : 2003-02-17T16:23:00.000000Z
  
        handler_name    : Apple Alias Data Handler
  
  File 'out.mp4' already exists. Overwrite ? [y/N] y
  
  Stream mapping:
  
    Stream #0:1 (h264) -> scale (graph 0)
  
    Stream #1:0 (svq3) -> scale (graph 0)
  
    overlay (graph 0) -> Stream #0:0 (libx264)
  
    Stream #0:0 -> #0:1 (aac (native) -> mp3 (libmp3lame))
  
  Press [q] to stop, [?] for help
  
  [swscaler @ 000002aa2355ba40] deprecated pixel format used, make sure you did set range correctly
  
  [libx264 @ 000002aa202d9600] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
  
  [libx264 @ 000002aa202d9600] profile Constrained Baseline, level 4.0
  
  [libx264 @ 000002aa202d9600] 264 - core 152 r2851 ba24899 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
  
  Output #0, flv, to 'out.mp4':
  
    Metadata:
  
      title           : session
  
      encoder         : Lavf58.3.100
  
      Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuv420p, 1920x1080, q=-1--1, 30 fps, 1k tbn, 30 tbc (default)
  
      Metadata:
  
        encoder         : Lavc58.9.100 libx264
  
      Side data:
  
        cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
  
      Stream #0:1: Audio: mp3 (libmp3lame) ([2][0][0][0] / 0x0002), 48000 Hz, stereo, fltp
  
      Metadata:
  
        encoder         : Lavc58.9.100 libmp3lame
  
  Past duration 0.711662 too large
  
  Past duration 0.822990 too large
  
  Past duration 0.854332 too large
  
  [rtsp @ 000002aa1fe4a680] max delay reached. need to consume packet
  
  [rtsp @ 000002aa1fe4a680] RTP: missed 1315 packets
  
  [h264 @ 000002aa20262580] Invalid level prefix
  
  [h264 @ 000002aa20262580] error while decoding MB 12 37
  
  [h264 @ 000002aa20262580] concealing 3757 DC, 3757 AC, 3757 MV errors in I frame
  
  Past duration 0.732323 too large     256kB time=00:00:02.88 bitrate= 727.2kbits/s dup=0 drop=13 speed=5.76x
  
  Past duration 0.824333 too large
  
  Past duration 0.988991 too large
  
  Past duration 0.862328 too large    1024kB time=00:00:03.38 bitrate=2477.4kbits/s dup=0 drop=21 speed=3.34x
  
  Past duration 0.861320 too large
  
  Past duration 0.883324 too large
  
  Past duration 0.910652 too large    1280kB time=00:00:03.94 bitrate=2659.3kbits/s dup=0 drop=30 speed=2.59x
  
  Past duration 0.954659 too large
  
  Past duration 0.790657 too large    1792kB time=00:00:04.40 bitrate=3335.6kbits/s dup=0 drop=37 speed=2.17x
  
  Past duration 0.861320 too large
  
  Past duration 0.943657 too large
  
  Past duration 0.988655 too large    2304kB time=00:00:04.90 bitrate=3849.6kbits/s dup=0 drop=45 speed=1.94x
  
  Past duration 0.810326 too large
  
  Past duration 0.893654 too large
  
  Past duration 0.822319 too large    3072kB time=00:00:05.40 bitrate=4655.2kbits/s dup=0 drop=53 speed=1.78x
  
  Past duration 0.902657 too large
  
  Past duration 0.909660 too large
  
  Past duration 0.779655 too large    3328kB time=00:00:05.93 bitrate=4596.7kbits/s dup=0 drop=61 speed=1.67x
  
  Past duration 0.957664 too large
  
  Past duration 0.778328 too large    3840kB time=00:00:06.43 bitrate=4890.0kbits/s dup=0 drop=68 speed=1.59x
  
  Past duration 0.861992 too large
  
  Past duration 0.992653 too large
  
  Past duration 0.729652 too large    4352kB time=00:00:06.94 bitrate=5133.4kbits/s dup=0 drop=76 speed=1.53x
  
  [rtsp @ 000002aa1fe4a680] max delay reached. need to consume packet
  
  [rtsp @ 000002aa1fe4a680] RTP: missed 2 packets
  
  Past duration 0.857994 too large
  
  [h264 @ 000002aa20260580] Invalid level prefix
  
  [h264 @ 000002aa20260580] error while decoding MB 43 62
  
  [h264 @ 000002aa20260580] concealing 726 DC, 726 AC, 726 MV errors in P frame
  
  Past duration 0.889000 too large
  
  Past duration 0.800987 too large    4864kB time=00:00:07.42 bitrate=5365.7kbits/s dup=0 drop=83 speed=1.47x
  
  Past duration 0.909660 too large
  
  Past duration 0.946327 too large
  
  Past duration 0.795998 too large
  
  Past duration 0.902321 too large    5376kB time=00:00:07.95 bitrate=5539.0kbits/s dup=0 drop=91 speed=1.43x
  
  Past duration 0.841652 too large
  
  Past duration 0.859657 too large
  
  Past duration 0.992332 too large    5632kB time=00:00:08.43 bitrate=5472.3kbits/s dup=0 drop=99 speed=1.39x
  
  Past duration 0.893990 too large
  
  Past duration 0.947990 too large    6400kB time=00:00:08.95 bitrate=5853.4kbits/s dup=0 drop=106 speed=1.36x
  
  Past duration 0.678322 too large
  
  Past duration 0.994987 too large
  
  Past duration 0.942329 too large
  
  Past duration 0.975319 too large
  
  Past duration 0.702660 too large    6912kB time=00:00:09.45 bitrate=5986.8kbits/s dup=0 drop=114 speed=1.34x
  
  Past duration 0.821663 too large
  
  Past duration 0.937325 too large
  
  Past duration 0.992989 too large
  
  Past duration 0.684990 too large    7168kB time=00:00:09.96 bitrate=5895.0kbits/s dup=0 drop=122 speed=1.31x
  
  Past duration 0.763664 too large
  
  Past duration 0.990318 too large
  
  Past duration 0.921333 too large
  
  Past duration 0.945320 too large
  
  Past duration 0.711327 too large    7680kB time=00:00:10.49 bitrate=5994.1kbits/s dup=0 drop=130 speed=1.29x
  
  Past duration 0.841652 too large
  
  Past duration 0.948997 too large
  
  Past duration 0.994652 too large
  
  Past duration 0.759987 too large    8448kB time=00:00:11.04 bitrate=6267.0kbits/s dup=0 drop=138 speed=1.28x
  
  Past duration 0.860985 too large
  
  Past duration 0.984993 too large
  
  Past duration 0.796989 too large    8960kB time=00:00:11.50 bitrate=6382.1kbits/s dup=0 drop=145 speed=1.26x
  
  Past duration 0.914665 too large
  
  Past duration 0.804329 too large    9472kB time=00:00:12.02 bitrate=6452.2kbits/s dup=0 drop=152 speed=1.25x
  
  frame=  303 fps= 31 q=-1.0 Lsize=   10246kB time=00:00:12.33 bitrate=6803.2kbits/s dup=0 drop=156 speed=1.25x
  
  video:10039kB audio:193kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.140463%
  
  [libx264 @ 000002aa202d9600] frame I:2     Avg QP:18.00  size:199681
  
  [libx264 @ 000002aa202d9600] frame P:301   Avg QP:18.96  size: 32823
  
  [libx264 @ 000002aa202d9600] mb I  I16..4: 100.0%  0.0%  0.0%
  
  [libx264 @ 000002aa202d9600] mb P  I16..4:  9.3%  0.0%  0.0%  P16..4: 32.1%  0.0%  0.0%  0.0%  0.0%    skip:58.6%
  
  [libx264 @ 000002aa202d9600] coded y,uvDC,uvAC intra: 12.9% 34.0% 11.0% inter: 12.0% 22.7% 6.2%
  
  [libx264 @ 000002aa202d9600] i16 v,h,dc,p: 38% 49%  9%  4%
  
  [libx264 @ 000002aa202d9600] i8c dc,h,v,p: 38% 42% 14%  6%
  
  [libx264 @ 000002aa202d9600] kb/s:6833.78
  
  Exiting normally, received signal 2.

  third :
  Combine RTMP from phone and RTSP video from server

  ffmpeg -i rtsp://192.168.1.203:554/livecast -i rtmp://192.168.1.152:1935/ingest/test -filter_complex "[0]scale=-1:-1[b];[1]scale=200:200[w];[b][w] overlay=10:10" -c:v libx264 -preset ultrafast -f flv outputRTMP.mp4
  
  ffmpeg version N-89672-g41e51fbcd9 Copyright (c) 2000-2018 the FFmpeg developers
  
    built with gcc 7.2.0 (GCC)
  
    configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-bzlib --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-amf --enable-cuda --enable-cuvid --enable-d3d11va --enable-nvenc --enable-dxva2 --enable-avisynth --enable-libmfx
  
    libavutil      56.  7.100 / 56.  7.100
  
    libavcodec     58.  9.100 / 58.  9.100
  
    libavformat    58.  3.100 / 58.  3.100
  
    libavdevice    58.  0.100 / 58.  0.100
  
    libavfilter     7.  8.100 /  7.  8.100
  
    libswscale      5.  0.101 /  5.  0.101
  
    libswresample   3.  0.101 /  3.  0.101
  
    libpostproc    55.  0.100 / 55.  0.100
  
  [udp @ 0000018c0ccbca40] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
  
  [udp @ 0000018c0ccd2900] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
  
  [udp @ 0000018c0cce3dc0] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
  
  [udp @ 0000018c0ccf4080] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
  
  Input #0, rtsp, from 'rtsp://192.168.1.203:554/livecast':
  
    Metadata:
  
      title           : session
  
    Duration: N/A, start: 0.000000, bitrate: N/A
  
      Stream #0:0: Audio: aac (LC), 48000 Hz, stereo, fltp
  
      Stream #0:1: Video: h264 (Constrained Baseline), yuv420p(progressive), 1920x1080, 30 fps, 30 tbr, 90k tbn, 60 tbc
  
  Input #1, flv, from 'rtmp://192.168.1.152:1935/ingest/test':
  
    Metadata:
  
      Server          : NGINX RTMP (github.com/arut/nginx-rtmp-module)
  
      displayWidth    : 640
  
      displayHeight   : 480
  
      fps             : 0
  
      profile         :
  
      level           :
  
    Duration: 00:00:00.00, start: 173.197000, bitrate: N/A
  
      Stream #1:0: Audio: aac (LC), 44100 Hz, mono, fltp, 47 kb/s
  
      Stream #1:1: Video: h264 (Baseline), yuv420p(tv, smpte170m/bt470bg/smpte170m, progressive), 640x480, 1999 kb/s, 29.92 fps, 29.92 tbr, 1k tbn
  
  File 'outputRTMP.mp4' already exists. Overwrite ? [y/N] y
  
  Stream mapping:
  
    Stream #0:1 (h264) -> scale (graph 0)
  
    Stream #1:1 (h264) -> scale (graph 0)
  
    overlay (graph 0) -> Stream #0:0 (libx264)
  
    Stream #0:0 -> #0:1 (aac (native) -> mp3 (libmp3lame))
  
  Press [q] to stop, [?] for help
  
  [libx264 @ 0000018c0d1321c0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
  
  [libx264 @ 0000018c0d1321c0] profile Constrained Baseline, level 4.0
  
  [libx264 @ 0000018c0d1321c0] 264 - core 152 r2851 ba24899 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
  
  Output #0, flv, to 'outputRTMP.mp4':
  
    Metadata:
  
      title           : session
  
      encoder         : Lavf58.3.100
  
      Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuv420p, 1920x1080, q=-1--1, 30 fps, 1k tbn, 30 tbc (default)
  
      Metadata:
  
        encoder         : Lavc58.9.100 libx264
  
      Side data:
  
        cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
  
      Stream #0:1: Audio: mp3 (libmp3lame) ([2][0][0][0] / 0x0002), 48000 Hz, stereo, fltp
  
      Metadata:
  
        encoder         : Lavc58.9.100 libmp3lame
  
  Past duration 0.931999 too large
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 92 packets
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 2904 packets
  
  Past duration 0.671654 too large
  
  [h264 @ 0000018c0d0da140] concealing 4160 DC, 4160 AC, 4160 MV errors in I frame
  
  Past duration 0.991997 too large
  
  Past duration 0.791328 too large     256kB time=00:00:05.12 bitrate= 409.3kbits/s dup=0 drop=12 speed=4.14x
  
  Past duration 0.969994 too large
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 573 packets
  
  [h264 @ 0000018c0d0d8a80] concealing 2468 DC, 2468 AC, 2468 MV errors in P frame
  
  Past duration 0.670326 too large
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 25 packets
  
  [h264 @ 0000018c0d0d8a80] concealing 6479 DC, 6479 AC, 6479 MV errors in P frame
  
  Past duration 0.961327 too large     512kB time=00:00:11.10 bitrate= 377.7kbits/s dup=0 drop=16 speed=1.57x
  
  Past duration 0.689323 too large
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 620 packets
  
  [h264 @ 0000018c0d0da140] negative number of zero coeffs at 33 32
  
  [h264 @ 0000018c0d0da140] error while decoding MB 33 32
  
  [h264 @ 0000018c0d0da140] concealing 4336 DC, 4336 AC, 4336 MV errors in P frame
  
  Past duration 0.790321 too large
  
  Past duration 0.910332 too large     768kB time=00:00:12.23 bitrate= 514.3kbits/s dup=0 drop=18 speed=1.62x
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 24 packets
  
  [h264 @ 0000018c0d0da140] corrupted macroblock 22 25 (total_coeff=-1)
  
  [h264 @ 0000018c0d0da140] error while decoding MB 22 25
  
  [h264 @ 0000018c0d0da140] concealing 5151 DC, 5151 AC, 5151 MV errors in P frame
  
  Past duration 0.924324 too large    1024kB time=00:00:17.03 bitrate= 492.5kbits/s dup=0 drop=19 speed= 1.3x
  
  Past duration 0.656654 too large
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 516 packets
  
  [h264 @ 0000018c0d0daf00] Invalid level prefix
  
  [h264 @ 0000018c0d0daf00] error while decoding MB 0 36
  
  [h264 @ 0000018c0d0daf00] concealing 3889 DC, 3889 AC, 3889 MV errors in P frame
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 132 packets
  
  [h264 @ 0000018c0d0d8a80] top block unavailable for requested intra mode -1
  
  [h264 @ 0000018c0d0d8a80] error while decoding MB 55 26
  
  [h264 @ 0000018c0d0d8a80] concealing 5034 DC, 5034 AC, 5034 MV errors in I frame
  
  [h264 @ 0000018c0d0d9cc0] concealing 2621 DC, 2621 AC, 2621 MV errors in P frame
  
  Past duration 0.935661 too large
  
  Past duration 0.878319 too large
  
  Past duration 0.929329 too large    1280kB time=00:00:18.32 bitrate= 572.1kbits/s dup=0 drop=22 speed=1.27x
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 560 packets
  
  [h264 @ 0000018c0d0d8140] Invalid level prefix
  
  Past duration 0.661659 too large
  
  [h264 @ 0000018c0d0d8140] error while decoding MB 91 24
  
  [h264 @ 0000018c0d0d8140] concealing 5214 DC, 5214 AC, 5214 MV errors in P frame
  
  Past duration 0.823662 too large
  
  Past duration 0.864662 too large
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 29 packets
  
  [h264 @ 0000018c0d0d8140] corrupted macroblock 42 3 (total_coeff=-1)
  
  [h264 @ 0000018c0d0d8140] error while decoding MB 42 3
  
  [h264 @ 0000018c0d0d8140] concealing 7760 DC, 7760 AC, 7760 MV errors in P frame
  
  Past duration 0.780663 too large    1792kB time=00:00:19.50 bitrate= 752.8kbits/s dup=0 drop=24 speed=1.27x
  
  Past duration 0.900322 too large
  
  Past duration 0.990990 too large
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 565 packets
  
  [h264 @ 0000018c0d0d8140] corrupted macroblock 83 27 (total_coeff=-1)
  
  [h264 @ 0000018c0d0d8140] Past duration 0.862999 too large
  
  error while decoding MB 83 27
  
  [h264 @ 0000018c0d0d8140] concealing 4866 DC, 4866 AC, 4866 MV errors in P frame
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 7 packets
  
  [h264 @ 0000018c0d0d8f00] top block unavailable for requested intra mode
  
  [h264 @ 0000018c0d0d8f00] error while decoding MB 79 16
  
  [h264 @ 0000018c0d0d8f00] concealing 2661 DC, 2661 AC, 2661 MV errors in P frame
  
  Past duration 0.907997 too large
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet= 816.0kbits/s dup=0 drop=28 speed=1.29x
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 19 packets
  
  [h264 @ 0000018c0d0d9cc0] corrupted macroblock 88 59 (total_coeff=-1)
  
  [h264 @ 0000018c0d0d9cc0] error while decoding MB 88 59
  
  [h264 @ 0000018c0d0d9cc0] concealing 1041 DC, 1041 AC, 1041 MV errors in P frame
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 6 packets
  
  [h264 @ 0000018c0d0d8140] out of range intra chroma pred mode
  
  [h264 @ 0000018c0d0d8140] error while decoding MB 118 6
  
  [h264 @ 0000018c0d0d8140] concealing 3320 DC, 3320 AC, 3320 MV errors in P frame
  
  Past duration 0.845665 too large    2304kB time=00:00:20.72 bitrate= 910.6kbits/s dup=0 drop=29 speed=1.23x
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 574 packets
  
  [h264 @ 0000018c0d0d8140] negative number of zero coeffs at 39 26
  
  [h264 @ 0000018c0d0d8140] error while decoding MB 39 26
  
  [h264 @ 0000018c0d0d8140] concealing 5050 DC, 5050 AC, 5050 MV errors in P frame
  
  Past duration 0.950996 too large
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 37 packets
  
  [h264 @ 0000018c0d0d8140] corrupted macroblock 48 37 (total_coeff=-1)
  
  [h264 @ 0000018c0d0d8140] error while decoding MB 48 37
  
  [h264 @ 0000018c0d0d8140] concealing 3721 DC, 3721 AC, 3721 MV errors in P frame
  
  Past duration 0.789330 too large    2816kB time=00:00:22.02 bitrate=1047.5kbits/s dup=0 drop=33 speed=1.23x
  
  Past duration 0.848320 too large
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 581 packets
  
  [h264 @ 0000018c0d0d8f00] corrupted macroblock 13 10 (total_coeff=-1)
  
  [h264 @ 0000018c0d0d8f00] error while decoding MB 13 10
  
  Past duration 0.875664 too large
  
  [h264 @ 0000018c0d0d8f00] concealing 6952 DC, 6952 AC, 6952 MV errors in P frame
  
  [rtsp @ 0000018c0ccba6c0] max delay reached. need to consume packet=1181.0kbits/s dup=0 drop=35 speed=1.25x
  
  [rtsp @ 0000018c0ccba6c0] RTP: missed 6 packets
  
  [h264 @ 0000018c0d0d8f00] corrupted macroblock 72 37 (total_coeff=-1)
  
  [h264 @ 0000018c0d0d8f00] error while decoding MB 72 37
  
  [h264 @ 0000018c0d0d8f00] concealing 3675 DC, 3675 AC, 3675 MV errors in P frame
  
  frame=  101 fps=5.3 q=-1.0 Lsize=    4095kB time=00:00:23.67 bitrate=1417.0kbits/s dup=0 drop=37 speed=1.25x
  
  video:3739kB audio:340kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.405426%
  
  [libx264 @ 0000018c0d1321c0] frame I:1     Avg QP:20.00  size:178581
  
  [libx264 @ 0000018c0d1321c0] frame P:100   Avg QP:20.42  size: 36495
  
  [libx264 @ 0000018c0d1321c0] mb I  I16..4: 100.0%  0.0%  0.0%
  
  [libx264 @ 0000018c0d1321c0] mb P  I16..4: 11.8%  0.0%  0.0%  P16..4: 31.7%  0.0%  0.0%  0.0%  0.0%    skip:56.5%
  
  [libx264 @ 0000018c0d1321c0] coded y,uvDC,uvAC intra: 26.5% 36.0% 9.2% inter: 15.0% 18.5% 3.4%
  
  [libx264 @ 0000018c0d1321c0] i16 v,h,dc,p: 44% 39% 10%  7%
  
  [libx264 @ 0000018c0d1321c0] i8c dc,h,v,p: 36% 36% 21%  7%
  
  [libx264 @ 0000018c0d1321c0] kb/s:1383.65
  
  Exiting normally, received signal 2.

  So If you have some ideas for me it will help. Thank you

• Revision 30966 : eviter le moche ’doctype_ecrire’ lors de l’upgrade

  17 août 2009, par fil@… — Log

  eviter le moche ’doctype_ecrire’ lors de l’upgrade