Sur d’autres sites (5335)

• Combine Audio and Images in Stream

  19 décembre 2017, par SenorContento

  I would like to be able to create images on the fly and also create audio on the fly too and be able to combine them together into an rtmp stream (for Twitch or YouTube). The goal is to accomplish this in Python 3 as that is the language my bot is written in. Bonus points for not having to save to disk.

  So far, I have figured out how to stream to rtmp servers using ffmpeg by loading a PNG image and playing it on loop as well as loading a mp3 and then combining them together in the stream. The problem is I have to load at least one of them from file.

  I know I can use Moviepy to create videos, but I cannot figure out whether or not I can stream the video from Moviepy to ffmpeg or directly to rtmp. I think that I have to generate a lot of really short clips and send them, but I want to know if there’s an existing solution.

  There’s also OpenCV which I hear can stream to rtmp, but cannot handle audio.

  A redacted version of an ffmpeg command I have successfully tested with is

  ffmpeg -loop 1 -framerate 15 -i ScreenRover.png -i "Song-Stereo.mp3" -c:v libx264 -preset fast -pix_fmt yuv420p -threads 0 -f flv rtmp://SITE-SUCH-AS-TWITCH/.../STREAM-KEY

  or

  cat Song-Stereo.mp3 | ffmpeg -loop 1 -framerate 15 -i ScreenRover.png -i - -c:v libx264 -preset fast -pix_fmt yuv420p -threads 0 -f flv rtmp://SITE-SUCH-AS-TWITCH/.../STREAM-KEY

  I know these commands are not set up properly for smooth streaming, the result manages to screw up both Twitch’s and Youtube’s player and I will have to figure out how to fix that.

  The problem with this is I don’t think I can stream both the image and the audio at once when creating them on the spot. I have to load one of them from the hard drive. This becomes a problem when trying to react to a command or user chat or anything else that requires live reactions. I also do not want to destroy my hard drive by constantly saving to it.

  As for the python code, what I have tried so far in order to create a video is the following code. This still saves to the HD and is not responsive in realtime, so this is not very useful to me. The video itself is okay, with the one exception that as time passes on, the clock the qr code says versus the video’s clock start to spread apart farther and farther as the video gets closer to the end. I can work around that limitation if it shows up while live streaming.

  def make_frame(t):
  
    img = qrcode.make("Hello! The second is %s!" % t)
  
    return numpy.array(img.convert("RGB"))
  
  
  
  clip = mpy.VideoClip(make_frame, duration=120)
  
  clip.write_gif("test.gif",fps=15)
  
  
  
  gifclip = mpy.VideoFileClip("test.gif")
  
  gifclip.set_duration(120).write_videofile("test.mp4",fps=15)

  My goal is to be able to produce something along the psuedo-code of

  original_video = qrcode_generator("I don't know, a clock, pyotp, today's news sources, just anything that can be generated on the fly!")
  
  original_video.overlay_text(0,0,"This is some sample text, the left two are coordinates, the right three are font, size, and color", Times_New_Roman, 12, Blue)
  
  original_video.add_audio(sine_wave_generator(0,180,2)) # frequency min-max, seconds
  
  
  
  # NOTICE - I did not add any time measurements to the actual video itself. The whole point is this is a live stream and not a video clip, so the time frame would be now. The 2 seconds list above is for our psuedo sine wave generator to know how long the audio clip should be, not for the actual streaming library.
  
  
  
  stream.send_to_rtmp_server(original_video) # Doesn't matter if ffmpeg or some native library

  The above example is what I am looking for in terms of video creation in Python and then streaming. I am not trying to create a clip and then stream it later, I am trying to have the program be able to respond to outside events and then update it’s stream to do whatever it wants. It is sort of like a chat bot, but with video instead of text.

  def track_movement(...):
  
    ...
  
    return ...
  
  
  
  original_video = user_submitted_clip(chat.lastVideoMessage)
  
  original_video.overlay_text(0,0,"The robot watches the user's movements and puts a blue square around it.", Times_New_Roman, 12, Blue)
  
  original_video.add_audio(sine_wave_generator(0,180,2)) # frequency min-max, seconds
  
  
  
  # It would be awesome if I could also figure out how to perform advance actions such as tracking movements or pulling a face out of a clip and then applying effects to it on the fly. I know OpenCV can track movements and I hear that it can work with streams, but I cannot figure out how that works. Any help would be appreciated! Thanks!

  Because I forgot to add the imports, here are some useful imports I have in my file !

  import pyotp
  
  import qrcode
  
  from io import BytesIO
  
  from moviepy import editor as mpy

  The library, pyotp, is for generating one time pad authenticator codes, qrcode is for the qr codes, BytesIO is used for virtual files, and moviepy is what I used to generate the GIF and MP4. I believe BytesIO might be useful for piping data to the streaming service, but how that happens, depends entirely on how data is sent to the service, whether it be ffmpeg over command line (from subprocess import Popen, PIPE) or it be a native library.

• FFmpeg : What re-encoding settings can be used to achieve results similar to Google Drive's video processing ?

  4 août 2023, par Mycroft_47

  Context :

  


  I have a large collection of videos recorded by my phone's camera, which is taking up a significant amount of space. Recently, I noticed that when I uploaded a video to Google Drive and then downloaded it again using IDM (by clicking on the pop-up that IDM displays when it detects something that can be downloaded here's what i mean), the downloaded video retained the same visual quality but occupied much less space. Upon further research, I discovered that Google re-encodes uploaded videos using H.264 video encoding, and I believe I can achieve similar compression using FFmpeg.

  


  Problem :

  


  Despite experimenting with various FFmpeg commands, I haven't been able to replicate Google Drive's compression. Every attempt using -codec:v libx264 option alone resulted in videos larger than the original files.

  


  While adjusting the -crf parameter to a higher value and opting for a faster -preset option did yield smaller file sizes, it unfortunately came at the cost of a noticeable degradation in visual quality and the appearance of some visible artifacts in the video.

  


  Google Drive's processing, on the other hand, strikes a commendable balance, achieving a satisfactory file size without compromising visual clarity, (I should note that upon zooming in on this video, I observed some minor blurring, but it was acceptable to me).

  


  Note :

  


  I'm aware that using the H.265 video encoder instead of H.264 may give better results. However, to ensure fairness and avoid any potential bias, I think the optimal approach is first to find the best command using the H.264 video encoder. Once identified, I can then replace -codec:v libx264 with -codec:v libx265. This approach will ensure that the chosen command is really the best that FFMPEG can achieve, and that it is not solely influenced by the superior performance of H.265 when used from the outset.

  


  Here's the FFMPEG command I am currently using :

  


  ffmpeg -hide_banner -loglevel verbose ^
    -i input.mp4 ^
    -codec:v libx264 ^
    -crf 36 -preset ultrafast ^
    -codec:a libopus -b:a 112k ^
    -movflags use_metadata_tags+faststart -map_metadata 0 ^
    output.mp4


  


  





  


  


  


  


  


  


  


  



  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  


  Video file	Size (bytes)	Bit rate (bps)	Encoder	FFPROB - JSON
  Original (named 'raw 1.mp4')	31,666,777	10,314,710	!!!	link
  Without crf	36,251,852	11,805,216	Lavf60.3.100	link
  With crf	10,179,113	3,314,772	Lavf60.3.100	link
  Gdrive	6,726,189	2,190,342	Google	link

  


  


  Those files can be found here.

  


  Update :

  


  I continued my experiments with the video "raw_1.mp4" and found some interesting results that resemble those shown in this blog post, (I recommend consulting this answer).

  


  In the following figure, I observed that using the -preset set to veryfast provided the most advantageous results, striking the optimal balance between compression ratio and compression time, (Note that a negative percentage in the compression variable indicates an increase in file size after processing) :


  


  In this figure, I used the H.264 encoder and compared the compression ratio of different outputted files resulting from seven different values of the -crf parameter (CRF values used : 25, 27, 29, 31, 33, 35, 37),


  


  For this figure, I've switched the encoder to H.265 while maintaining the same CRF values used in the previous figure :


  


  Based on these results, the -preset veryfast and a -crf value of 31 are my current preferred settings for FFmpeg, until they are proven to be suboptimal choices.
As a result, the FFmpeg command I'll use is as follows :

  


  ffmpeg -hide_banner -loglevel verbose ^
    -i input.mp4 ^
    -codec:v libx264 ^
    -crf 31 -preset veryfast ^
    -codec:a libopus -b:a 112k ^
    -movflags use_metadata_tags+faststart -map_metadata 0 ^
    output.mp4


  


  Note that these choices are based solely on the compression results obtained so far, and they do not take into account the visual quality of the outputted files.

  


• How to save animations with tight layout, transparency and in high quality with matplotlib ?

  29 novembre 2019, par mapf

  I am trying to implement an option in my GUI to save an image sequence displayed using matplotlib. The code looks something like this :

  import matplotlib.pyplot as plt
  
  from matplotlib.backends.backend_qt5agg import \
  
      FigureCanvasQTAgg as FigureCanvas
  
  from matplotlib.animation import FuncAnimation
  
  from PIL import Image
  
  
  
  
  
  plt.rcParams['savefig.bbox'] = 'tight' 
  
  
  
  
  
  class Printer:
  
      def __init__(self, data):
  
          self.fig, self.ax = plt.subplots()
  
          self.canvas = FigureCanvas(self.fig)
  
  
  
          # some irrelevant color adjustment here
  
          #self.ax.spines['bottom'].set_color('#f9f2d7')
  
          #self.ax.spines['top'].set_color('#f9f2d7')
  
          #self.ax.spines['right'].set_color('#f9f2d7')
  
          #self.ax.spines['left'].set_color('#f9f2d7')
  
          #self.ax.tick_params(axis='both', colors='#f9f2d7')
  
          #self.ax.yaxis.label.set_color('#f9f2d7')
  
          #self.ax.xaxis.label.set_color('#f9f2d7')
  
          #self.fig.subplots_adjust(left=0.1, right=0.975, bottom=0.09, top=0.98)
  
          self.fig.patch.set_alpha(0)
  
          self.fig.patch.set_visible(False)
  
          self.canvas.setStyleSheet("background-color:transparent;")
  
          self.fig.set_size_inches(10, 10, True)
  
          self.fig.tight_layout()
  
  
  
          self.data = data
  
          self.image_artist = self.ax.imshow(data[0])
  
  
  
      def animate(self, i):
  
          self.image_artist.set_data(self.data[i])
  
          self.canvas.draw()
  
  
  
  
  
  def save_animation():
  
      data = [
  
          Image.open("test000.png"),
  
          Image.open("test001.png"),
  
      ]
  
      file = 'test.gif'
  
      printer = Printer(data)
  
  
  
      ani = FuncAnimation(
  
          printer.fig, printer.animate, interval=100, frames=len(data),
  
      )
  
      # writer = animation.writers['pillow'](bitrate=1000)
  
      ani.save(
  
          file, writer='pillow', savefig_kwargs={'transparent': True, 'bbox_inches': 'tight'}
  
      )
  
  
  
  
  
  save_animation()

  Transparency :

  As you can see I have already tried several different approaches as suggested elsewhere (1, 2), but didn’t manage to find a solution. All of the settings and arguments patch.set_alpha(0), patch.set_visible(False), canvas.setStyleSheet("background-color:transparent;"), savefig_kwargs={'transparent': True} seem to have no effect at all on the transparency. I found this post but I didn’t get the code to work (for one I had to comment out this %matplotlib inline, but then I ended up getting some error during the MovieWriter.cleanup out = TextIOWrapper(BytesIO(out)).read() TypeError: a bytes-like object is required, not 'str'). Here, it was suggested that this is actually a bug, but the proposed workaroud doesn’t work for me since I would have to rely on third-party software. There also exists this bug report which was supposedly solved, so maybe it is unrelated.

  Tight layout

  I actually couldn’t really find much on this, but all the things I tried (plt.rcParams['savefig.bbox'] = 'tight', fig.tight_layout(), savefig_kwargs={'bbox_inches': 'tight'}) don’t have any effect or are even actively discarded in the case of the bbox_inches argument. How does this work ?

  High quality

  Since I cannot use ImageMagick and can’t get ffmpeg to work (more on this below), I rely on pillow to save my animation. But the only argument in terms of quality that I can pass on seems to be the bitrate, which doesn’t have any effect. The files still have the same size and the animation still looks like mush. The only way that I found to increase the resolution was to use fig.set_size_inches(10, 10, True), but this still doesn’t improve the overall quality of the animation. It still looks bad. I saw that you can pass on codec and extra_args so maybe that is something that might help, but I have no idea how to use these because I couldn’t find a list with allowed arguments.

  ffmpeg

  I can’t get ffmpeg to work. I installed the python package from here and can import it into a python session but I don’t know how I can get matplotlib to use that. I also got ffmpeg from here (Windows 64-bit version) and set the plt.rcParams['animation.ffmpeg_path'] to where I saved the files (there was no intaller to run, not sure if I did it correctly). But this didn’t help either. Also this is of course also third-party software, so if somebody else were to use my code/program it wouldn’t work.