Recherche avancée

Sur d’autres sites (12830)

• MediaCodec - save timing info for ffmpeg ?

  18 novembre 2014, par Mark

  I have a requirement to encrypt video before it hits the disk. It seems on Android the only way to do this is to use MediaCodec, and encrypt and save the raw h264 elementary streams. (The MediaRecorder and Muxer classes operate on FileDescriptors, not an OutputStream, so I can’t wrap it with a CipherOutputStream).

  Using the grafika code as a base, I’m able to save a raw h264 elementary stream by replacing the Muxer in the VideoEncoderCore class with a WriteableByteChannel, backed by a CipherOutputStream (code below, minus the CipherOutputStream).

  If I take the resulting output file over to the desktop I’m able to use ffmpeg to mux the h264 stream to a playable mp4 file. What’s missing however is timing information. ffmpeg always assumes 25fps. What I’m looking for is a way to save the timing info, perhaps to a separate file, that I can use to give ffmpeg the right information on the desktop.

  I’m not doing audio yet, but I can imagine I’ll need to do the same thing there, if I’m to have any hope of remotely accurate syncing.

  FWIW, I’m a total newbie here, and I really don’t know much of anything about SPS, NAL, Atoms, etc.

  /*
  
   * Copyright 2014 Google Inc. All rights reserved.
  
   *
  
   * Licensed under the Apache License, Version 2.0 (the "License");
  
   * you may not use this file except in compliance with the License.
  
   * You may obtain a copy of the License at
  
   *
  
   *      http://www.apache.org/licenses/LICENSE-2.0
  
   *
  
   * Unless required by applicable law or agreed to in writing, software
  
   * distributed under the License is distributed on an "AS IS" BASIS,
  
   * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  
   * See the License for the specific language governing permissions and
  
   * limitations under the License.
  
   */
  
  
  
  
  
  import android.media.MediaCodec;
  
  import android.media.MediaCodecInfo;
  
  import android.media.MediaFormat;
  
  import android.util.Log;
  
  import android.view.Surface;
  
  
  
  import java.io.BufferedOutputStream;
  
  import java.io.File;
  
  import java.io.FileOutputStream;
  
  import java.io.IOException;
  
  import java.nio.ByteBuffer;
  
  import java.nio.channels.Channels;
  
  import java.nio.channels.WritableByteChannel;
  
  
  
  /**
  
   * This class wraps up the core components used for surface-input video encoding.
  
   * <p>
  
   * Once created, frames are fed to the input surface.  Remember to provide the presentation
  
   * time stamp, and always call drainEncoder() before swapBuffers() to ensure that the
  
   * producer side doesn't get backed up.
  
   * </p><p>
  
   * This class is not thread-safe, with one exception: it is valid to use the input surface
  
   * on one thread, and drain the output on a different thread.
  
   */
  
  public class VideoEncoderCore {
  
      private static final String TAG = MainActivity.TAG;
  
      private static final boolean VERBOSE = false;
  
  
  
      // TODO: these ought to be configurable as well
  
      private static final String MIME_TYPE = "video/avc";    // H.264 Advanced Video Coding
  
      private static final int FRAME_RATE = 30;               // 30fps
  
      private static final int IFRAME_INTERVAL = 5;           // 5 seconds between I-frames
  
  
  
      private Surface mInputSurface;
  
      private MediaCodec mEncoder;
  
      private MediaCodec.BufferInfo mBufferInfo;
  
      private int mTrackIndex;
  
      //private MediaMuxer mMuxer;
  
      //private boolean mMuxerStarted;
  
      private WritableByteChannel outChannel;
  
  
  
      /**
  
       * Configures encoder and muxer state, and prepares the input Surface.
  
       */
  
      public VideoEncoderCore(int width, int height, int bitRate, File outputFile)
  
              throws IOException {
  
          mBufferInfo = new MediaCodec.BufferInfo();
  
  
  
          MediaFormat format = MediaFormat.createVideoFormat(MIME_TYPE, width, height);
  
  
  
          // Set some properties.  Failing to specify some of these can cause the MediaCodec
  
          // configure() call to throw an unhelpful exception.
  
          format.setInteger(MediaFormat.KEY_COLOR_FORMAT,
  
                  MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
  
          format.setInteger(MediaFormat.KEY_BIT_RATE, bitRate);
  
          format.setInteger(MediaFormat.KEY_FRAME_RATE, FRAME_RATE);
  
          format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, IFRAME_INTERVAL);
  
          if (VERBOSE) Log.d(TAG, "format: " + format);
  
  
  
          // Create a MediaCodec encoder, and configure it with our format.  Get a Surface
  
          // we can use for input and wrap it with a class that handles the EGL work.
  
          mEncoder = MediaCodec.createEncoderByType(MIME_TYPE);
  
          mEncoder.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
  
          mInputSurface = mEncoder.createInputSurface();
  
          mEncoder.start();
  
  
  
          // Create a MediaMuxer.  We can't add the video track and start() the muxer here,
  
          // because our MediaFormat doesn't have the Magic Goodies.  These can only be
  
          // obtained from the encoder after it has started processing data.
  
          //
  
          // We're not actually interested in multiplexing audio.  We just want to convert
  
          // the raw H.264 elementary stream we get from MediaCodec into a .mp4 file.
  
          //mMuxer = new MediaMuxer(outputFile.toString(),
  
          //        MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
  
  
  
          mTrackIndex = -1;
  
          //mMuxerStarted = false;
  
          outChannel = Channels.newChannel(new BufferedOutputStream(new FileOutputStream(outputFile)));
  
      }
  
  
  
      /**
  
       * Returns the encoder's input surface.
  
       */
  
      public Surface getInputSurface() {
  
          return mInputSurface;
  
      }
  
  
  
      /**
  
       * Releases encoder resources.
  
       */
  
      public void release() {
  
          if (VERBOSE) Log.d(TAG, "releasing encoder objects");
  
          if (mEncoder != null) {
  
              mEncoder.stop();
  
              mEncoder.release();
  
              mEncoder = null;
  
          }
  
          try {
  
              outChannel.close();
  
          }
  
          catch (Exception e) {
  
              Log.e(TAG,"Couldn't close output stream.");
  
          }
  
      }
  
  
  
      /**
  
       * Extracts all pending data from the encoder and forwards it to the muxer.
  
       * </p><p>
  
       * If endOfStream is not set, this returns when there is no more data to drain.  If it
  
       * is set, we send EOS to the encoder, and then iterate until we see EOS on the output.
  
       * Calling this with endOfStream set should be done once, right before stopping the muxer.
  
       * </p><p>
  
       * We're just using the muxer to get a .mp4 file (instead of a raw H.264 stream).  We're
  
       * not recording audio.
  
       */
  
      public void drainEncoder(boolean endOfStream) {
  
          final int TIMEOUT_USEC = 10000;
  
          if (VERBOSE) Log.d(TAG, "drainEncoder(" + endOfStream + ")");
  
  
  
          if (endOfStream) {
  
              if (VERBOSE) Log.d(TAG, "sending EOS to encoder");
  
              mEncoder.signalEndOfInputStream();
  
          }
  
  
  
          ByteBuffer[] encoderOutputBuffers = mEncoder.getOutputBuffers();
  
          while (true) {
  
              int encoderStatus = mEncoder.dequeueOutputBuffer(mBufferInfo, TIMEOUT_USEC);
  
              if (encoderStatus == MediaCodec.INFO_TRY_AGAIN_LATER) {
  
                  // no output available yet
  
                  if (!endOfStream) {
  
                      break;      // out of while
  
                  } else {
  
                      if (VERBOSE) Log.d(TAG, "no output available, spinning to await EOS");
  
                  }
  
              } else if (encoderStatus == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
  
                  // not expected for an encoder
  
                  encoderOutputBuffers = mEncoder.getOutputBuffers();
  
              } else if (encoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
  
                  // should happen before receiving buffers, and should only happen once
  
                  //if (mMuxerStarted) {
  
                  //    throw new RuntimeException("format changed twice");
  
                  //}
  
                  MediaFormat newFormat = mEncoder.getOutputFormat();
  
                  Log.d(TAG, "encoder output format changed: " + newFormat);
  
  
  
                  // now that we have the Magic Goodies, start the muxer
  
                  //mTrackIndex = mMuxer.addTrack(newFormat);
  
                  //mMuxer.start();
  
                  //mMuxerStarted = true;
  
              } else if (encoderStatus &lt; 0) {
  
                  Log.w(TAG, "unexpected result from encoder.dequeueOutputBuffer: " +
  
                          encoderStatus);
  
                  // let's ignore it
  
              } else {
  
                  ByteBuffer encodedData = encoderOutputBuffers[encoderStatus];
  
                  if (encodedData == null) {
  
                      throw new RuntimeException("encoderOutputBuffer " + encoderStatus +
  
                              " was null");
  
                  }
  
  
  
                  /*
  
                     FFMPEG needs this info.
  
                  if ((mBufferInfo.flags &amp; MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
  
                      // The codec config data was pulled out and fed to the muxer when we got
  
                      // the INFO_OUTPUT_FORMAT_CHANGED status.  Ignore it.
  
                      if (VERBOSE) Log.d(TAG, "ignoring BUFFER_FLAG_CODEC_CONFIG");
  
                      mBufferInfo.size = 0;
  
                  }
  
                  */
  
  
  
                  if (mBufferInfo.size != 0) {
  
                      /*
  
                      if (!mMuxerStarted) {
  
                          throw new RuntimeException("muxer hasn't started");
  
                      }
  
                      */
  
  
  
                      // adjust the ByteBuffer values to match BufferInfo (not needed?)
  
                      encodedData.position(mBufferInfo.offset);
  
                      encodedData.limit(mBufferInfo.offset + mBufferInfo.size);
  
  
  
                      try {
  
                          outChannel.write(encodedData);
  
                      }
  
                      catch (Exception e) {
  
                          Log.e(TAG,"Error writing output.",e);
  
                      }
  
                      if (VERBOSE) {
  
                          Log.d(TAG, "sent " + mBufferInfo.size + " bytes to muxer, ts=" +
  
                                  mBufferInfo.presentationTimeUs);
  
                      }
  
                  }
  
  
  
                  mEncoder.releaseOutputBuffer(encoderStatus, false);
  
  
  
                  if ((mBufferInfo.flags &amp; MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
  
                      if (!endOfStream) {
  
                          Log.w(TAG, "reached end of stream unexpectedly");
  
                      } else {
  
                          if (VERBOSE) Log.d(TAG, "end of stream reached");
  
                      }
  
                      break;      // out of while
  
                  }
  
              }
  
          }
  
      }
  
  }
  
  </p>

• Diamond Rio Artifacts

  30 août 2012, par Multimedia Mike — Multimedia History

  Remember the Diamond Rio PMP300 ? It’s credited with being the very first portable MP3 player, released all the way back in 1998 (I say ‘credited’ because I visited an audio museum once which exhibited a Toshiba MP3 player from 1997). I recently rescued a pristine set of Rio artifacts from a recycle pile.

  
  
  

  I wondered if I should scan the manual for posterity. However, a Google search indicates that a proper PDF (loaded with pleas to not illegally copy music) isn’t very difficult to come by. Here are the other items that came with the unit :

  
  
  Click for larger image
  

  Ah, more memories (of dialup internet) : A tie-in with another Diamond product, this time a modem which claims to enable the user to download songs at up to 112 kilobits per second. I wonder if that was really possible. I remember that 56k modems were a stretch and 33.6k was the best that most users could hope for.

  There is also a separate piece of paper that advises the buyer that the parallel port adapter might look a bit different than what is seen in the printed copy. Imagine the age of downloading to your MP3 player via parallel port while pulling down new songs via dialup internet.

  The artifacts also included not one, but two CD-ROMs :

  
  
  Click for larger image
  

  One is a driver and software disc, so no big surprise there. The other has a selection of MP3 files for your shiny new MP3 player. I’m wondering if these should be proactively preserved. I was going to process the files’ metadata and publish it here, for the benefit of search engines. However, while metadata is present, the files don’t conform to any metadata format that FFmpeg/Libav recognize. The files mention Brava Software Inc. in their metadata sections. Still, individual filenames at the end of this post.

  Leftovers :
  A few other miscellaneous multimedia acquisitions :

  
  
  

  I still want to study all of these old multimedia creation programs in depth some day. Theatrix Hollywood is a creative writing game, Wikipedia alleges (I’m a bit rigid with my exact definition of what constitutes a game). Here is an example movie output from this software. Meanwhile, the Mad Dog Multimedia CD-ROM apparently came packaged with a 56X CD-ROM drive (roughly the pinnacle of CD-ROM speeds). I found it has some version of Sonic Foundry’s ACID software, thus making good on the “applications” claim on the CD-ROM copy.

  Diamond Rio MP3 Sampler
  These are the names of the MP3 files found on the Diamond Rio MP3 sampler for the benefit of search engines.

  13_days.mp3
  albert_einstein_dreams.mp3
  a_man_of_many_colours.mp3
  anything_for_love.mp3
  a_secret_place.mp3
  bake_sale.mp3
  bigger_than_the_both_of_us.mp3
  boogie_beat.mp3
  bring_it_on.mp3
  buskersoundcheck_hippo.mp3
  charm.mp3
  chemical_disturbance.mp3
  coastin.mp3
  credit_is_due.mp3
  dance_again.mp3
  destiny.mp3
  dig_a_little_deeper.mp3
  diplomat6_bigmouthshut.mp3
  dirty_littlemonster.mp3
  dirty.mp3
  drivin.mp3
  Eric_Clapton_Last_Train.mp3
  etude_in_c_sharp_minor_op_42_n.mp3
  everybody_here.mp3
  freedom_4_all.mp3
  grandpas_advice.mp3
  groove.mp3
  heartland.mp3
  he_loved_her_so.mp3
  highway_to_hell.mp3
  hit_the_ground_runnin.mp3
  i_feel_fine_today.mp3
  im_not_lost_im_exploring.mp3
  into_the_void.mp3
  its_alright.mp3
  i_will_be_there.mp3
  i_will_pass_this_way_again.mp3
  juiceboxwilly_hepcat.mp3
  just_an_illusion.mp3
  keepin_time_by_the_river.mp3
  king_of_the_brooklyn_delta.mp3
  lovermilou_ringingbell.mp3
  middle_aged_rock_and_rollers.mp3
  midnight_high.mp3
  mr_schwinn.mp3
  my_brilliant_masterpiece.mp3
  my_gallery.mp3
  on_the_river_road.mp3
  pouring_rain.mp3
  prayer.mp3
  rats_in_my_bedroom.mp3
  razor_serpent_and_the_dub_mix.mp3
  ruthbuzzy_pleasestophangin.mp3
  secret_love.mp3
  ships.mp3
  silence_the_thunder.mp3
  sleeping_beauty.mp3
  slow_burn.mp3
  standing_in_my_own_way.mp3
  take_no_prisoners.mp3
  takin_up_space.mp3
  Taylor_Dayne_Unstoppable.mp3
  the_laundromat_song.mp3
  the_old_dun_cow.mp3
  the_people_i_meet.mp3
  trip_trigger_avenue.mp3
  tru-luv.mp3
  unfortunate_man.mp3
  vertigo.mp3
  when_she_runs.mp3
  where_do_we_go_from_here.mp3
  words_of_earnest.mp3
  

• Vedanti and Max Sound vs. Google

  14 août 2014, par Multimedia Mike — Legal/Ethical

  Vedanti Systems Limited (VSL) and Max Sound Coporation filed a lawsuit against Google recently. Ordinarily, I wouldn’t care about corporate legal battles. However, this one interests me because it’s multimedia-related. I’m curious to know how coding technology patents might hold up in a real court case.

  Here’s the most entertaining complaint in the lawsuit :

  Despite Google’s well-publicized Code of Conduct — “Don’t be Evil” — which it explains is “about doing the right thing,” “following the law,” and “acting honorably,” Google, in fact, has an established pattern of conduct which is the exact opposite of its claimed piety.

  I wonder if this is the first known case in which Google has been sued over its long-obsoleted “Don’t be evil” mantra ?

  Researching The Plaintiffs
  
  I think I made a mistake by assuming this lawsuit might have merit. My first order of business was to see what the plaintiff organizations have produced. I have a strong feeling that these might be run of the mill patent trolls.

  VSL currently has a blank web page. Further, the Wayback Machine only has pages reaching back to 2011. The earliest page lists these claims against a plain black background (I’ve highlighted some of the more boisterous claims and the passages that make it appear that Vedanti doesn’t actually produce anything but is strictly an IP organization) :

  The inventions key :
  The patent and software reduced any data content, without compressing, up to a 97% total reduction of the data which also produces a lossless result. This physics based invention is often called the Holy Grail.

  Vedanti Systems Intellectual Property
  Our strategic IP portfolio is granted in all of the world’s largest technology development and use countries. A major value indemnification of our licensee products is the early date of invention filing and subsequent Issue. Vedanti IP has an intrinsic 20 year patent protection and valuation in royalties and licensing. The original data transmission art has no prior art against it.

  Vedanti Systems invented among other firsts, The Slice and Partitioning of Macroblocks within a RGB Tri level region in a frame to select or not, the pixel.

  Vedanti Systems invention is used in nearly every wireless chipset and handset in the world

  Our original pixel selection system revolutionized wireless handset communications. An example of this system “Slice” and “Macroblock Partitioning” is used throughout Satellite channel expansion, Wireless partitioning, Telecom – Video Conferencing, Surveillance Cameras, and 2010 developing Media applications.

  Vedanti Systems is a Semiconductor based software, applications, and IP Continuations Intellectual Property company.

  Let’s move onto the other plaintiff, Max Sound. They have a significantly more substantive website. They also have an Android app named Spins HD Audio, which appears to be little more than a music player based on the screenshots.

  Max Sound also has a stock ticker symbol : MAXD. Something clicked into place when I looked up their ticker symbol : While worth only a few pennies, it was worth a few more pennies after this lawsuit was announced, which might be one of the motivations behind the lawsuit.

  Here’s a trick I learned when I was looking for a new tech job last year : When I first look at a company’s website and am trying to figure out what they really do, I head straight to their jobs/careers page. A lot of corporate websites have way too much blathering corporatese that can be tough to cut through. But when I see what mix of talent and specific skills they are hoping to hire, that gives me a much better portrait of what the company does.

  The reason I bring this up is because this tech company doesn’t seem to have jobs/careers page.

  The Lawsuit
  The core complaint centers around Patent 7974339 : Optimized data transmission system and method. It was filed in July 2004 (or possibly as early as January 2002), issued in July 2011, and assigned (purchased ?) by Vedanti in May 2012. The lawsuit alleges that nearly everything Google has ever produced (or, more accurately, purchased) leverages the patented technology.

  The patent itself has 5 drawings. If you’ve ever seen a multimedia codec patent, or any whitepaper on a multimedia codec, you’ve seen these graphs before. E.g., “Raw pixels come in here -> some analysis happens here -> more analysis happens over here -> entropy coding -> final bitstream”. The text of a patent document isn’t meant to be particularly useful. I’ve tried to understand this stuff before and it never goes well. Skimming the text, I just see a blur of the words data, transmission, pixel, and matrix.

  So I read the complaint to try to figure out what this is all about. To summarize the storyline as narrated by the lawsuit, some inventors were unhappy with the state of video compression in 2001 and endeavored to create something better. So they did, and called it the VSL codec. This codec is so far undocumented on the MultimediaWiki, so it probably has yet to be seen “in the wild”. Good luck finding hard technical data on it now since searches for “VSL codec” are overwhelmed by articles about this lawsuit. Also, the original codec probably wasn’t called VSL because VSL is apparently an IP organization formed much later.

  Then, the protagonists of the lawsuit patented the codec. Then, years later, Google wanted to purchase a video codec that they could open source and use to supplant H.264.

  The complaint goes on to allege that in 2010, Google specifically contacted VSL to possibly license or acquire this mysterious VSL technology. Google was allegedly allowed to study the technology, eventually decided not to continue discussions, and shipped back the proprietary materials.

  Here’s where things get weird. When Google shipped back the materials, they allegedly shipped back a bunch of Post-It notes. The notes are alleged to contain a ton of incriminating evidence. The lawsuit claims that the notes contained such tidbits as :

  • Google was concerned that its infringement could be considered “recklessness” (the standard applicable to willful infringement) ;
  • Google personnel should “try” to destroy incriminating emails ;
  • Google should consider a “design around” because it was facing a “risk of litigation.”

  Actually, given Google’s acquisition of On2, I can totally believe that last one (On2’s codecs have famously contained a lot of weirdness which is commonly suspected to be attributable to designing around known patents).

  Anyway, a lot of this case seems to hinge on the authenticity of these Post-It notes :

  “65. The Post-It notes are unequivocal evidence of Google’s knowledge of the ’339 Patent and infringement by Defendants”

  I wish I could find a stock photo of a stack of Post-It notes in an evidence bag.

  I’ve worked at big technology companies. Big tech companies these days are very diligent about indoctrinating employees about IP liability issues. The reason this Post-It situation strikes me as odd is because the alleged contents of the notes basically outline everything the corporate lawyers tell you NOT to do.

  Analysis
  I’m trying to determine what specific algorithms and coding techniques. I guess I was expecting to see a specific claim that, “Our patent outlines this specific coding technique and here is unequivocal proof that Google A) uses the same technique, and B) specifically did so after looking at our patent.” I didn’t find that (well, a bit of part B, c.f., the Post-It note debacle), but maybe that’s not how these patent lawsuits operate. I’ve never kept up before.

  Maybe it’s just a patent troll. Maybe it’s for the stock bump. I’m expecting to see pump-n-dump stock spam featuring the stock symbol MAXD anytime now.

  I’ve never been interested in following a lawsuit case carefully before. I suddenly find myself wondering if I can subscribe to the RSS feed for this case ? Too much to hope for. But I found this item through Pando and maybe they’ll stay on top of it.