Recherche avancée

Sur d’autres sites (4458)

• Emscripten and Web Audio API

  29 avril 2015, par Multimedia Mike — HTML5

  Ha ! They said it couldn’t be done ! Well, to be fair, I said it couldn’t be done. Or maybe that I just didn’t have any plans to do it. But I did it– I used Emscripten to cross-compile a CPU-intensive C/C++ codebase (Game Music Emu) to JavaScript. Then I leveraged the Web Audio API to output audio and visualize the audio using an HTML5 canvas.

  Want to see it in action ? Here’s a demonstration. Perhaps I will be able to expand the reach of my Game Music site when I can drop the odd Native Client plugin. This JS-based player works great on Chrome, Firefox, and Safari across desktop operating systems.

  But this endeavor was not without its challenges.

  Programmatically Generating Audio
  First, I needed to figure out the proper method for procedurally generating audio and making it available to output. Generally, there are 2 approaches for audio output :

  1. Sit in a loop and generate audio, writing it out via a blocking audio call
  2. Implement a callback that the audio system can invoke in order to generate more audio when needed

  Option #1 is not a good idea for an event-driven language like JavaScript. So I hunted through the rather flexible Web Audio API for a method that allowed something like approach #2. Callbacks are everywhere, after all.

  I eventually found what I was looking for with the ScriptProcessorNode. It seems to be intended to apply post-processing effects to audio streams. A program registers a callback which is passed configurable chunks of audio for processing. I subverted this by simply overwriting the input buffers with the audio generated by the Emscripten-compiled library.

  The ScriptProcessorNode interface is fairly well documented and works across multiple browsers. However, it is already marked as deprecated :

  Note : As of the August 29 2014 Web Audio API spec publication, this feature has been marked as deprecated, and is soon to be replaced by Audio Workers.

  Despite being marked as deprecated for 8 months as of this writing, there exists no appreciable amount of documentation for the successor API, these so-called Audio Workers.

  Vive la web standards !

  Visualize This
  The next problem was visualization. The Web Audio API provides the AnalyzerNode API for accessing both time and frequency domain data from a running audio stream (and fetching the data as both unsigned bytes or floating-point numbers, depending on what the application needs). This is a pretty neat idea. I just wish I could make the API work. The simple demos I could find worked well enough. But when I wired up a prototype to fetch and visualize the time-domain wave, all I got were center-point samples (an array of values that were all 128).

  Even if the API did work, I’m not sure if it would have been that useful. Per my reading of the AnalyserNode API, it only returns data as a single channel. Why would I want that ? My application supports audio with 2 channels. I want 2 channels of data for visualization.

  How To Synchronize
  So I rolled my own visualization solution by maintaining a circular buffer of audio when samples were being generated. Then, requestAnimationFrame() provided the rendering callbacks. The next problem was audio-visual sync. But that certainly is not unique to this situation– maintaining proper A/V sync is a perennial puzzle in real-time multimedia programming. I was able to glean enough timing information from the environment to achieve reasonable A/V sync (verify for yourself).

  Pause/Resume
  The next problem I encountered with the Web Audio API was pause/resume facilities, or the lack thereof. For all its bells and whistles, the API’s omission of such facilities seems most unusual, as if the design philosophy was, “Once the user starts playing audio, they will never, ever have cause to pause the audio.”

  Then again, I must understand that mine is not a use case that the design committee considered and I’m subverting the API in ways the designers didn’t intend. Typical use cases for this API seem to include such workloads as :

  • Downloading, decoding, and playing back a compressed audio stream via the network, applying effects, and visualizing the result
  • Accessing microphone input, applying effects, visualizing, encoding and sending the data across the network
  • Firing sound effects in a gaming application
  • MIDI playback via JavaScript (this honestly amazes me)

  What they did not seem to have in mind was what I am trying to do– synthesize audio in real time.

  I implemented pause/resume in a sub-par manner : pausing has the effect of generating 0 values when the ScriptProcessorNode callback is invoked, while also canceling any animation callbacks. Thus, audio output is technically still occurring, it’s just that the audio is pure silence. It’s not a great solution because CPU is still being used.

  Future Work
  I have a lot more player libraries to port to this new system. But I think I have a good framework set up.

• Is it possible to merge video files with FFmpeg interlaced not concatenated ?

  11 juin 2019, par TudorT

  I am trying to create a video file that has the frames from 2 source mp4 video files interlaced like this :

  I1 I2 P1 P2 I1 I2 P1 P2 I1 I2 P1 P2 ...

  where I = Intra, P = predicted, and the numbers are from which source file they come.

  Is this possible with FFmpeg commands ? If not, if I use the ffplay code from read_thread, how do I put the content returned from av_read_frame into a proper video file ?

  Thanks

• Why are no dts pts written to my mp4 container

  14 mai 2019, par Kiamur

  Based on my (self-answered) question here Muxing AVPackets into mp4 file - revisited, I have to ask, what could be the reason why there are no values written for pts/dts in the resulting mp4 container.

  I examined the container file with the tool MediaInfo.
  I observe that only the very first Frame contains a value for pts in the container. After that, pts is not even shown in the mp4 file anymore, but dts is, with a value of all zeros.

  This is the output from MediaInfo for the first 3 frames :

  0000A2   slice_layer_without_partitioning (IDR) - 0 (0x0) - Frame 0 - slice_type I - frame_num 0 - DTS 00:00:00.000 - PTS 00:00:00.017 (141867 bytes)
  
  0000A2    Header (5 bytes)
  
  0000A2     zero_byte:                          0 (0x00)
  
  0000A3     start_code_prefix_one_3bytes:       1 (0x000001)
  
  0000A6     nal_ref_idc:                        3 (0x3) - (2 bits)
  
  0000A6     nal_unit_type:                      5 (0x05) - (5 bits)
  
  0000A7    slice_header (3 bytes)
  
  0000A7     first_mb_in_slice:                  0 (0x0)
  
  0000A7     slice_type:                         7 (0x07) - I
  
  0000A8     pic_parameter_set_id:               0 (0x0)
  
  0000A8     frame_num:                          0 (0x0)
  
  0000A8     idr_pic_id:                         0 (0x0)
  
  0000A8     no_output_of_prior_pics_flag:       No
  
  0000A8     long_term_reference_flag:           No
  
  0000A9     slice_qp_delta:                     -5 (0xFFFFFFFB)
  
  0000AA     disable_deblocking_filter_idc:      0 (0x0)
  
  0000AA     slice_alpha_c0_offset_div2:         0 (0x0)
  
  0000AA     slice_beta_offset_div2:             0 (0x0)
  
  0000AA    slice_data (141856 bytes)
  
  0000AA     (ToDo):                             (Data)
  
  022ACD   slice_layer_without_partitioning (IDR) - 0 (0x0) - Frame 0 - slice_type I - frame_num 0 - DTS 00:00:00.000 - PTS 00:00:00.017 - first_mb_in_slice 8040 (2248 bytes)
  
  022ACD    Header (5 bytes)
  
  022ACD     zero_byte:                          0 (0x00)
  
  022ACE     start_code_prefix_one_3bytes:       1 (0x000001)
  
  022AD1     nal_ref_idc:                        3 (0x3) - (2 bits)
  
  022AD1     nal_unit_type:                      5 (0x05) - (5 bits)
  
  022AD2    slice_header (6 bytes)
  
  022AD2     first_mb_in_slice:                  8040 (0x001F68)
  
  022AD5     slice_type:                         7 (0x07) - I
  
  022AD6     pic_parameter_set_id:               0 (0x0)
  
  022AD6     frame_num:                          0 (0x0)
  
  022AD6     idr_pic_id:                         0 (0x0)
  
  022AD6     no_output_of_prior_pics_flag:       No
  
  022AD6     long_term_reference_flag:           No
  
  022AD7     slice_qp_delta:                     -5 (0xFFFFFFFB)
  
  022AD8     disable_deblocking_filter_idc:      0 (0x0)
  
  022AD8     slice_alpha_c0_offset_div2:         0 (0x0)
  
  022AD8     slice_beta_offset_div2:             0 (0x0)
  
  022AD8    slice_data (2237 bytes)
  
  022AD8     (ToDo):                             (Data)
  
  023395  1 (36212 bytes)
  
  023395   slice_layer_without_partitioning (non-IDR) - 2 (0x2) - Frame 1 - slice_type P - frame_num 1 - DTS 00:00:00.000 (36017 bytes)
  
  023395    Header (5 bytes)
  
  023395     zero_byte:                          0 (0x00)
  
  023396     start_code_prefix_one_3bytes:       1 (0x000001)
  
  023399     nal_ref_idc:                        3 (0x3) - (2 bits)
  
  023399     nal_unit_type:                      1 (0x01) - (5 bits)
  
  02339A    slice_header (3 bytes)
  
  02339A     first_mb_in_slice:                  0 (0x0)
  
  02339A     slice_type:                         5 (0x5) - P
  
  02339A     pic_parameter_set_id:               0 (0x0)
  
  02339A     frame_num:                          1 (0x1)
  
  02339B     num_ref_idx_active_override_flag (0 bytes)
  
  02339B      num_ref_idx_active_override_flag:  Yes
  
  02339B      num_ref_idx_l0_active_minus1:      0 (0x0)
  
  02339B     ref_pic_list_modification_flag_l0:  No
  
  02339B     adaptive_ref_pic_marking_mode_flag: No
  
  02339C     cabac_init_idc:                     0 (0x0)
  
  02339C     slice_qp_delta:                     -3 (0xFFFFFFFD)
  
  02339C     disable_deblocking_filter_idc:      0 (0x0)
  
  02339C     slice_alpha_c0_offset_div2:         0 (0x0)
  
  02339D     slice_beta_offset_div2:             0 (0x0)
  
  02339D    slice_data (36012 bytes)
  
  02339D     (ToDo):                             (Data)
  
  02C046   slice_layer_without_partitioning (non-IDR) - 2 (0x2) - Frame 1 - slice_type P - frame_num 1 - DTS 00:00:00.000 - first_mb_in_slice 8040 (195 bytes)
  
  02C046    Header (5 bytes)
  
  02C046     zero_byte:                          0 (0x00)
  
  02C047     start_code_prefix_one_3bytes:       1 (0x000001)
  
  02C04A     nal_ref_idc:                        3 (0x3) - (2 bits)
  
  02C04A     nal_unit_type:                      1 (0x01) - (5 bits)
  
  02C04B    slice_header (6 bytes)
  
  02C04B     first_mb_in_slice:                  8040 (0x001F68)
  
  02C04E     slice_type:                         5 (0x5) - P
  
  02C04E     pic_parameter_set_id:               0 (0x0)
  
  02C04E     frame_num:                          1 (0x1)
  
  02C04F     num_ref_idx_active_override_flag (0 bytes)
  
  02C04F      num_ref_idx_active_override_flag:  Yes
  
  02C04F      num_ref_idx_l0_active_minus1:      0 (0x0)
  
  02C04F     ref_pic_list_modification_flag_l0:  No
  
  02C04F     adaptive_ref_pic_marking_mode_flag: No
  
  02C050     cabac_init_idc:                     0 (0x0)
  
  02C050     slice_qp_delta:                     -3 (0xFFFFFFFD)
  
  02C050     disable_deblocking_filter_idc:      0 (0x0)
  
  02C050     slice_alpha_c0_offset_div2:         0 (0x0)
  
  02C051     slice_beta_offset_div2:             0 (0x0)
  
  02C051    slice_data (190 bytes)
  
  02C051     (ToDo):                             (Data)
  
  02C109  1 (26280 bytes)
  
  02C109   slice_layer_without_partitioning (non-IDR) - 4 (0x4) - Frame 2 - slice_type P - frame_num 2 - DTS 00:00:00.000 (26157 bytes)
  
  02C109    Header (5 bytes)
  
  02C109     zero_byte:                          0 (0x00)
  
  02C10A     start_code_prefix_one_3bytes:       1 (0x000001)
  
  02C10D     nal_ref_idc:                        3 (0x3) - (2 bits)
  
  02C10D     nal_unit_type:                      1 (0x01) - (5 bits)
  
  02C10E    slice_header (3 bytes)
  
  02C10E     first_mb_in_slice:                  0 (0x0)
  
  02C10E     slice_type:                         5 (0x5) - P
  
  02C10E     pic_parameter_set_id:               0 (0x0)
  
  02C10E     frame_num:                          2 (0x2)
  
  02C10F     num_ref_idx_active_override_flag (0 bytes)
  
  02C10F      num_ref_idx_active_override_flag:  Yes
  
  02C10F      num_ref_idx_l0_active_minus1:      0 (0x0)
  
  02C10F     ref_pic_list_modification_flag_l0:  No
  
  02C10F     adaptive_ref_pic_marking_mode_flag: No
  
  02C110     cabac_init_idc:                     0 (0x0)
  
  02C110     slice_qp_delta:                     -2 (0xFFFFFFFE)
  
  02C110     disable_deblocking_filter_idc:      0 (0x0)
  
  02C110     slice_alpha_c0_offset_div2:         0 (0x0)
  
  02C111     slice_beta_offset_div2:             0 (0x0)
  
  02C111    slice_data (26152 bytes)
  
  02C111     (ToDo):                             (Data)
  
  032736   slice_layer_without_partitioning (non-IDR) - 4 (0x4) - Frame 2 - slice_type P - frame_num 2 - DTS 00:00:00.000 - first_mb_in_slice 8040 (123 bytes)
  
  032736    Header (5 bytes)
  
  032736     zero_byte:                          0 (0x00)
  
  032737     start_code_prefix_one_3bytes:       1 (0x000001)
  
  03273A     nal_ref_idc:                        3 (0x3) - (2 bits)
  
  03273A     nal_unit_type:                      1 (0x01) - (5 bits)
  
  03273B    slice_header (6 bytes)
  
  03273B     first_mb_in_slice:                  8040 (0x001F68)
  
  03273E     slice_type:                         5 (0x5) - P
  
  03273E     pic_parameter_set_id:               0 (0x0)
  
  03273E     frame_num:                          2 (0x2)
  
  03273F     num_ref_idx_active_override_flag (0 bytes)
  
  03273F      num_ref_idx_active_override_flag:  Yes
  
  03273F      num_ref_idx_l0_active_minus1:      0 (0x0)
  
  03273F     ref_pic_list_modification_flag_l0:  No
  
  03273F     adaptive_ref_pic_marking_mode_flag: No
  
  032740     cabac_init_idc:                     0 (0x0)
  
  032740     slice_qp_delta:                     -2 (0xFFFFFFFE)
  
  032740     disable_deblocking_filter_idc:      0 (0x0)
  
  032740     slice_alpha_c0_offset_div2:         0 (0x0)
  
  032741     slice_beta_offset_div2:             0 (0x0)
  
  032741    slice_data (118 bytes)
  
  032741     (ToDo):                             (Data)
  
  0327B1  1 (21125 bytes)

  It goes on like that, even though I set pts and dts. The settings may not be correct already (I do some calculations like (1 / framerate) * FrameNumber), but I would expect at least some numbers in pts and dts, when I set the according fields in the avPacket structure and write that via av_interleaved_write_frame(outFmtCtx, &avPacket) ; to the file.

  What could be wrong here ?

  Edit :

  (please see below in the comments the download to my testdata and source file)
  One thing that bugs me is the fact, if I compare the output of MediaInfo from my file and that of the muxing.c generated is, that in the header, the muxing.c generated already mentions the duration of the file as 9960 ms, whereas mine is only 40 ms.

  muxing.c also does call avformat_write_header before even one frame is drawn. Yes, I suppose that the header will be updated, when the either av_interleaved_write_frame or av_write_trailer is called, but I totally not understand the mechanics behind it.
  Maybe somebody can enlighten me with some background information of any kind.

  Additionally, I think it could be necessarry to extract some SPS and PPS from my raw data (preceding the I-slice), and give that as extra data to the avformat_write_header call. But I just cannot figure out myself if I have to do that at all and if so, how to do it.