Recherche avancée

Sur d’autres sites (5139)

• Screeching white sound coming while playing audio as a raw stream

  27 avril 2020, par Sri Nithya Sharabheshwarananda

  I. Background

  



  

  1. I am trying to make an application which helps to match subtitles to the audio waveform very accurately at the waveform level, at the word level or even at the character level.

  


  2. The audio is expected to be Sanskrit chants (Yoga, rituals etc.) which are extremely long compound words [ example - aṅganyā-sokta-mātaro-bījam is traditionally one word broken only to assist reading ]

  


  3. The input transcripts / subtitles might be roughly in sync at the sentence/verse level but surely would not be in sync at the word level.

  


  4. The application should be able to figure out points of silence in the audio waveform, so that it can guess the start and end points of each word (or even letter/consonant/vowel in a word), such that the audio-chanting and visual-subtitle at the word level (or even at letter/consonant/vowel level) perfectly match, and the corresponding UI just highlights or animates the exact word (or even letter) in the subtitle line which is being chanted at that moment, and also show that word (or even the letter/consonant/vowel) in bigger font. This app's purpose is to assist learning Sanskrit chanting.

  


  5. It is not expected to be a 100% automated process, nor 100% manual but a mix where the application should assist the human as much as possible.

  


  



  II. Following is the first code I wrote for this purpose, wherein

  



  

  1. First I open a mp3 (or any audio format) file,

  


  2. Seek to some arbitrary point in the timeline of the audio file // as of now playing from zero offset

  


  3. Get the audio data in raw format for 2 purposes - (1) playing it and (2) drawing the waveform.

  


  4. Playing the raw audio data using standard java audio libraries

  


  



  III. The problem I am facing is, between every cycle there is screeching sound.

  



  

  • Probably I need to close the line between cycles ? Sounds simple, I can try.

  


  • But I am also wondering if this overall approach itself is correct ? Any tip, guide, suggestion, link would be really helpful.

  


  • Also I just hard coded the sample-rate etc ( 44100Hz etc. ), are these good to set as default presets or it should depend on the input format ?

  


  



  IV. Here is the code

  



  import com.github.kokorin.jaffree.StreamType;
import com.github.kokorin.jaffree.ffmpeg.FFmpeg;
import com.github.kokorin.jaffree.ffmpeg.FFmpegProgress;
import com.github.kokorin.jaffree.ffmpeg.FFmpegResult;
import com.github.kokorin.jaffree.ffmpeg.NullOutput;
import com.github.kokorin.jaffree.ffmpeg.PipeOutput;
import com.github.kokorin.jaffree.ffmpeg.ProgressListener;
import com.github.kokorin.jaffree.ffprobe.Stream;
import com.github.kokorin.jaffree.ffmpeg.UrlInput;
import com.github.kokorin.jaffree.ffprobe.FFprobe;
import com.github.kokorin.jaffree.ffprobe.FFprobeResult;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.SourceDataLine;


public class FFMpegToRaw {
    Path BIN = Paths.get("f:\\utilities\\ffmpeg-20190413-0ad0533-win64-static\\bin");
    String VIDEO_MP4 = "f:\\org\\TEMPLE\\DeviMahatmyamRecitationAudio\\03_01_Devi Kavacham.mp3";
    FFprobe ffprobe;
    FFmpeg ffmpeg;

    public void basicCheck() throws Exception {
        if (BIN != null) {
            ffprobe = FFprobe.atPath(BIN);
        } else {
            ffprobe = FFprobe.atPath();
        }
        FFprobeResult result = ffprobe
                .setShowStreams(true)
                .setInput(VIDEO_MP4)
                .execute();

        for (Stream stream : result.getStreams()) {
            System.out.println("Stream " + stream.getIndex()
                    + " type " + stream.getCodecType()
                    + " duration " + stream.getDuration(TimeUnit.SECONDS));
        }    
        if (BIN != null) {
            ffmpeg = FFmpeg.atPath(BIN);
        } else {
            ffmpeg = FFmpeg.atPath();
        }

        //Sometimes ffprobe can't show exact duration, use ffmpeg trancoding to NULL output to get it
        final AtomicLong durationMillis = new AtomicLong();
        FFmpegResult fFmpegResult = ffmpeg
                .addInput(
                        UrlInput.fromUrl(VIDEO_MP4)
                )
                .addOutput(new NullOutput())
                .setProgressListener(new ProgressListener() {
                    @Override
                    public void onProgress(FFmpegProgress progress) {
                        durationMillis.set(progress.getTimeMillis());
                    }
                })
                .execute();
        System.out.println("audio size - "+fFmpegResult.getAudioSize());
        System.out.println("Exact duration: " + durationMillis.get() + " milliseconds");
    }

    public void toRawAndPlay() throws Exception {
        ProgressListener listener = new ProgressListener() {
            @Override
            public void onProgress(FFmpegProgress progress) {
                System.out.println(progress.getFrame());
            }
        };

        // code derived from : https://stackoverflow.com/questions/32873596/play-raw-pcm-audio-received-in-udp-packets

        int sampleRate = 44100;//24000;//Hz
        int sampleSize = 16;//Bits
        int channels   = 1;
        boolean signed = true;
        boolean bigEnd = false;
        String format  = "s16be"; //"f32le"

        //https://trac.ffmpeg.org/wiki/audio types
        final AudioFormat af = new AudioFormat(sampleRate, sampleSize, channels, signed, bigEnd);
        final DataLine.Info info = new DataLine.Info(SourceDataLine.class, af);
        final SourceDataLine line = (SourceDataLine) AudioSystem.getLine(info);

        line.open(af, 4096); // format , buffer size
        line.start();

        OutputStream destination = new OutputStream() {
            @Override public void write(int b) throws IOException {
                throw new UnsupportedOperationException("Nobody uses thi.");
            }
            @Override public void write(byte[] b, int off, int len) throws IOException {
                String o = new String(b);
                boolean showString = false;
                System.out.println("New output ("+ len
                        + ", off="+off + ") -> "+(showString?o:"")); 
                // output wave form repeatedly

                if(len%2!=0) {
                    len -= 1;
                    System.out.println("");
                }
                line.write(b, off, len);
                System.out.println("done round");
            }
        };

        // src : http://blog.wudilabs.org/entry/c3d357ed/?lang=en-US
        FFmpegResult result = FFmpeg.atPath(BIN).
            addInput(UrlInput.fromPath(Paths.get(VIDEO_MP4))).
            addOutput(PipeOutput.pumpTo(destination).
                disableStream(StreamType.VIDEO). //.addArgument("-vn")
                setFrameRate(sampleRate).            //.addArguments("-ar", sampleRate)
                addArguments("-ac", "1").
                setFormat(format)              //.addArguments("-f", format)
            ).
            setProgressListener(listener).
            execute();

        // shut down audio
        line.drain();
        line.stop();
        line.close();

        System.out.println("result = "+result.toString());
    }

    public static void main(String[] args) throws Exception {
        FFMpegToRaw raw = new FFMpegToRaw();
        raw.basicCheck();
        raw.toRawAndPlay();
    }
}



  



  Thank You

  


• JavaCPP FFMpeg to JavaSound

  8 août 2020, par TW2

  I have a problem to be able to read audio using JavaCPP FFMpeg library. I don’t know how to pass it to java sound and I don’t know too if my code is correct.

  


  Let’s see the more important part of my code (video is OK so I drop this)  :

  


  The variables  :

  


  //==========================================================================&#xA;// FFMpeg 4.x - Video and Audio&#xA;//==========================================================================&#xA;&#xA;private final AVFormatContext   pFormatCtx = new AVFormatContext(null);&#xA;private final AVDictionary      OPTIONS_DICT = null;&#xA;private AVPacket                pPacket = new AVPacket();&#xA;    &#xA;//==========================================================================&#xA;// FFMpeg 4.x - Audio&#xA;//==========================================================================&#xA;    &#xA;private AVCodec                 pAudioCodec;&#xA;private AVCodecContext          pAudioCodecCtx;&#xA;private final List<streaminfo>  audioStreams = new ArrayList&lt;>();&#xA;private int                     audio_data_size;&#xA;private final BytePointer       audio_data = new BytePointer(0);&#xA;private int                     audio_ret;&#xA;private AVFrame                 pAudioDecodedFrame = null;&#xA;private AVCodecParserContext    pAudioParser;&#xA;private SwrContext              audio_swr_ctx = null;&#xA;</streaminfo>

  


  Then I call prepare functions in this order  :

  


  private void prepareFirst() throws Exception{
    oldFile = file;
            
    // Initialize packet and check for error
    pPacket = av_packet_alloc();
    if(pPacket == null){
        throw new Exception("ALL: Couldn't allocate packet");
    }

    // Open video file
    if (avformat_open_input(pFormatCtx, file.getPath(), null, null) != 0) {
        throw new Exception("ALL: Couldn't open file");
    }

    // Retrieve stream information
    if (avformat_find_stream_info(pFormatCtx, (PointerPointer)null) < 0) {
        throw new Exception("ALL: Couldn't find stream information");
    }

    // Dump information about file onto standard error
    av_dump_format(pFormatCtx, 0, file.getPath(), 0);

    // Find the first audio/video stream
    for (int i = 0; i < pFormatCtx.nb_streams(); i++) {
        switch(pFormatCtx.streams(i).codecpar().codec_type()){
            case AVMEDIA_TYPE_VIDEO -> videoStreams.add(new StreamInfo(i, pFormatCtx.streams(i)));
            case AVMEDIA_TYPE_AUDIO -> audioStreams.add(new StreamInfo(i, pFormatCtx.streams(i)));
        }
    }
    
    if(videoStreams.isEmpty() && type != PlayType.AudioOnly){
        throw new Exception("Didn't find an audio stream");
    }
    if(audioStreams.isEmpty() && type != PlayType.VideoOnly){
        throw new Exception("Didn't find a video stream");
    }
}

private void prepareAudio() throws Exception{
    //++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    // AUDIO
    //------------------------------------------------------------------

    if(audioStreams.isEmpty() == false){
        //===========================
        //------------
        
//                // Let's search for AVCodec
//                pAudioCodec = avcodec_find_decoder(pFormatCtx.streams(audioStreams.get(0).getStreamIndex()).codecpar().codec_id());
//                if (pAudioCodec == null) {
//                    throw new Exception("AUDIO: Unsupported codec or not found!");
//                }
//
//                // Let's alloc AVCodecContext
//                pAudioCodecCtx = avcodec_alloc_context3(pAudioCodec);
//                if (pAudioCodecCtx == null) {            
//                    throw new Exception("AUDIO: Unallocated codec context or not found!");
//                }
        
        // Get a pointer to the codec context for the video stream
        pAudioCodecCtx = pFormatCtx.streams(audioStreams.get(0).getStreamIndex()).codec();

        // Find the decoder for the video stream
        pAudioCodec = avcodec_find_decoder(pAudioCodecCtx.codec_id());
        if (pAudioCodec == null) {
            throw new Exception("AUDIO: Unsupported codec or not found!");
        }

        //===========================
        //------------

        /* open it */
        if (avcodec_open2(pAudioCodecCtx, pAudioCodec, OPTIONS_DICT) < 0) {
            throw new Exception("AUDIO: Could not open codec");
        }

        pAudioDecodedFrame = av_frame_alloc();
        if (pAudioDecodedFrame == null){
            throw new Exception("AUDIO: DecodedFrame allocation failed");
        }

        audio_swr_ctx = swr_alloc_set_opts(
                null,                           // existing Swr context or NULL
                AV_CH_LAYOUT_STEREO,            // output channel layout (AV_CH_LAYOUT_*)
                AV_SAMPLE_FMT_S16,              // output sample format (AV_SAMPLE_FMT_*).
                44100,                          // output sample rate (frequency in Hz)
                pAudioCodecCtx.channels(),  // input channel layout (AV_CH_LAYOUT_*)
                pAudioCodecCtx.sample_fmt(),    // input sample format (AV_SAMPLE_FMT_*).
                pAudioCodecCtx.sample_rate(),   // input sample rate (frequency in Hz)
                0,                              // logging level offset
                null                            // parent logging context, can be NULL
        );
        
        swr_init(audio_swr_ctx);
        
        av_samples_fill_arrays(
                pAudioDecodedFrame.data(),      // audio_data,
                pAudioDecodedFrame.linesize(),  // linesize
                audio_data,                     // buf
                (int)AV_CH_LAYOUT_STEREO,       // nb_channels
                44100,                          // nb_samples
                AV_SAMPLE_FMT_S16,              // sample_fmt
                0                               // align
        );
        
    }
    
    // Audio treatment end ---------------------------------------------
    //==================================================================
}


  


  And then when I launch the thread  :

  


  private void doPlay() throws Exception{
    av_init_packet(pPacket);

    // Read frames
    while (av_read_frame(pFormatCtx, pPacket) >= 0) {
        if (type != PlayType.AudioOnly && pPacket.stream_index() == videoStreams.get(0).getStreamIndex()) {
            // Is this a packet from the video stream?
            decodeVideo();
            renewPacket();
        }

        if (type != PlayType.VideoOnly && pPacket.stream_index() == audioStreams.get(0).getStreamIndex()) {
            // Is this a packet from the audio stream?
            if(pPacket.size() > 0){
                decodeAudio();
                renewPacket();
            }
        }
    }
}

private void renewPacket(){
    // Free the packet that was allocated by av_read_frame
    av_packet_unref(pPacket);

    pPacket.data(null);
    pPacket.size(0);
    av_init_packet(pPacket);
}


  


  And again, this is where I don’t read audio  :

  


  private void decodeAudio() throws Exception{

    do {
        audio_ret = avcodec_send_packet(pAudioCodecCtx, pPacket);
    } while(audio_ret == AVERROR_EAGAIN());
    System.out.println("packet sent return value: " + audio_ret);

    if(audio_ret == AVERROR_EOF || audio_ret == AVERROR_EINVAL()) {
        StringBuilder sb = new StringBuilder();
        Formatter formatter = new Formatter(sb, Locale.US);
        formatter.format("AVERROR(EAGAIN): %d, AVERROR_EOF: %d, AVERROR(EINVAL): %d\n", AVERROR_EAGAIN(), AVERROR_EOF, AVERROR_EINVAL());
        formatter.format("Audio frame getting error (%d)!\n", audio_ret);
        throw new Exception(sb.toString());
    }

    audio_ret = avcodec_receive_frame(pAudioCodecCtx, pAudioDecodedFrame);
    System.out.println("frame received return value: " + audio_ret);

    audio_data_size = av_get_bytes_per_sample(AV_SAMPLE_FMT_S16);

    if (audio_data_size < 0) {
        /* This should not occur, checking just for paranoia */
        throw new Exception("Failed to calculate data size");
    }
    
    double frame_nb = 44100d / pAudioCodecCtx.sample_rate() * pAudioDecodedFrame.nb_samples();
    long out_count = Math.round(Math.floor(frame_nb));

    int out_samples = swr_convert(
            audio_swr_ctx,
            audio_data, 
            (int)out_count,
            pAudioDecodedFrame.data(0),
            pAudioDecodedFrame.nb_samples()
    );
    
    if (out_samples < 0) {
        throw new Exception("AUDIO: Error while converting");
    }
    
    int dst_bufsize = av_samples_get_buffer_size(
        pAudioDecodedFrame.linesize(), 
        (int)AV_CH_LAYOUT_STEREO,  
        out_samples,
        AV_SAMPLE_FMT_S16,
        1
    );
    
    AudioFormat audioFormat = new AudioFormat(
            pAudioDecodedFrame.sample_rate(),
            16,
            2, 
            true, 
            false
    );
    
    BytePointer bytePointer = pAudioDecodedFrame.data(0);
    ByteBuffer byteBuffer = bytePointer.asBuffer();

    byte[] bytes = new byte[byteBuffer.remaining()];
    byteBuffer.get(bytes);
    
    try (SourceDataLine sdl = AudioSystem.getSourceDataLine(audioFormat)) {
        sdl.open(audioFormat);                
        sdl.start();
        sdl.write(bytes, 0, bytes.length);
        sdl.drain();
        sdl.stop();
    } catch (LineUnavailableException ex) {
        Logger.getLogger(AVEntry.class.getName()).log(Level.SEVERE, null, ex);
    }    
}


  


  Do you have an idea  ?

  


• Revision 29188 : 2 options de plus pour personnaliser la page d’activation de la mutu : * ...

  15 juin 2009, par real3t@… — Log

  2 options de plus pour personnaliser la page d’activation de la mutu :
  * ’branding’ : texte libre en HTML
  * ’branding_logo’ => logo (sous forme de HTML)