Newest 'ffmpeg' Questions - Stack Overflow
Les articles publiés sur le site
-
FFmpeg extract every frame with timestamp
7 mars, par Ruan MattHow can I extract every frame of an video with the respective timestamp?
ffmpeg -r 1 -i in.mp4 images/frame-%d.jpg
This code extracts all frames, but without timestamp. Since Windows do not allow ":" in filenames, I'll replace it for ".". I would like something like this:
frame-h.m.s.ms.random_value.jpg frame-00.10.15.17.1.jpg frame-00.11.16.04.2.jpg frame-00.11.17.11.3.jpg frame-00.11.17.22.4.jpg frame-00.12.04.01.5.jpg ...
The reason of "random id" is to allow repeated frames without replacing them.
I have tried:
ffmpeg -r 1 -i rabbit.mp4 images/frame-%{pts\:hms}.jpg
But this doesn't work! => Could not open file : images/
I have no clue how can I do that! Can you help me? Thank you.
-
Mp4 video from ffmpeg to rtsp format
6 mars, par damini chopraI want to MP4 file feed from FFmpeg to RTSP stream.
I am using below command :
ffmpeg -re -i /root/test_video.mp4 -f rtsp -muxdelay 0.1 http://x.x.x.x:8050/feed1.ffm Connection to tcp://x.x.x.x:8050?timeout=0 failed: Connection refused Could not write header for output file #0 (incorrect codec parameters ?): Connection refused Error initializing output stream 0:0 -- [aac @ 0x25b3200] Qavg: 11662.538 [aac @ 0x25b3200] 2 frames left in the queue on closing Conversion failed!
Please help?
-
Batch Convert all .MOV in a directory to mp4s
6 mars, par anjchangJust downloaded a whole bunch of videos from my phone to desktop. How to convert all the .MOV files in a directory to mp4s quickly, using ffmpeg.
-
How does FFmpeg determine the “attached pic” and “timed thumbnails” dispositions of an MP4 track ?
6 mars, par obskyrThe Issue
FFmpeg has a concept of “dispositions” – a property that describes the purpose of a stream in a media file. For example, here are the streams in a file I have lying around, with the dispositions emphasized:
Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 251 kb/s (default) Metadata: creation_time : 2021-11-10T20:14:06.000000Z handler_name : Core Media Audio vendor_id : [0][0][0][0] Stream #0:1[0x2](und): Video: mjpeg (Baseline) (jpeg / 0x6765706A), yuvj420p(pc, bt470bg/unknown/unknown), 1024x1024, 0 kb/s, 0.0006 fps, 3.08 tbr, 600 tbn (default) (attached pic) (timed thumbnails) Metadata: creation_time : 2021-11-10T20:14:06.000000Z handler_name : Core Media Video vendor_id : [0][0][0][0] Stream #0:2[0x3](und): Data: bin_data (text / 0x74786574) Metadata: creation_time : 2021-11-10T20:14:06.000000Z handler_name : Core Media Text Stream #0:3[0x0]: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/ unknown), 1024x1024 [SAR 144:144 DAR 1:1], 90k tbr, 90k tbn (attached pic)
However, if I make any modification to this file’s chapter markers using the C++ library MP4v2 (even just re-saving the existing ones:
auto f = MP4Modify("test.m4a"); MP4Chapter_t* chapterList; uint32_t chapterCount; MP4GetChapters(f, &chapterList, &chapterCount); MP4SetChapters(f, chapterList, chapterCount); MP4Close(f);
), some of these dispositions are removed:Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 251 kb/s (default) Metadata: creation_time : 2021-11-10T20:14:06.000000Z handler_name : Core Media Audio vendor_id : [0][0][0][0] Stream #0:1[0x2](und): Video: mjpeg (Baseline) (jpeg / 0x6765706A), yuvj420p(pc, bt470bg/unknown/unknown), 1024x1024, 0 kb/s, 0.0006 fps, 3.08 tbr, 600 tbn (default) ← “attached pic” and “timed thumbnails” removed! Metadata: creation_time : 2021-11-10T20:14:06.000000Z handler_name : Core Media Video vendor_id : [0][0][0][0] Stream #0:2[0x0]: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/ unknown), 1024x1024 [SAR 144:144 DAR 1:1], 90k tbr, 90k tbn (attached pic) Stream #0:3[0x4](und): Data: bin_data (text / 0x74786574) This stream was moved to the end, but that’s intended behavior. It contains chapter titles, and we just edited the chapters. Metadata: creation_time : 2025-03-05T09:56:31.000000Z
It also renders the file unplayable in MPC-HC (but not in VLC!), which is apparently a bug in MP4v2. I’m currently investigating that bug to report and potentially fix it, but that’s a separate issue – in my journey there, I’m wracking my brain trying to understand what it is that MP4v2 changes to make FFmpeg stop reporting the “attached pic” and “timed thumbnails” dispositions. I’ve explored the before-and-afters in MP4 Box, and I can’t for the life of me find which atom it is that differs in a relevant way.
(I’d love to share the files, but unfortunately the contents are under copyright – if anyone knows of a way to remove the audio from an MP4 file without changing anything else, let me know and I’ll upload dummied-out versions. Without them, I can’t really ask about the issue directly. I can at least show you the files’ respective atom trees, but I’m not sure how relevant that is.)
The Question
I thought I’d read FFmpeg’s source code to find out how it determines dispositions for MP4 streams, but of course, FFmpeg is very complex. Could someone who’s more familiar with C and/or FFmpeg’s codebase help me sleuth out how FFmpeg determines dispositions for MP4 files (in particular, “attached pic” and “timed thumbnails”)?
Some Thoughts…
- I figure searching for “attached_pic” might be a good start?
- Could the MP4 muxer
movenc.c
be helpful? - I’d imagine what we’d really like to look at is the MP4 demuxing process, as it’s during demuxing that FFmpeg determines dispositions from the data in the file. After poring over the code for hours, however, I’ve been utterly unable to find where that happens.
-
How to stream synchronized video and audio in real-time from an Android smartphone using HLS while preserving orientation metadata ?
6 mars, par Jérôme LAROSEHello, I am working on an Android application where I need to stream video from one or two cameras on my smartphone, along with audio from the microphone, in real-time via a link or web page accessible to users. The stream should be live, allow rewinding (DVR functionality), and be recorded simultaneously. A latency of 1 to 2 minutes is acceptable, and the streaming is one-way. I have chosen HLS (HTTP Live Streaming) for its browser compatibility and DVR support. However, I am encountering issues with audio-video synchronization, managing camera orientation metadata, and format conversions.
Here are my attempts:
MP4 segmentation with
MediaRecorder
- I used
MediaRecorder
withsetNextOutputFile
to generate short MP4 segments, thenffmpeg-kit
to convert them to fMP4 for HLS. - Expected: Well-aligned segments for smooth HLS playback.
- Result: Timestamp issues causing jumps or interruptions in playback.
- I used
MPEG2-TS via local socket
- I configured
MediaRecorder
to produce an MPEG2-TS stream sent via a local socket toffmpeg-kit
. - Expected: Stable streaming with preserved metadata.
- Result: Streaming works, but orientation metadata is lost, leading to incorrectly oriented video (e.g., rotated 90°).
- I configured
Orientation correction with
ffmpeg
- I tested
-vf transpose=1
inffmpeg
to correct the orientation. - Expected: Correctly oriented video without excessive latency.
- Result: Re-encoding takes too long for real-time streaming, causing unacceptable latency.
- I tested
MPEG2-TS to fMP4 conversion
- I converted the MPEG2-TS stream to fMP4 with
ffmpeg
to preserve orientation. - Expected: Perfect audio-video synchronization.
- Result: Slight desynchronization between audio and video, affecting the user experience.
- I converted the MPEG2-TS stream to fMP4 with
I am looking for a solution to:
- Stream an HLS feed from Android with correctly timestamped segments.
- Preserve orientation metadata without heavy re-encoding.
- Ensure perfect audio-video synchronization.
UPDATE
package com.example.angegardien import android.Manifest import android.content.Context import android.content.pm.PackageManager import android.graphics.SurfaceTexture import android.hardware.camera2.* import android.media.* import android.os.* import android.util.Log import android.view.Surface import android.view.TextureView import android.view.WindowManager import androidx.activity.ComponentActivity import androidx.core.app.ActivityCompat import com.arthenica.ffmpegkit.FFmpegKit import fi.iki.elonen.NanoHTTPD import kotlinx.coroutines.* import java.io.File import java.io.IOException import java.net.ServerSocket import android.view.OrientationEventListener /** * MainActivity class: * - Manages camera operations using the Camera2 API. * - Records video using MediaRecorder. * - Pipes data to FFmpeg to generate HLS segments. * - Hosts a local HLS server using NanoHTTPD to serve the generated HLS content. */ class MainActivity : ComponentActivity() { // TextureView used for displaying the camera preview. private lateinit var textureView: TextureView // Camera device instance. private lateinit var cameraDevice: CameraDevice // Camera capture session for managing capture requests. private lateinit var cameraCaptureSession: CameraCaptureSession // CameraManager to access camera devices. private lateinit var cameraManager: CameraManager // Directory where HLS output files will be stored. private lateinit var hlsDir: File // Instance of the HLS server. private lateinit var hlsServer: HlsServer // Camera id ("1" corresponds to the rear camera). private val cameraId = "1" // Flag indicating whether recording is currently active. private var isRecording = false // MediaRecorder used for capturing audio and video. private lateinit var activeRecorder: MediaRecorder // Surface for the camera preview. private lateinit var previewSurface: Surface // Surface provided by MediaRecorder for recording. private lateinit var recorderSurface: Surface // Port for the FFmpeg local socket connection. private val ffmpegPort = 8080 // Coroutine scope to manage asynchronous tasks. private val scope = CoroutineScope(Dispatchers.IO + SupervisorJob()) // Variables to track current device rotation and listen for orientation changes. private var currentRotation = 0 private lateinit var orientationListener: OrientationEventListener override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) // Initialize the TextureView and set it as the content view. textureView = TextureView(this) setContentView(textureView) // Get the CameraManager system service. cameraManager = getSystemService(CAMERA_SERVICE) as CameraManager // Setup the directory for HLS output. setupHLSDirectory() // Start the local HLS server on port 8081. hlsServer = HlsServer(8081, hlsDir, this) try { hlsServer.start() Log.d("HLS_SERVER", "HLS Server started on port 8081") } catch (e: IOException) { Log.e("HLS_SERVER", "Error starting HLS Server", e) } // Initialize the current rotation. currentRotation = getDeviceRotation() // Add a listener to detect orientation changes. orientationListener = object : OrientationEventListener(this) { override fun onOrientationChanged(orientation: Int) { if (orientation == ORIENTATION_UNKNOWN) return // Skip unknown orientations. // Determine the new rotation angle. val newRotation = when { orientation >= 315 || orientation < 45 -> 0 orientation >= 45 && orientation < 135 -> 90 orientation >= 135 && orientation < 225 -> 180 orientation >= 225 && orientation < 315 -> 270 else -> 0 } // If the rotation has changed and recording is active, update the rotation. if (newRotation != currentRotation && isRecording) { Log.d("ROTATION", "Orientation change detected: $newRotation") currentRotation = newRotation } } } orientationListener.enable() // Set up the TextureView listener to know when the surface is available. textureView.surfaceTextureListener = object : TextureView.SurfaceTextureListener { override fun onSurfaceTextureAvailable(surface: SurfaceTexture, width: Int, height: Int) { // Open the camera when the texture becomes available. openCamera() } override fun onSurfaceTextureSizeChanged(surface: SurfaceTexture, width: Int, height: Int) {} override fun onSurfaceTextureDestroyed(surface: SurfaceTexture) = false override fun onSurfaceTextureUpdated(surface: SurfaceTexture) {} } } /** * Sets up the HLS directory in the public Downloads folder. * If the directory exists, it deletes it recursively and creates a new one. */ private fun setupHLSDirectory() { val downloadsDir = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS) hlsDir = File(downloadsDir, "HLS_Output") if (hlsDir.exists()) { hlsDir.deleteRecursively() } hlsDir.mkdirs() Log.d("HLS", "📂 HLS folder created: ${hlsDir.absolutePath}") } /** * Opens the camera after checking for necessary permissions. */ private fun openCamera() { if (ActivityCompat.checkSelfPermission(this, Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED || ActivityCompat.checkSelfPermission(this, Manifest.permission.RECORD_AUDIO) != PackageManager.PERMISSION_GRANTED) { // Request permissions if they are not already granted. ActivityCompat.requestPermissions(this, arrayOf(Manifest.permission.CAMERA, Manifest.permission.RECORD_AUDIO), 101) return } try { // Open the specified camera using its cameraId. cameraManager.openCamera(cameraId, object : CameraDevice.StateCallback() { override fun onOpened(camera: CameraDevice) { cameraDevice = camera // Start the recording session once the camera is opened. startNextRecording() } override fun onDisconnected(camera: CameraDevice) { camera.close() } override fun onError(camera: CameraDevice, error: Int) { camera.close() } }, null) } catch (e: CameraAccessException) { e.printStackTrace() } } /** * Starts a new recording session: * - Sets up the preview and recorder surfaces. * - Creates a pipe for MediaRecorder output. * - Creates a capture session for simultaneous preview and recording. */ private fun startNextRecording() { // Get the SurfaceTexture from the TextureView and set its default buffer size. val texture = textureView.surfaceTexture!! texture.setDefaultBufferSize(1920, 1080) // Create the preview surface. previewSurface = Surface(texture) // Create and configure the MediaRecorder. activeRecorder = createMediaRecorder() // Create a pipe to route MediaRecorder data. val pipe = ParcelFileDescriptor.createPipe() val pfdWrite = pipe[1] // Write end used by MediaRecorder. val pfdRead = pipe[0] // Read end used by the local socket server. // Set MediaRecorder output to the file descriptor of the write end. activeRecorder.setOutputFile(pfdWrite.fileDescriptor) setupMediaRecorder(activeRecorder) // Obtain the recorder surface from MediaRecorder. recorderSurface = activeRecorder.surface // Create a capture request using the RECORD template. val captureRequestBuilder = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_RECORD) captureRequestBuilder.addTarget(previewSurface) captureRequestBuilder.addTarget(recorderSurface) // Create a capture session including both preview and recorder surfaces. cameraDevice.createCaptureSession( listOf(previewSurface, recorderSurface), object : CameraCaptureSession.StateCallback() { override fun onConfigured(session: CameraCaptureSession) { cameraCaptureSession = session captureRequestBuilder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_AUTO) // Start a continuous capture request. cameraCaptureSession.setRepeatingRequest(captureRequestBuilder.build(), null, null) // Launch a coroutine to start FFmpeg and MediaRecorder with synchronization. scope.launch { startFFmpeg() delay(500) // Wait for FFmpeg to be ready. activeRecorder.start() isRecording = true Log.d("HLS", "🎥 Recording started...") } // Launch a coroutine to run the local socket server to forward data. scope.launch { startLocalSocketServer(pfdRead) } } override fun onConfigureFailed(session: CameraCaptureSession) { Log.e("Camera2", "❌ Configuration failed") } }, null ) } /** * Coroutine to start a local socket server. * It reads from the MediaRecorder pipe and sends the data to FFmpeg. */ private suspend fun startLocalSocketServer(pfdRead: ParcelFileDescriptor) { withContext(Dispatchers.IO) { val serverSocket = ServerSocket(ffmpegPort) Log.d("HLS", "Local socket server started on port $ffmpegPort") // Accept connection from FFmpeg. val socket = serverSocket.accept() Log.d("HLS", "Connection accepted from FFmpeg") // Read data from the pipe and forward it through the socket. val inputStream = ParcelFileDescriptor.AutoCloseInputStream(pfdRead) val outputStream = socket.getOutputStream() val buffer = ByteArray(8192) var bytesRead: Int while (inputStream.read(buffer).also { bytesRead = it } != -1) { outputStream.write(buffer, 0, bytesRead) } outputStream.close() inputStream.close() socket.close() serverSocket.close() } } /** * Coroutine to start FFmpeg using a local TCP input. * Applies a video rotation filter based on device orientation and generates HLS segments. */ private suspend fun startFFmpeg() { withContext(Dispatchers.IO) { // Retrieve the appropriate transpose filter based on current rotation. val transposeFilter = getTransposeFilter(currentRotation) // FFmpeg command to read from the TCP socket and generate an HLS stream. // Two alternative commands are commented below. // val ffmpegCommand = "-fflags +genpts -i tcp://localhost:$ffmpegPort -c copy -bsf:a aac_adtstoasc -movflags +faststart -f dash -seg_duration 10 -hls_playlist 1 ${hlsDir.absolutePath}/manifest.mpd" // val ffmpegCommand = "-fflags +genpts -i tcp://localhost:$ffmpegPort -c copy -bsf:a aac_adtstoasc -movflags +faststart -f hls -hls_time 5 -hls_segment_type fmp4 -hls_flags split_by_time -hls_list_size 0 -hls_playlist_type event -hls_fmp4_init_filename init.mp4 -hls_segment_filename ${hlsDir.absolutePath}/segment_%03d.m4s ${hlsDir.absolutePath}/playlist.m3u8" val ffmpegCommand = "-fflags +genpts -i tcp://localhost:$ffmpegPort -vf $transposeFilter -c:v libx264 -preset ultrafast -crf 23 -c:a copy -movflags +faststart -f hls -hls_time 0.1 -hls_segment_type mpegts -hls_flags split_by_time -hls_list_size 0 -hls_playlist_type event -hls_segment_filename ${hlsDir.absolutePath}/segment_%03d.ts ${hlsDir.absolutePath}/playlist.m3u8" FFmpegKit.executeAsync(ffmpegCommand) { session -> if (session.returnCode.isValueSuccess) { Log.d("HLS", "✅ HLS generated successfully") } else { Log.e("FFmpeg", "❌ Error generating HLS: ${session.allLogsAsString}") } } } } /** * Gets the current device rotation using the WindowManager. */ private fun getDeviceRotation(): Int { val windowManager = getSystemService(Context.WINDOW_SERVICE) as WindowManager return when (windowManager.defaultDisplay.rotation) { Surface.ROTATION_0 -> 0 Surface.ROTATION_90 -> 90 Surface.ROTATION_180 -> 180 Surface.ROTATION_270 -> 270 else -> 0 } } /** * Returns the FFmpeg transpose filter based on the rotation angle. * Used to rotate the video stream accordingly. */ private fun getTransposeFilter(rotation: Int): String { return when (rotation) { 90 -> "transpose=1" // 90° clockwise 180 -> "transpose=2,transpose=2" // 180° rotation 270 -> "transpose=2" // 90° counter-clockwise else -> "transpose=0" // No rotation } } /** * Creates and configures a MediaRecorder instance. * Sets up audio and video sources, formats, encoders, and bitrates. */ private fun createMediaRecorder(): MediaRecorder { return MediaRecorder().apply { setAudioSource(MediaRecorder.AudioSource.MIC) setVideoSource(MediaRecorder.VideoSource.SURFACE) setOutputFormat(MediaRecorder.OutputFormat.MPEG_2_TS) setVideoEncodingBitRate(5000000) setVideoFrameRate(24) setVideoSize(1080, 720) setVideoEncoder(MediaRecorder.VideoEncoder.H264) setAudioEncoder(MediaRecorder.AudioEncoder.AAC) setAudioSamplingRate(16000) setAudioEncodingBitRate(96000) // 96 kbps } } /** * Prepares the MediaRecorder and logs the outcome. */ private fun setupMediaRecorder(recorder: MediaRecorder) { try { recorder.prepare() Log.d("HLS", "✅ MediaRecorder prepared") } catch (e: IOException) { Log.e("HLS", "❌ Error preparing MediaRecorder", e) } } /** * Custom HLS server class extending NanoHTTPD. * Serves HLS segments and playlists from the designated HLS directory. */ private inner class HlsServer(port: Int, private val hlsDir: File, private val context: Context) : NanoHTTPD(port) { override fun serve(session: IHTTPSession): Response { val uri = session.uri.trimStart('/') // Intercept the request for `init.mp4` and serve it from assets. /* if (uri == "init.mp4") { Log.d("HLS Server", "📡 Intercepting init.mp4, sending file from assets...") return try { val assetManager = context.assets val inputStream = assetManager.open("init.mp4") newFixedLengthResponse(Response.Status.OK, "video/mp4", inputStream, inputStream.available().toLong()) } catch (e: Exception) { Log.e("HLS Server", "❌ Error reading init.mp4 from assets: ${e.message}") newFixedLengthResponse(Response.Status.INTERNAL_ERROR, MIME_PLAINTEXT, "Server error") } } */ // Serve all other HLS files normally from the hlsDir. val file = File(hlsDir, uri) return if (file.exists()) { newFixedLengthResponse(Response.Status.OK, getMimeTypeForFile(uri), file.inputStream(), file.length()) } else { newFixedLengthResponse(Response.Status.NOT_FOUND, MIME_PLAINTEXT, "File not found") } } } /** * Clean up resources when the activity is destroyed. * Stops recording, releases the camera, cancels coroutines, and stops the HLS server. */ override fun onDestroy() { super.onDestroy() if (isRecording) { activeRecorder.stop() activeRecorder.release() } cameraDevice.close() scope.cancel() hlsServer.stop() orientationListener.disable() Log.d("HLS", "🛑 Activity destroyed") } }
I have three examples of ffmpeg commands.
- One command segments into DASH, but the camera does not have the correct rotation.
- One command segments into HLS without re-encoding with 5-second segments; it’s fast but does not have the correct rotation.
- One command segments into HLS with re-encoding, which applies a rotation. It’s too slow for 5-second segments, so a 1-second segment was chosen.
Note:
- In the second command ("One command segments into HLS without re-encoding with 5-second segments; it’s fast but does not have the correct rotation."), it returns fMP4. To achieve the correct rotation, I provide a preconfigured
init.mp4
file during the HTTP request to retrieve it (see comment). - In the third command ("One command segments into HLS with re-encoding, which applies a rotation. It’s too slow for 5-second segments, so a 1-second segment was chosen."), it returns TS.