Recherche avancée

Sur d’autres sites (5532)

• Using FFmpeg to stitch together H.264 videos and variably-spaced JPEG pictures ; dealing with ffmpeg warnings

  19 octobre 2022, par LB2

  Context

  


  I have a process flow that may output either H.264 Annex B streams, variably-spaced JPEGs, or a mixture of two. By variably-spaced I mean where elapsed time between any two adjacent JPEGs may (and likely to be) different from any other two adjacent JPEGs. So an example of possible inputs are :

  


  

  1. stream1.h264

  


  2. {Set of JPEGs}

  


  3. stream1.h264 + stream2.h264

  


  4. stream1.h264 + {Set of JPEGs}

  


  5. stream1.h264 + {Set of JPEGs} + stream2.h264

  


  6. stream1.h264 + {Set of JPEGs} + stream2.h264 + {Set of JPEGs} + ...

  


  7. stream1.h264 + stream2.h264 + {Set of JPEGs} + ...

  


  


  The output needs to be a single stitched (i.e. concatenated) output in MPEG-4 container.

  


  Requirements : No re-encoding or transcoding of existing video compression (One time conversion of JPEG sets to video format is OKay).

  


  Solution Prototype

  


  To prototype the solution I have found that ffmpeg has concat demuxer that would let me specify an ordered sequence of inputs that ffmpeg would then concatenate together, but all inputs must be of the same format. So, to meet that requirement, I :

  


  

  1. Convert every JPEG set to an .mp4 using concat (and using delay # directive to specify time-spacing between each JPEG)

  


  2. Convert every .h264 to .mp4 using -c copy to avoid transcoding.

  


  3. Stitch all generated interim .mp4 files into the single final .mp4 using -f concat and -c copy.

  


  


  Here's the bash script, in parts, that performs the above :

  


  


  1. Ignore the curl comment ; that's from originally generating a 100 jpeg images with numbers and these are simply saved locally. What the loop does is it generates concat input file with file sequence#.jpeg directives and duration # directive where each successive JPEG delay is incremented by 0.1 seconds (0.1 between first and second, 0.2 b/w 2nd and 3rd, 0.3 b/w 3rd and 4th, and so on). Then it runs ffmpeg command to convert the set of JPEGs to .mp4 interim file.

     


     echo "ffconcat version 1.0" >ffconcat-jpeg.txt
echo >>ffconcat-jpeg.txt

for i in {1..100}
do
    echo "file $i.jpeg" >>ffconcat-jpeg.txt
    d=$(echo "$i" | awk '{printf "%f", $1 / 10}')
    # d=$(echo "scale=2; $i/10" | bc)
    echo "duration $d" >>ffconcat-jpeg.txt
    echo "" >>ffconcat-jpeg.txt
    # curl -o "$i.jpeg" "https://math.tools/equation/get_equaimages?equation=$i&fontsize=256"
done

ffmpeg \
    -hide_banner \
    -vsync vfr \
    -f concat \
    -i ffconcat-jpeg.txt \
    -r 30 \
    -video_track_timescale 90000 \
    video-jpeg.mp4


     


  


  2. Convert two streams from .h264 to .mp4 via copy (no transcoding).

     


     ffmpeg \
    -hide_banner \
    -i low-motion-video.h264 \
    -c copy \
    -vsync vfr \
    -video_track_timescale 90000 \
    low-motion-video.mp4

ffmpeg \
    -hide_banner \
    -i full-video.h264 \
    -c copy \
    -video_track_timescale 90000 \
    -vsync vfr \
    full-video.mp4


     


  


  3. Stitch all together by generating another concat directive file.

     


     echo "ffconcat version 1.0" >ffconcat-h264.txt
echo >>ffconcat-h264.txt
echo "file low-motion-video.mp4" >>ffconcat-h264.txt
echo >>ffconcat-h264.txt
echo "file full-video.mp4" >>ffconcat-h264.txt
echo >>ffconcat-h264.txt
echo "file video-jpeg.mp4" >>ffconcat-h264.txt
echo >>ffconcat-h264.txt

ffmpeg \
    -hide_banner \
    -f concat \
    -i ffconcat-h264.txt \
    -pix_fmt yuv420p \
    -c copy \
    -video_track_timescale 90000 \
    -vsync vfr \
    video-out.mp4



     


  


  


  Problem (and attempted troubleshooting)

  


  The above does produce a reasonable output — it plays first video, then plays second video with no timing/rate issues AFAICT, then plays JPEGs with time between each JPEG "frame" growing successively, as expected.

  


  But, the conversion process produces warnings that concern me (for compatibility with players ; or potentially other IRL streams that may result in some issue my prototyping content doesn't make obvious). Initial attempts generated 100s of warnings, but with some arguments added, I reduced it down to just a handful, but this handful is stubborn and nothing I tried would help.

  


  The first conversion of JPEGs to .mp4 goes fine with the following output :

  


  Input #0, concat, from 'ffconcat-jpeg.txt':
  Duration: 00:08:25.00, start: 0.000000, bitrate: 0 kb/s
  Stream #0:0: Video: png, pal8(pc), 176x341 [SAR 3780:3780 DAR 16:31], 25 fps, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (png (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x7fe418008e00] using SAR=1/1
[libx264 @ 0x7fe418008e00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2
[libx264 @ 0x7fe418008e00] profile High 4:4:4 Predictive, level 1.3, 4:4:4, 8-bit
[libx264 @ 0x7fe418008e00] 264 - core 163 r3060 5db6aa6 - H.264/MPEG-4 AVC codec - Copyleft 2003-2021 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=11 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'video-jpeg.mp4':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv444p(tv, progressive), 176x341 [SAR 1:1 DAR 16:31], q=2-31, 30 fps, 90k tbn
    Metadata:
      encoder         : Lavc58.134.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame=  100 fps=0.0 q=-1.0 Lsize=     157kB time=00:07:55.33 bitrate=   2.7kbits/s speed=2.41e+03x    
video:155kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.800846%
[libx264 @ 0x7fe418008e00] frame I:1     Avg QP:20.88  size:   574
[libx264 @ 0x7fe418008e00] frame P:43    Avg QP:14.96  size:  2005
[libx264 @ 0x7fe418008e00] frame B:56    Avg QP:21.45  size:  1266
[libx264 @ 0x7fe418008e00] consecutive B-frames: 14.0% 24.0% 30.0% 32.0%
[libx264 @ 0x7fe418008e00] mb I  I16..4: 36.4% 55.8%  7.9%
[libx264 @ 0x7fe418008e00] mb P  I16..4:  5.1%  7.5% 11.2%  P16..4:  5.6%  8.1%  4.5%  0.0%  0.0%    skip:57.9%
[libx264 @ 0x7fe418008e00] mb B  I16..4:  2.4%  0.9%  3.9%  B16..8: 16.2%  8.8%  4.6%  direct: 1.2%  skip:62.0%  L0:56.6% L1:38.7% BI: 4.7%
[libx264 @ 0x7fe418008e00] 8x8 transform intra:28.3% inter:3.7%
[libx264 @ 0x7fe418008e00] coded y,u,v intra: 26.5% 0.0% 0.0% inter: 9.8% 0.0% 0.0%
[libx264 @ 0x7fe418008e00] i16 v,h,dc,p: 82% 13%  4%  0%
[libx264 @ 0x7fe418008e00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20%  8% 71%  1%  0%  0%  0%  0%  0%
[libx264 @ 0x7fe418008e00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 41% 11% 29%  4%  2%  3%  1%  7%  1%
[libx264 @ 0x7fe418008e00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fe418008e00] ref P L0: 44.1%  4.2% 28.4% 23.3%
[libx264 @ 0x7fe418008e00] ref B L0: 56.2% 32.1% 11.6%
[libx264 @ 0x7fe418008e00] ref B L1: 92.4%  7.6%
[libx264 @ 0x7fe418008e00] kb/s:2.50


  


  The conversion of individual streams from .h264 to .mp4 generates two types of warnings each. One is [mp4 @ 0x7faee3040400] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly, and the other is [mp4 @ 0x7faee3040400] pts has no value.

  


  Some posts on SO (can't find my original finds on that now) suggested that it's safe to ignore and comes from H.264 being an elementary stream that supposedly doesn't contain timestamps. It surprises me a bit since I produce that stream using NVENC API and clearly supply timing information for each frame via PIC_PARAMS structure : NV_STRUCT(PIC_PARAMS, pp); ...; pp.inputTimeStamp = _frameIndex++ * (H264_CLOCK_RATE / _params.frameRate);, where #define H264_CLOCK_RATE 9000 and _params.frameRate = 30.

  


  Input #0, h264, from 'low-motion-video.h264':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 1440x3040 [SAR 1:1 DAR 9:19], 30 fps, 30 tbr, 1200k tbn, 60 tbc
Output #0, mp4, to 'low-motion-video.mp4':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1440x3040 [SAR 1:1 DAR 9:19], q=2-31, 30 fps, 30 tbr, 90k tbn, 1200k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0x7faee3040400] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[mp4 @ 0x7faee3040400] pts has no value
[mp4 @ 0x7faee3040400] pts has no value0kB time=-00:00:00.03 bitrate=N/A speed=N/A    
    Last message repeated 17985 times
frame=17987 fps=0.0 q=-1.0 Lsize=   79332kB time=00:09:59.50 bitrate=1084.0kbits/s speed=1.59e+03x    
video:79250kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.103804%
Input #0, h264, from 'full-video.h264':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 1440x3040 [SAR 1:1 DAR 9:19], 30 fps, 30 tbr, 1200k tbn, 60 tbc
Output #0, mp4, to 'full-video.mp4':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1440x3040 [SAR 1:1 DAR 9:19], q=2-31, 30 fps, 30 tbr, 90k tbn, 1200k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0x7f9381864600] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[mp4 @ 0x7f9381864600] pts has no value
[mp4 @ 0x7f9381864600] pts has no value0kB time=-00:00:00.03 bitrate=N/A speed=N/A    
    Last message repeated 17981 times
frame=17983 fps=0.0 q=-1.0 Lsize=   52976kB time=00:09:59.36 bitrate= 724.1kbits/s speed=1.33e+03x    
video:52893kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.156232%


  


  But the most worrisome error for me is from stitching together all interim .mp4 files into one :

  


  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f9ff2010e00] Auto-inserting h264_mp4toannexb bitstream filter
Input #0, concat, from 'ffconcat-h264.txt':
  Duration: N/A, bitrate: 1082 kb/s
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1440x3040 [SAR 1:1 DAR 9:19], 1082 kb/s, 30 fps, 30 tbr, 90k tbn, 60 tbc
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
Output #0, mp4, to 'video-out.mp4':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1440x3040 [SAR 1:1 DAR 9:19], q=2-31, 1082 kb/s, 30 fps, 30 tbr, 90k tbn, 90k tbc
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f9fe1009c00] Auto-inserting h264_mp4toannexb bitstream filter
[mp4 @ 0x7f9ff2023400] Non-monotonous DTS in output stream 0:0; previous: 53954460, current: 53954460; changing to 53954461. This may result in incorrect timestamps in the output file.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f9fd1008a00] Auto-inserting h264_mp4toannexb bitstream filter
[mp4 @ 0x7f9ff2023400] Non-monotonous DTS in output stream 0:0; previous: 107900521, current: 107874150; changing to 107900522. This may result in incorrect timestamps in the output file.
[mp4 @ 0x7f9ff2023400] Non-monotonous DTS in output stream 0:0; previous: 107900522, current: 107886150; changing to 107900523. This may result in incorrect timestamps in the output file.
frame=36070 fps=0.0 q=-1.0 Lsize=  132464kB time=00:27:54.26 bitrate= 648.1kbits/s speed=6.54e+03x    
video:132296kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.126409%


  


  I'm not sure how to deal with those non-monotonous DTS errors, and no matter what I try, nothing budges. I analyzed the interim .mp4 files using ffprobe -show_frames and found that the last frame of each interim .mp4 does not have DTS, while previous frames do. E.g. :

  


  ...
[FRAME]
media_type=video
stream_index=0
key_frame=0
pkt_pts=53942461
pkt_pts_time=599.360678
pkt_dts=53942461
pkt_dts_time=599.360678
best_effort_timestamp=53942461
best_effort_timestamp_time=599.360678
pkt_duration=3600
pkt_duration_time=0.040000
pkt_pos=54161377
pkt_size=1034
width=1440
height=3040
pix_fmt=yuv420p
sample_aspect_ratio=1:1
pict_type=B
coded_picture_number=17982
display_picture_number=0
interlaced_frame=0
top_field_first=0
repeat_pict=0
color_range=unknown
color_space=unknown
color_primaries=unknown
color_transfer=unknown
chroma_location=left
[/FRAME]
[FRAME]
media_type=video
stream_index=0
key_frame=0
pkt_pts=53927461
pkt_pts_time=599.194011
pkt_dts=N/A
pkt_dts_time=N/A
best_effort_timestamp=53927461
...


  


  My guess is that as concat demuxer reads in (or elsewhere in ffmpeg's conversion pipeline), for the last frame it sees no DTS set, and produces a virtual value equal to the last seen. Then further in pipeline it consumes this input, sees that DTS value is being repeated, issues a warning and offsets it with increment by one, which might be somewhat nonsensical/unrealistic timing value.

  


  I tried using -fflags +genpts as suggested in this SO answer, but that doesn't change anything.

  


  Per yet other posts suggesting issue being with incompatible tbn and tbc values and possible timebase issues, I tried adding -time_base 1:90000 and -enc_time_base 1:90000 and -copytb 1 and nothing budges. The -video_track_timescale 90000 is there b/c it helped reduce those DTS warnings from 100s down to 3, but doesn't eliminate them all.

  


  Question

  


  What is missing and how can I get ffmpeg to perform conversions without these warnings, to be sure it produces proper, well-formed output ?

  


• ffmpeg encoding leaves me with blank space at the end where the video pauses and there is nothing ahead

  6 novembre 2022, par Nisarg Desai

  I was trying to slice some of the video being played and clip it in mpv.net using a .lua script which uses ffmpeg to encode the webm output. ffmpeg sometimes leaves some seconds blank and without any video/audio ahead while clipping from source. Is there any solution to this ?

  


  The code for the script is given below (was taken from here https://github.com/occivink/mpv-scripts) :

  


  local utils = require "mp.utils"
local msg = require "mp.msg"
local options = require "mp.options"

local ON_WINDOWS = (package.config:sub(1,1) ~= "/")

local start_timestamp = nil
local profile_start = ""

-- implementation detail of the osd message
local timer = nil
local timer_duration = 2

-- folder creation if it doesnt exist
function exists(file)
   local ok, err, code = os.rename(file, file)
   if not ok then
      if code == 13 then
         return true
      end
   end
   return ok, err
end

--- Check if a directory exists in this path
function create_dir(path)
    local dir = "\"" .. path .. "\""
    if not exists(path .."/") then
        os.execute("mkdir " .. dir)
    end
end

function append_table(lhs, rhs)
    for i = 1,#rhs do
        lhs[#lhs+1] = rhs[i]
    end
    return lhs
end

function file_exists(name)
    local f = io.open(name, "r")
    if f ~= nil then
        io.close(f)
        return true
    else
        return false
    end
end

function get_extension(path)
    local candidate = string.match(path, "%.([^.]+)$")
    if candidate then
        for _, ext in ipairs({ "mkv", "webm", "mp4", "avi" }) do
            if candidate == ext then
                return candidate
            end
        end
    end
    return "mkv"
end

function get_output_string(dir, format, input, extension, title, from, to, profile)
    local res = utils.readdir(dir)
    if not res then
        return nil
    end
    local files = {}
    for _, f in ipairs(res) do
        files[f] = true
    end
    local output = format
    output = string.gsub(output, "$f", function() return input end)
    output = string.gsub(output, "$t", function() return title end)
    output = string.gsub(output, "$s", function() return seconds_to_time_string(from, true) end)
    output = string.gsub(output, "$e", function() return seconds_to_time_string(to, true) end)
    output = string.gsub(output, "$d", function() return seconds_to_time_string(to-from, true) end)
    output = string.gsub(output, "$x", function() return extension end)
    output = string.gsub(output, "$p", function() return profile end)
    if ON_WINDOWS then
        output = string.gsub(output, "[/\\|<>?:\"*]", "_")
    end
    if not string.find(output, "$n") then
        return files[output] and nil or output
    end
    local i = 1
    while true do
        local potential_name = string.gsub(output, "$n", tostring(i))
        if not files[potential_name] then
            return potential_name
        end
        i = i + 1
    end
end

function get_video_filters()
    local filters = {}
    for _, vf in ipairs(mp.get_property_native("vf")) do
        local name = vf["name"]
        name = string.gsub(name, '^lavfi%-', '')
        local filter
        if name == "crop" then
            local p = vf["params"]
            filter = string.format("crop=%d:%d:%d:%d", p.w, p.h, p.x, p.y)
        elseif name == "mirror" then
            filter = "hflip"
        elseif name == "flip" then
            filter = "vflip"
        elseif name == "rotate" then
            local rotation = vf["params"]["angle"]
            -- rotate is NOT the filter we want here
            if rotation == "90" then
                filter = "transpose=clock"
            elseif rotation == "180" then
                filter = "transpose=clock,transpose=clock"
            elseif rotation == "270" then
                filter = "transpose=cclock"
            end
        end
        filters[#filters + 1] = filter
    end
    return filters
end

function get_input_info(default_path, only_active)
    local accepted = {
        video = true,
        audio = not mp.get_property_bool("mute"),
        sub = mp.get_property_bool("sub-visibility")
    }
    local ret = {}
    for _, track in ipairs(mp.get_property_native("track-list")) do
        local track_path = track["external-filename"] or default_path
        if not only_active or (track["selected"] and accepted[track["type"]]) then
            local tracks = ret[track_path]
            if not tracks then
                ret[track_path] = { track["ff-index"] }
            else
                tracks[#tracks + 1] = track["ff-index"]
            end
        end
    end
    return ret
end

function seconds_to_time_string(seconds, full)
    local ret = string.format("%02d:%02d.%03d"
        , math.floor(seconds / 60) % 60
        , math.floor(seconds) % 60
        , seconds * 1000 % 1000
    )
    if full or seconds > 3600 then
        ret = string.format("%d:%s", math.floor(seconds / 3600), ret)
    end
    return ret
end

function start_encoding(from, to, settings)
    local args = {
        settings.ffmpeg_command,
        "-loglevel", "panic", "-hide_banner",
    }
    local append_args = function(table) args = append_table(args, table) end

    local path = mp.get_property("path")
    local is_stream = not file_exists(path)
    if is_stream then
        path = mp.get_property("stream-path")
    end

    local track_args = {}
    local start = seconds_to_time_string(from, false)
    local input_index = 0
    for input_path, tracks in pairs(get_input_info(path, settings.only_active_tracks)) do
       append_args({
            "-ss", start,
            "-i", input_path,
        })
        if settings.only_active_tracks then
            for _, track_index in ipairs(tracks) do
                track_args = append_table(track_args, { "-map", string.format("%d:%d", input_index, track_index)})
            end
        else
            track_args = append_table(track_args, { "-map", tostring(input_index)})
        end
        input_index = input_index + 1
    end

    append_args({"-to", tostring(to-from)})
    append_args(track_args)

    -- apply some of the video filters currently in the chain
    local filters = {}
    if settings.preserve_filters then
        filters = get_video_filters()
    end
    if settings.append_filter ~= "" then
        filters[#filters + 1] = settings.append_filter
    end
    if #filters > 0 then
        append_args({ "-filter:v", table.concat(filters, ",") })
    end

    -- split the user-passed settings on whitespace
    for token in string.gmatch(settings.codec, "[^%s]+") do
        args[#args + 1] = token
    end

    -- path of the output
    local output_directory = mp.get_property("options/screenshot-directory")
    -- local checkbool = exists(output_directory.."/")
    -- mp.osd_message("" .. type(checkbool), timer_duration)
    -- if not checkbool then    
    --     os.execute("mkdir" .. output_directory)
    -- end
    if output_directory == "" then
        if is_stream then
            output_directory = "."
        else
            output_directory, _ = utils.split_path(path)
        end
    else
        output_directory = string.gsub(output_directory, "^~", os.getenv("HOME") or "~")
    end
    local input_name = mp.get_property("filename/no-ext") or "encode"
    local title = mp.get_property("media-title")
    local extension = get_extension(path)
    local output_name = get_output_string(output_directory, settings.output_format, input_name, extension, title, from, to, settings.profile)
    if not output_name then
        mp.osd_message("Invalid path " .. output_directory)
        return
    end
    args[#args + 1] = utils.join_path(output_directory, output_name)

    if settings.print then
        local o = ""
        -- fuck this is ugly
        for i = 1, #args do
            local fmt = ""
            if i == 1 then
                fmt = "%s%s"
            elseif i >= 2 and i <= 4 then
                fmt = "%s"
            elseif args[i-1] == "-i" or i == #args or args[i-1] == "-filter:v" then
                fmt = "%s '%s'"
            else
                fmt = "%s %s"
            end
            o = string.format(fmt, o, args[i])
        end
        print(o)
    end
    if settings.detached then
        utils.subprocess_detached({ args = args })
    else
        local res = utils.subprocess({ args = args, max_size = 0, cancellable = false })
        if res.status == 0 then
            mp.osd_message("Finished encoding succesfully")
        else
            mp.osd_message("Failed to encode, check the log")
        end
    end
end

function clear_timestamp()
    timer:kill()
    start_timestamp = nil
    profile_start = ""
    mp.remove_key_binding("encode-ESC")
    mp.remove_key_binding("encode-ENTER")
    mp.osd_message("", 0)
end

function set_timestamp(profile)
    if not mp.get_property("path") then
        mp.osd_message("No file currently playing")
        return
    end
    if not mp.get_property_bool("seekable") then
        mp.osd_message("Cannot encode non-seekable media")
        return
    end
    create_dir(mp.get_property("options/screenshot-directory"))
    if not start_timestamp or profile ~= profile_start then
        profile_start = profile
        start_timestamp = mp.get_property_number("time-pos")
        local msg = function()
            mp.osd_message(
                string.format("encode [%s]: waiting for end timestamp", profile or "default"),
                timer_duration
            )
        end
        msg()
        timer = mp.add_periodic_timer(timer_duration, msg)
        mp.add_forced_key_binding("ESC", "encode-ESC", clear_timestamp)
        mp.add_forced_key_binding("ENTER", "encode-ENTER", function() set_timestamp(profile) end)
    else
        local from = start_timestamp
        local to = mp.get_property_number("time-pos")
        if to <= from then
            mp.osd_message("Second timestamp cannot be before the first", timer_duration)
            timer:kill()
            timer:resume()
            return
        end
        clear_timestamp()
        mp.osd_message(string.format("Encoding from %s to %s"
            , seconds_to_time_string(from, false)
            , seconds_to_time_string(to, false)
        ), timer_duration)
        -- include the current frame into the extract
        local fps = mp.get_property_number("container-fps") or 30
        to = to + 1 / fps / 2
        local settings = {
            detached = false,
            container = "",
            only_active_tracks = false,
            preserve_filters = true,
            append_filter = "",
            codec = "-c:v libvpx-vp9 -lossless 1 -b:v 1000k -deadline good",
            output_format = "$f_$n.webm",
            output_directory = "",
            ffmpeg_command = "ffmpeg",
            print = true,
        }
        if profile then
            options.read_options(settings, profile)
            if settings.container ~= "" then
                msg.warn("The 'container' setting is deprecated, use 'output_format' now")
                settings.output_format = settings.output_format .. "." .. settings.container
            end
            settings.profile = profile
        else
            settings.profile = "default"
        end        
        start_encoding(from, to, settings)
    end
end

mp.add_key_binding(nil, "set-timestamp", set_timestamp)


  


• Saving frames as JPG with FFMPEG (Visual Studio / C++)

  10 novembre 2022, par Diego Satizabal

  I am trying to save all frames from a mp4 video in separate JPG files, I have a code that runs and actually saves something to JPG files but files are not recognized as images and nothing is showing.

  


  Below my full code, I am using Visual Studio 2022 in Windows 11 and FFMPEG 5.1. The function that saves the images is save_frame_as_jpeg which is actually an adaption from the code provided here but changing the use of avcodec_encode_video2 for avcodec_send_frame/avcodec_receive_packet as indicated in the documentation.

  


  I am obiously doing something wrong but cannot quite find it, BTW, I know that a simple command (ffmpeg -i input.mp4 -vf fps=1 vid_%d.png) will do this but I am requiring to do it by code.

  


  Any help is appreciated, thanks in advance !

  


      // FfmpegTests.cpp : This file contains the &#x27;main&#x27; function. Program execution begins and ends there.&#xA;//&#xA;#pragma warning(disable : 4996)&#xA;extern "C"&#xA;{&#xA;    #include "libavformat/avformat.h"&#xA;    #include "libavcodec/avcodec.h"&#xA;    #include "libavfilter/avfilter.h"&#xA;    #include "libavutil/opt.h"&#xA;    #include "libavutil/avutil.h"&#xA;    #include "libavutil/error.h"&#xA;    #include "libavfilter/buffersrc.h"&#xA;    #include "libavfilter/buffersink.h"&#xA;    #include "libswscale/swscale.h"&#xA;}&#xA;&#xA;#pragma comment(lib, "avcodec.lib")&#xA;#pragma comment(lib, "avformat.lib")&#xA;#pragma comment(lib, "avfilter.lib")&#xA;#pragma comment(lib, "avutil.lib")&#xA;#pragma comment(lib, "swscale.lib")&#xA;&#xA;#include <cstdio>&#xA;#include <iostream>&#xA;#include <chrono>&#xA;#include <thread>&#xA;&#xA;&#xA;static AVFormatContext* fmt_ctx;&#xA;static AVCodecContext* dec_ctx;&#xA;AVFilterGraph* filter_graph;&#xA;AVFilterContext* buffersrc_ctx;&#xA;AVFilterContext* buffersink_ctx;&#xA;static int video_stream_index = -1;&#xA;&#xA;const char* filter_descr = "scale=78:24,transpose=cclock";&#xA;static int64_t last_pts = AV_NOPTS_VALUE;&#xA;&#xA;static int open_input_file(const char* filename)&#xA;{&#xA;    const AVCodec* dec;&#xA;    int ret;&#xA;&#xA;    if ((ret = avformat_open_input(&amp;fmt_ctx, filename, NULL, NULL)) &lt; 0) {&#xA;        av_log(NULL, AV_LOG_ERROR, "Cannot open input file\n");&#xA;        return ret;&#xA;    }&#xA;&#xA;    if ((ret = avformat_find_stream_info(fmt_ctx, NULL)) &lt; 0) {&#xA;        av_log(NULL, AV_LOG_ERROR, "Cannot find stream information\n");&#xA;        return ret;&#xA;    }&#xA;&#xA;    /* select the video stream */&#xA;    ret = av_find_best_stream(fmt_ctx, AVMEDIA_TYPE_VIDEO, -1, -1, &amp;dec, 0);&#xA;    if (ret &lt; 0) {&#xA;        av_log(NULL, AV_LOG_ERROR, "Cannot find a video stream in the input file\n");&#xA;        return ret;&#xA;    }&#xA;    video_stream_index = ret;&#xA;&#xA;    /* create decoding context */&#xA;    dec_ctx = avcodec_alloc_context3(dec);&#xA;    if (!dec_ctx)&#xA;        return AVERROR(ENOMEM);&#xA;    avcodec_parameters_to_context(dec_ctx, fmt_ctx->streams[video_stream_index]->codecpar);&#xA;&#xA;    /* init the video decoder */&#xA;    if ((ret = avcodec_open2(dec_ctx, dec, NULL)) &lt; 0) {&#xA;        av_log(NULL, AV_LOG_ERROR, "Cannot open video decoder\n");&#xA;        return ret;&#xA;    }&#xA;&#xA;    return 0;&#xA;}&#xA;&#xA;static int init_filters(const char* filters_descr)&#xA;{&#xA;    char args[512];&#xA;    int ret = 0;&#xA;    const AVFilter* buffersrc = avfilter_get_by_name("buffer");&#xA;    const AVFilter* buffersink = avfilter_get_by_name("buffersink");&#xA;    AVFilterInOut* outputs = avfilter_inout_alloc();&#xA;    AVFilterInOut* inputs = avfilter_inout_alloc();&#xA;    AVRational time_base = fmt_ctx->streams[video_stream_index]->time_base;&#xA;    enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_GRAY8, AV_PIX_FMT_NONE };&#xA;&#xA;    filter_graph = avfilter_graph_alloc();&#xA;    if (!outputs || !inputs || !filter_graph) {&#xA;        ret = AVERROR(ENOMEM);&#xA;        goto end;&#xA;    }&#xA;&#xA;    /* buffer video source: the decoded frames from the decoder will be inserted here. */&#xA;    snprintf(args, sizeof(args),&#xA;        "video_size=%dx%d:pix_fmt=%d:time_base=%d/%d:pixel_aspect=%d/%d",&#xA;        dec_ctx->width, dec_ctx->height, dec_ctx->pix_fmt,&#xA;        time_base.num, time_base.den,&#xA;        dec_ctx->sample_aspect_ratio.num, dec_ctx->sample_aspect_ratio.den);&#xA;&#xA;    ret = avfilter_graph_create_filter(&amp;buffersrc_ctx, buffersrc, "in",&#xA;        args, NULL, filter_graph);&#xA;    if (ret &lt; 0) {&#xA;        av_log(NULL, AV_LOG_ERROR, "Cannot create buffer source\n");&#xA;        goto end;&#xA;    }&#xA;&#xA;    /* buffer video sink: to terminate the filter chain. */&#xA;    ret = avfilter_graph_create_filter(&amp;buffersink_ctx, buffersink, "out",&#xA;        NULL, NULL, filter_graph);&#xA;    if (ret &lt; 0) {&#xA;        av_log(NULL, AV_LOG_ERROR, "Cannot create buffer sink\n");&#xA;        goto end;&#xA;    }&#xA;&#xA;    ret = av_opt_set_int_list(buffersink_ctx, "pix_fmts", pix_fmts, AV_PIX_FMT_NONE, AV_OPT_SEARCH_CHILDREN);&#xA;    if (ret &lt; 0) {&#xA;        av_log(NULL, AV_LOG_ERROR, "Cannot set output pixel format\n");&#xA;        goto end;&#xA;    }&#xA;&#xA;    outputs->name = av_strdup("in");&#xA;    outputs->filter_ctx = buffersrc_ctx;&#xA;    outputs->pad_idx = 0;&#xA;    outputs->next = NULL;&#xA;&#xA;    inputs->name = av_strdup("out");&#xA;    inputs->filter_ctx = buffersink_ctx;&#xA;    inputs->pad_idx = 0;&#xA;    inputs->next = NULL;&#xA;&#xA;    if ((ret = avfilter_graph_parse_ptr(filter_graph, filters_descr,&#xA;        &amp;inputs, &amp;outputs, NULL)) &lt; 0)&#xA;        goto end;&#xA;&#xA;    if ((ret = avfilter_graph_config(filter_graph, NULL)) &lt; 0)&#xA;        goto end;&#xA;&#xA;end:&#xA;    avfilter_inout_free(&amp;inputs);&#xA;    avfilter_inout_free(&amp;outputs);&#xA;&#xA;    return ret;&#xA;}&#xA;&#xA;static void display_frame(const AVFrame* frame, AVRational time_base)&#xA;{&#xA;    int x, y;&#xA;    uint8_t* p0, * p;&#xA;    int64_t delay;&#xA;&#xA;    if (frame->pts != AV_NOPTS_VALUE) {&#xA;        if (last_pts != AV_NOPTS_VALUE) {&#xA;            /* sleep roughly the right amount of time;&#xA;             * usleep is in microseconds, just like AV_TIME_BASE. */&#xA;            AVRational timeBaseQ;&#xA;            timeBaseQ.num = 1;&#xA;            timeBaseQ.den = AV_TIME_BASE;&#xA;&#xA;            delay = av_rescale_q(frame->pts - last_pts, time_base, timeBaseQ);&#xA;            if (delay > 0 &amp;&amp; delay &lt; 1000000)&#xA;                std::this_thread::sleep_for(std::chrono::microseconds(delay));&#xA;        }&#xA;        last_pts = frame->pts;&#xA;    }&#xA;&#xA;    /* Trivial ASCII grayscale display. */&#xA;    p0 = frame->data[0];&#xA;    puts("\033c");&#xA;    for (y = 0; y &lt; frame->height; y&#x2B;&#x2B;) {&#xA;        p = p0;&#xA;        for (x = 0; x &lt; frame->width; x&#x2B;&#x2B;)&#xA;            putchar(" .-&#x2B;#"[*(p&#x2B;&#x2B;) / 52]);&#xA;        putchar(&#x27;\n&#x27;);&#xA;        p0 &#x2B;= frame->linesize[0];&#xA;    }&#xA;    fflush(stdout);&#xA;}&#xA;&#xA;int save_frame_as_jpeg(AVCodecContext* pCodecCtx, AVFrame* pFrame, int FrameNo) {&#xA;    int ret = 0;&#xA;&#xA;    const AVCodec* jpegCodec = avcodec_find_encoder(AV_CODEC_ID_JPEG2000);&#xA;    if (!jpegCodec) {&#xA;        return -1;&#xA;    }&#xA;    AVCodecContext* jpegContext = avcodec_alloc_context3(jpegCodec);&#xA;    if (!jpegContext) {&#xA;        return -1;&#xA;    }&#xA;&#xA;    jpegContext->pix_fmt = pCodecCtx->pix_fmt;&#xA;    jpegContext->height = pFrame->height;&#xA;    jpegContext->width = pFrame->width;&#xA;    jpegContext->time_base = AVRational{ 1,10 };&#xA;&#xA;    ret = avcodec_open2(jpegContext, jpegCodec, NULL);&#xA;    if (ret &lt; 0) {&#xA;        return ret;&#xA;    }&#xA;    FILE* JPEGFile;&#xA;    char JPEGFName[256];&#xA;&#xA;    AVPacket packet;&#xA;    packet.data = NULL;&#xA;    packet.size = 0;&#xA;    av_init_packet(&amp;packet);&#xA;&#xA;    int gotFrame;&#xA;&#xA;    ret = avcodec_send_frame(jpegContext, pFrame);&#xA;    if (ret &lt; 0) {&#xA;        return ret;&#xA;    }&#xA;&#xA;    ret = avcodec_receive_packet(jpegContext, &amp;packet);&#xA;    if (ret &lt; 0) {&#xA;        return ret;&#xA;    }&#xA;&#xA;    sprintf(JPEGFName, "c:\\folder\\dvr-%06d.jpg", FrameNo);&#xA;    JPEGFile = fopen(JPEGFName, "wb");&#xA;    fwrite(packet.data, 1, packet.size, JPEGFile);&#xA;    fclose(JPEGFile);&#xA;&#xA;    av_packet_unref(&amp;packet);&#xA;    avcodec_close(jpegContext);&#xA;    return 0;&#xA;}&#xA;&#xA;int main(int argc, char** argv)&#xA;{&#xA;    AVFrame* frame;&#xA;    AVFrame* filt_frame;&#xA;    AVPacket* packet;&#xA;    int ret;&#xA;&#xA;    if (argc != 2) {&#xA;        fprintf(stderr, "Usage: %s file\n", argv[0]);&#xA;        exit(1);&#xA;    }&#xA;&#xA;    frame = av_frame_alloc();&#xA;    filt_frame = av_frame_alloc();&#xA;    packet = av_packet_alloc();&#xA;&#xA;    if (!frame || !filt_frame || !packet) {&#xA;        fprintf(stderr, "Could not allocate frame or packet\n");&#xA;        exit(1);&#xA;    }&#xA;&#xA;    if ((ret = open_input_file(argv[1])) &lt; 0)&#xA;        goto end;&#xA;    if ((ret = init_filters(filter_descr)) &lt; 0)&#xA;        goto end;&#xA;&#xA;    while (true)&#xA;    {&#xA;        if ((ret = av_read_frame(fmt_ctx, packet)) &lt; 0)&#xA;            break;&#xA;&#xA;        if (packet->stream_index == video_stream_index) {&#xA;            ret = avcodec_send_packet(dec_ctx, packet);&#xA;            if (ret &lt; 0) {&#xA;                av_log(NULL, AV_LOG_ERROR, "Error while sending a packet to the decoder\n");&#xA;                break;&#xA;            }&#xA;&#xA;            while (ret >= 0)&#xA;            {&#xA;                ret = avcodec_receive_frame(dec_ctx, frame);&#xA;                if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF) {&#xA;                    break;&#xA;                }&#xA;                else if (ret &lt; 0) {&#xA;                    av_log(NULL, AV_LOG_ERROR, "Error while receiving a frame from the decoder\n");&#xA;                    goto end;&#xA;                }&#xA;&#xA;                frame->pts = frame->best_effort_timestamp;&#xA;&#xA;                /* push the decoded frame into the filtergraph */&#xA;                if (av_buffersrc_add_frame_flags(buffersrc_ctx, frame, AV_BUFFERSRC_FLAG_KEEP_REF) &lt; 0) {&#xA;                    av_log(NULL, AV_LOG_ERROR, "Error while feeding the filtergraph\n");&#xA;                    break;&#xA;                }&#xA;&#xA;                /* pull filtered frames from the filtergraph */&#xA;                while (1) {&#xA;                    ret = av_buffersink_get_frame(buffersink_ctx, filt_frame);&#xA;                    if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)&#xA;                        break;&#xA;                    if (ret &lt; 0)&#xA;                        goto end;&#xA;                    display_frame(filt_frame, buffersink_ctx->inputs[0]->time_base);&#xA;                    av_frame_unref(filt_frame);&#xA;                    &#xA;                    ret = save_frame_as_jpeg(dec_ctx, frame, dec_ctx->frame_number);&#xA;                    if (ret &lt; 0)&#xA;                        goto end;&#xA;                }&#xA;                av_frame_unref(frame);&#xA;            }&#xA;        }&#xA;        av_packet_unref(packet);&#xA;    }&#xA;&#xA;end:&#xA;    avfilter_graph_free(&amp;filter_graph);&#xA;    avcodec_free_context(&amp;dec_ctx);&#xA;    avformat_close_input(&amp;fmt_ctx);&#xA;    av_frame_free(&amp;frame);&#xA;    av_frame_free(&amp;filt_frame);&#xA;    av_packet_free(&amp;packet);&#xA;&#xA;    if (ret &lt; 0 &amp;&amp; ret != AVERROR_EOF) {&#xA;        char errBuf[AV_ERROR_MAX_STRING_SIZE]{0};&#xA;        int res = av_strerror(ret, errBuf, AV_ERROR_MAX_STRING_SIZE);&#xA;        fprintf(stderr, "Error:  %s\n", errBuf);&#xA;        exit(1);&#xA;    }&#xA;&#xA;    exit(0);&#xA;}&#xA;</thread></chrono></iostream></cstdio>