FFMPEG / libx264: How to specify a variable frame rate but with a maximum?

Question

Instead of providing a fixed frame rate to FFMPEG/libx264 (-r/-framerate), I would like to specify a variable frame rate with a MAXIMUM value, and allow libx264 to down the frame rate as it sees fit. The idea here is to get extra compression when there is something like an extended still frame (which happens A LOT in my source videos).

I realize that a predictive or bidirectional MPEG frame will compress really well, but it's also possible that the source frame rate is smaller than the one I intend to transcode to (possibly resulting in a BIGGER stream!).

Answer 1

Frustrated that you hadn't found an answer either, I was going to at least answer other people's questions about how to enable VFR (not VBR) output from FFMPEG.

The answer to that is the oddly named -vsync option. You can set it to a few different options, but the one you want is '2' or vfr. From the man page:

-vsync parameter
Video sync method. For compatibility reasons old values can be specified as numbers. Newly added values will have to be specified as strings always.

• 0, passthrough

  • Each frame is passed with its timestamp from the demuxer to the muxer.

• 1, cfr

  • Frames will be duplicated and dropped to achieve exactly the requested constant frame rate.

• 2, vfr

  • Frames are passed through with their timestamp or dropped so as to prevent 2 frames from having the same timestamp.

• drop

  • As passthrough but destroys all timestamps, making the muxer generate fresh timestamps based on frame-rate.

• -1, auto

  • Chooses between 1 and 2 depending on muxer capabilities. This is the default method.

Note that the timestamps may be further modified by the muxer, after this. For example, in the case that the format option avoid_negative_ts is enabled.

With -map you can select from which stream the timestamps should be taken. You can leave either video or audio unchanged and sync the remaining stream(s) to the unchanged one.

However, I don't quite have enough reputation to post a comment to just answer that 'sub-question' that everyone seems to be having. But I did have a few ideas that I wasn't honestly very optimistic about... But the first one I tried actually worked. So.

You simply need to combine the -vsync 2 option with the -r $maxfps option, of course where you replace $maxfps with the maximum framerate you want! And it WORKS! It doesn't duplicate frames from a source file, but it will drop frames that cause the file to go over the maximum framerate!

By default it seems that -r $maxfps by itself just causes it to duplicate/drop frames to achieve a constant framerate, and -vsync 2 by itself causes it to pull the frames in directly without really affecting the PTS values.

I wasn't optimistic about this because I already knew that -r $maxfps puts it at a constant framerate. I honestly expected an error or for it to just obey whichever came first or last or whatever. The fact that it does exactly what I wanted makes me quite pleased with the FFMPEG developers.

I hope this helps you, or someone else later on if you no longer need to know this.

Answer 2

I would like to specify a variable frame rate with a MAXIMUM value, and allow libx264 to down the frame rate as it sees fit. The idea here is to get extra compression when there is something like an extended still frame

In my understanding, this may be possible in a comparably clumsy way, but is undesirable for some complex and counterintuitive reasons

Even though an x264 stream has a framerate(s), frame rate is more a container-level problem than a codec one.

In a passthrough VFR encode, there will be what is essentially a text file detailing what the frame rate is over what frames/times, and in encoding a source, a function like tcfile-in or tcfile-out pass the timestamps through to the encode, to map the rate locations and keep the video subjectively consistent from source.

The low-framerate idea is a logical one, but doesn't work out for several reasons. Though x264 is VFR-aware with some capabilities, I don't think there is an analysis function that will vary the framerate with regard to motion in order to lower the file size (in a way analagous to the many bitrate controls).

The source is also a problem: VFR sources will by default retain their frame variability, but apparently encoding a CFR file at variable bitrate (a good idea sometimes, esp. when telecine is needed) will simply produce the same CFR.

This means you would probably have to re-write bitrate by hand (i.e. timestamps of slow scenes muxed into the file), or resort to a frame decimation algorith like dup, dedup, and exactDedup for avisynth. If your video does have extremely low motion, some frames (even half?) would be thrown out. The problem is that these algorithms are not advanced, and don't make good choices with "real life" footage as to what will contribute to the best encode.

Also, removing frames that contain things like I and B frames reduces the amount of detail available over time, which causes motion to look "steppy" and can interfere with the other basic video parameters and cause artifacts like aliasing.

And because of the way the quantizers work, x264 will actually decrease the bitrate disproportionately further in these scenes of low motion. Unless you have a slideshow of identical images, there will be motion (if only grain and other artifacts) and there will be a loss in quality that would not be seen without drastic changes to the bitrate.

And finally, the reason there aren't many options to do what you want is that x264 is really good at managing bitrate just using temporal compression (recording changes in partial frames). Going to 1/2 framerate will not cut the file size in half; 10% is probably a realistic gain to expect from low motion or animation.

So in short, dropping the bitrate of your static scenes will do very little for your file size, but will add a host of quality and sync issues, not to mention incompatibility with video editing software.

If you do want to try a decimator, you might be able to limit the maximum new frame rate by using the levels options, each of which species a maximum resolution and framerate. Unfortunately, you would probably have to work at very low resolutions to get the kind of frame rates you want, using profiles. It comes back to editing the rates by hand, either entirely or to correct frame rates you think are too high. Either way, it will take juggling to keep the sound in sync with the new framerates if alterations are made after the encoding process when the tcfile is conserved.

The takeaway is that spending time optimizing the many bitrate settings will yield much more in the way of file size management, and improve the quality of your video, rather than cause complications for little gain. Preserving the original FPS is probably the best idea unless you're aiming for broadcast or media standards. Players are well-designed to handle varying bitrates (unlike NLEs)—and the more frames in your video, the smoother the playback and perhaps the smaller the file size, due to smaller changes in motion between frames.

Here's a collection of links to standards info and forum discussions that should help with this confusing aspect of encoding:

-avisynth decimation tools

-fps and -r switches
-x264 General (tcfile, fps)
-timecode file standards
-Levels and profiles
-Short, clear CFR/VFR setting summary ("framerate" section)

doom9, videohelp, &c theoretical discussions
1 2 3 4 5 6 7

Answer 3

Perfect solution: Reduce repeated very similar frames and save output at (peak) variable frame rate

• For content with long still scenes dropping duplicate frames can reduce the framerate dynamically and significantly low to achieve more file size reduction than the codec and its intra-frame compression can ever achieve!
• In my example:
  • 100% original mezzanine file size at 60 FPS
  • 15% CR28 at 60 FPS constant frame rate
  • 6% CR28 at 60 FPS peak frame rate (almost a factor of 3x !)

Original available

ffmpeg -i screen-recording.mov -movflags faststart -c:v libx264 -vf mpdecimate -vsync vfr -r 120 -preset veryslow -crf 24 screen-recording-vfr.mp4

Transcode already compressed video to dynamic frame rate without re-encoding

ffmpeg -i video-export-old.mp4 -vf mpdecimate -vsync vfr video-export-mpdecimated-without-reencoding.mp4

In detail

Original available

ffmpeg -i screen-recording.mov -movflags faststart -c:v libx264 -vf mpdecimate -vsync vfr -r 120 -preset veryslow -crf 24 "screen-recording-vfr.mp4"

-i video-mezzanine.mov   Original or your high quality rendered export as the input file.
-movflags faststart      Streaming ready by putting the moov atom to the file start.
-c:v libx264             H-264 codec
-vf mpdecimate           Drop frames not differing greatly from previous frame to reduce frame rate.
-vsync vfr               Output as variable frame rate (vfr)! Necessary sister option to 'mpdecimate'.
                         - To maintain the playback speed while benefitting from the file size reduction.
-r 30                    If you specify -r then in this combo it serves as the peak framerate!
                         I recommend to omit this option:
                         - Then the peak framerate IS the peak/constant rate of the source.
                         - Which preserves dynamic scenes fully and compresses long still sequences.
                         - So the best of both aspects.
                         - State a peak FPS if loosing FPS in dynamic scenes
                           is acceptable for the reduction in file size.
-crf 24                  - Constant Rate Factor (constant quality at variable bitrate)
-preset veryslow         - Quality effort put into the encoding
screen-recording-vfr.mp4 - Output file

Convert already exported video to dynamic frame rate without re-encoding

ffmpeg -i video-export-old.mp4 -vf mpdecimate -vsync vfr video-export-mpdecimated.mp4

• This is a lossless operation. No lossy re-encoding happens, only repacking/re-referencing.

• It will drop as many duplicate frames as it can!

Example:

• Slideshow of only 3 still images with no transitions * each lasting 5 secs * at 50 FPS = 750 frames

• ffmpeg will reduce it indeed to just 3 frames at 1/5 (=0.2) FPS! Confirmed with mediainfo!

Analysis & Learnings

• Videos with long still image scenes can get their frame amount and file size significantly reduced!
• For both scenarios
  • Encoding from the mezzanine file at the same CFR maintains visual quality
    • But file size reduction from constant to variable framerate is 15% to 6%.
  • Framerate reduction without re-encoding brings less of a file size reduction, but still:
    • Brings filesize reduction (15% to 10%)

screen-recording.mov
- Frame rate mode: Variable
- Frame rate: 58.628 FPS
- Minimum frame rate: 30.000 FPS
- Maximum frame rate: 60.000 FPS
- Frame count: 732                100 %
- Size: 1675084 bytes             100 %

screen-recording CR24 fps 60.mp4
- Frame rate mode: Constant
- Frame rate: 60.000 FPS
- Frame count: 750                102 %
- Size: 255863 bytes               15 %

screen-recording CR24 mpdecimated vfr.mp4
- Frame rate mode: Variable
- Frame rate: 17.398 FPS
- Minimum frame rate: 1.132 FPS
- Maximum frame rate: 60.000 FPS
- Frame count: 214                 29 %
- Size: 101860 bytes                6 %


screen-recording CR24 mpdecimated fps 30 max.mp4
- Frame rate mode: Variable
- Frame rate: 11.078 FPS
- Minimum frame rate: 1.154 FPS
- Maximum frame rate: 30.000 FPS
- Frame count: 137                 17 %
- Size: 94382 bytes                 5.6

screen-recording CR24 mpdecimated vfr without re-encoding faststart.mp4
- Frame rate mode: Variable
- Frame rate: 19.974 FPS
- Minimum frame rate: 1.132 FPS
- Maximum frame rate: 60.000 FPS  
- Frame count: 247                 33 % 
- Size: 165 KB (164947 bytes)      10 %