I would like to specify a variable frame rate with a MAXIMUM value, and allow libx264 to down the frame rate as it sees fit. The idea here is to get extra compression when there is something like an extended still frame
In my understanding, this may be possibly in a comparably clumsy way, but is undesirable for some complex and counterintuitive reasons
Even though an x264 stream has a framerate(s), frame rate is more a container-level problem than a codec one.
In a passthrough VFR encode, there will be what is essentially a text file detailing what the frame rate is over what frames/times, and in encoding a source, a function like tcfile-in or tcfile-out pass the timestamps through to the encode, to map the rate locations and keep the video subjectively consistent from source.
The low-framerate idea is a logical one, but doesn't work out for several reasons. Though x264 is VFR-aware with some capabilities, I don't think there is an analysis function that will vary the framerate with regard to motion in order to lower the file size (in a way analagous to the many bitrate controls).
The source is also a problem: VFR sources will by default retain their frame variability, but apparently encoding a CFR file at variable bitrate (a good idea sometimes, esp. when telecine is needed) will simply produce the same CFR.
This means you would probably have to re-write bitrate by hand (i.e. timestamps of slow scenes muxed into the file), or resort to a frame decimation algorith like dup, dedup, and exactDedup for avisynth. If your video does have extremely low motion, some frames (even half?) would be thrown out. The problem is that these algorithms are not advanced, and don't make good choices with "real life" footage as to what will contribute to the best encode.
Also, removing frames that contain things like I and B frames reduces the amount of detail available over time, which causes motion to look "steppy" and can interfere with the other basic video parameters and cause artifacts like aliasing.
And because of the way the quantizers work, x264 will actually decrease the bitrate disproportionately further in these scenes of low motion. Unless you have a slideshow of identical images, there will be motion (if only grain and other artifacts) and there will be a loss in quality that would not be seen without drastic changes to the bitrate.
And finally, the reason there aren't many options to do what you want is that x264 is really good at managing bitrate just using temporal compression (recording changes in partial frames). Going to 1/2 framerate will not cut the file size in half; 10% is probably a realistic gain to expect from low motion or animation.
So in short, dropping the bitrate of your static scenes will do very little for your file size, but will add a host of quality and sync issues, not to mention incompatibility with video editing software.
If you do want to try a decimator, you might be able to limit the maximum new frame rate by using the levels options, each of which species a maximum resolution and framerate. Unfortunately, you would probably have to work at very low resolutions to get the kind of frame rates you want, using profiles. It comes back to editing the rates by hand, either entirely or to correct frame rates you think are too high. Either way, it will take juggling to keep the sound in sync with the new framerates if alterations are made after the encoding process when the tcfile is conserved.
The takeaway is that spending time optimizing the many bitrate settings will yield much more in the way of file size management, and improve the quality of your video, rather than cause complications for little gain. Preserving the original FPS is probably the best idea unless you're aiming for broadcast or media standards. Players are well capable of playing variable bitrates (unlike editors), and the more frames in your video, the smoother the playback and perhaps the smaller the file size, due to smaller changes in motion between frames.
Here's a collection of links to standards info and forum discussions that should help with this confusing aspect of encoding:
-avisynth decimation tools
-fps and -r switches
-x264 General (tcfile, fps)
-timecode file standards
-Levels and profiles
-Short, clear CFR/VFR setting summary ("framerate" section)
doom9, videohelp, &c theoretical discussions