I almost never get stuck with things like this and kinda figure things out myself... but I'm at a complete loss here.
I've used Handbrake, X-Media Recode, and lots of other encoders and they're great.. but they just don't have clearly defined options for maximum and minimum bitrate control in VBR. That, or I just can't find the options. Is there a GUI I could use for this? Or is it only possible through command-line in x264 core? I've only been able to work with constant bitrate so far and I REALLY need to use variable bitrate.
I found these extra options and I think they might be for variable bitrate.. but experimenting and encoding with that still produces a result with constant bitrate:
VBV buffer size, VBV maximum bitrate, VBV initial buffer, Bitrate variance.