Like so many things relating to performance, it has a lot to do with what you're doing and the other components in your system.
If you have a low-end PC, which I doubt anyone on a site called "Super User" would :-), it's probably beneficial. For those with capable systems with at least 2G memory (and 4-8G is becoming more common), ReadyBoost probably doesn't add enough performance increase to warrant dedication of an entire flash drive and more important USB port.
Memory is cheap, I just doubled my HTPC upgrading to 4G, using the same exact memory I built the system with originally (2x Corsair twinxsomethingfast) for less than half the price I paid a year ago ($35 vs ~$75). I have a 4G memory flash drive that I'd used with ReadyBoost prior, and it "feels" faster with the actual memory. Boot times are irrelevant, my system is a media/gaming PC and runs pretty much 24x7.