Others mention RAM which is also quite true. RAM is the most cost-effective upgrade/enhancement for "bang-for-buck" ratio. The second option others have mentioned is solid-state disk (SSD) drives, but your question seems geared more for the CPU/RAM/Chipset/bus rather than storage itself. SSDs are great but way too expensive. I'd have to say the biggest bang (overall) IMO is the integrated CPU memory controller.
For a long time, front-side buses (FSB) were a big deal as the FSB frequency (MHz) and bus width affected I/O from RAM to the CPUs. Nowadays, newer CPUs are putting the main meat of the FSB onto the CPU itself which is a huge performance gain.
Case in point: Intel Nehalem. AMD started using integrated memory controller (IMC) years ago back in 2003/2004 for their Opteron line and I believe passed it onto Athlons and the like. Intel? Nope. FSB all the way. They stuck by the FSB and simply tried to gain performance back by upping the L2 cache to make up for the slower I/O. The outcome from those few years? Intel was struggling to keep up with AMD in terms of price/performance. Now? Oh look - Intel has finally put an integrated memory controller on Nehalem. How quaint. Now everyone loves Nehalem.
For years, AMD has used IMC and with it been able to meet if not exceed a large portion of comparable microprocessors from Intel with lower clock speeds, slower/cheaper RAM and typically less power. Nehalem performance-wise is a great CPU. Until you factor in the costs of a newer chipset, newer RAM and the energy required to run the thing, then the "bang for your buck" picture starts to look a little foggy.
So for me "bang for the buck" in CPU terms means solid performance, overall costs (AMD tyically wins this battle), low energy costs and affordable/stable RAM compatibility. Other people may interpret differently, but I find it easier (and cheaper) to live by this set of principles/mantras/whatever. My wallet is a little heavier and the TCO of the computer itself will always be less rather than choosing speed/uber-cool/energy-guzzling/premium-priced components.
Edit: I realize I sound like an AMD fan-boy and to a certain degree, I fully admit I am one. I just wanted to point out IMC architecture and its huge effects on the CPU performance and how everyone seems to neglect its importance. It pays to pay attention to the minutia of technical details each vendor offers. I'm all for competition so having both Intel and AMD to me is a good thing. We as consumers need choice in the marketplace.