Well obviously we all have different understanding of what is needed.
First off - I would reuse every single thing you can from previous computers that does not "directly" (read hugely) influence performance - such as chassis, optical drives, chassis fans and so forth.
Next up - get yourself a good PSU (as you seem to have understood already, I too made that mistake once), a quality motherboard (quality doesn't mean it has to be the most expensive, but get one from a known brand with a known good chipset in it - the chipset means ALOT in the motherboard). The memory and CPU I'd "skimp" a bit on as well like others have suggested. That is, I'd go with a Duo/Quad Core CPU - try to get the newest product-line possible, but pay close attention to the prices. Say their clock speeds and prices are as follows
2.4GHz 100$
2.6GHz 110$
2.8GHz 170$
3.0GHz 300$
In this case I'd go for the 2.6GHz, simply because you pay only a little bit more compared with what you get - but once we go beyond this the price jumps become too much for me wanting to pay that much extra for a few extra Hz. IMO the product line of the CPU is often way more important than the model within the product line.
As far as memory goes, decide how much memory you want - and again, try to find the cheap sweet spot. Memory is cheap at the moment and it shouldn't be too hard to find cheap good memory - try go get some that is not running with the slowest timings you can find, or get some that is rated to run just slightly faster than you intend to run them at, so you can get slightly better timings - but there's no need to spend a large amount of money on memory.
If this is a rig for gaming - the graphics card is where you want to REALLY spend cash - it is for the most part the deciding factor in everything related to your gaming performance so go nuts here. If not, you can EASILY save a ton of money there as well.
A harddrive for gaming really only needs to be big - a faster harddrive will get you faster loading times, but for the main part - that's it. Get a normal 7200RPM drive and you'll be good. If you want a general faster computer elsewhere though, a good harddrive is a worthwhile investment too - maybe even considering a RAID setup (I gotta warn you about RAID 0 though - it sucks when the RAID breaks - but a RAID 10 if you can afford it, is really nice).
The last bits and pieces all comes down to what you can save from previous rig's. Do you have keyboard and mouse? Do you have a monitor? Do you have optical drives? Etc. Apart from optical drives - the rest of the parts left are the ones you'll physically interact with. As far as the performance of the computer, it doesn't really matter that much. But when we talk about interacting physically with the computer this is where the differences are made - so it all depends on how you want that to feel. Especially a bad chassis can really make assembling a computer a pain, but then again, you don't do that so often - the rest is pretty much up to you - whether you think it is worth the money or not, and of course, whether or not you can afford it.