When I first came to the UK I built two small footprint, quiet machines, working on the assumption that I would be living in a small shoebox sized flat. These machines had P3 chips in them, so cooling was somewat easier than a modern system, but many of the issues still apply - the machines had to be small and quiet.
Most of the parts were recycled from my existing desktop machines at the time. Essentially I was downsizing my computers. Some things I used:
Quiet CPU fans (in this case from Molex, but many outfits make quiet PC fans now).
CPUs that didn't draw much power, in this case the existing P3 chips from my other machines. Most modern CPU lines come in low power versions. They are often a bit more expensive and not as fast as the top-of-the-line model, but the speed is usually not much slower. You could also look into something based on an Intel Atom, which should run cool enough for a passive cooling system.
I picked a specific motherboard (in this case an Asus CUV4X-M). This was mainly to get a micro-ATX form factor as the machines were going into small footprint cases.
The existing hard drives I already had. Modern hard disks don't make all that much noise - much less than high speed fans.
The existing memory, floppy drives, CD/DVD drives and other components I already had.
Low RPM fans where I had to. In this case I didn't actually have to change the case or PSU fans as they were quiet enough. The only fans I had to change were the CPU fans.
Video cards with passive cooling, in particular a Matrox G550. You can get passively cooled cards with reasonable performance for anything short of a top-line gaming rig.
Rounded cables to minimise the interference with airflow in the case (remember these were small cases with relatively slow fans). I also got some adhesive backed anchors and used ties to keep the cabling tidy and out of the airflow.
Cooling and soundproofing technology has gotten a bit more mainstream these days. You can get quite a variety of kit from QuietPC and various other outfits now (I got my CPU fans from there). Quiet and passive (fanless) cooling solutions, liquid cooling and soundproofing materials are all readily available now.
I got quite acceptable noise levels from the fairly basic setups I built there. You can build more elaborate solutions with the application of more money but there's no point in paying for stuff you don't need.
These were the last machines I built and were essentially small footprint rebuilds of machines I already had. They were fairly cheap to make - I bought CPU fans, cables, two motherboards and one secondhand graphics card. I could also re-sell the old motherboards and cases.