The most common audible noise coming from computers (besides fan noise, of course) is from the transformers used in the power supplies. They are switched at very high frequencies and they generate a powerful magnetic field (that's how they couple energy from one side of the transformer to the other). That powerful magnetic field is essentially a big electromagnet, so any ferromagnetic material in the vicinity will be pulled towards the transformer and pushed away from the transformer thousands of times a second. Most things are soldered down, but some things (like the windings of the transformer themselves) can have a little play, so they move back and forth at the switching frequency (or a harmonic or sub-harmonic of the switching frequency). This is the most common source of physical noise, and it can be modulated by the load on the CPU (as the current draw from the CPU changes, the magnetic field intensity and duty cycle changes). However the most common source of this kind of noise in this environment is the transformers (sometimes called inverters) used to create the high voltages for the backlight of LCD monitors and TVs.
Since this seems to be a popular topic, I'll add a note on the other big source of noise in PCs. The noise discussed above is produced mechanically, you can hear it without any soundcards or speakers. If you're talking about noise you hear through your speakers, there's another source. CPUs and GPUs use 10s of amps of current from the power supplies, and that current varies depending on what the CPU/GPU is doing. The power supplies typically use the same ground return (usually a copper ground plane layer in the motherboard PCB) that all the other chips (including audio) use. Ohm's law says Voltage (V) = current (I) times resistance (R). An ideal ground plane (made of an ideal conductor) would be zero ohms from any point to any other point, so even 100A of current wouldn't generate a voltage (100 A * 0 ohms = 0 V). But a real-world copper ground plane has some resistance, say 0.010 ohms from one end to the other. So if the CPU current switches between 30A and 10A, the voltage across the ground plane may vary between 0.3V and 0.1V. This means that the ground the audio IC is relying on to "stay still" is actually moving up and down by 200mV. That makes the audio output of the IC jump up and down by up to 200mV (depending on what the processor is doing). Which you hear as noise.
This is a very, very simplified example - people have written books about this topic. I'm just trying to convey the basic mechanism.
