Most power conversion devices contains coils, such as transformers or inductors. These components use electromagnetism to convert AC mains power to low-voltage DC power. The varying magnetic fields generated by these components can cause them to physically vibrate at high frequency, resulting in a high-pitched noise.
Most modern AC adapters are switched-mode power supplies. The internal switching frequency of an SMPS is typically low when unloaded and increases with load up to a certain point depending on the design. The no-load frequency is often low enough to be within the human hearing range. In addition, in low or no-load situations, the PWM used to regulate voltage at the inverter stage will be at a low duty cycle creating a "spikey" output profile which is more prone to causing vibration in coils, and the transformer itself will tend to vibrate as well (see Daniel R Hick's answer below for more details). Together, these can lead to audible noise especially in cheaper units which fail to suppress this noise.
Under load, a properly functioning SMPS should operate at a frequency well above the human hearing range, typically 50 kHz or higher (although some older designs operate at 33 kHz). However, the same noise can occur under load with a poorly designed or defective power supply as the coils may vibrate under electrical stress at a subharmonic frequency.
Coils used as inductors or transformers in other electronic devices, including those on motherboards, graphics cards, or other computer components, can also vibrate during operation. As such, a defective device can generate audible coil whine during operation.
This is why you sometimes see weird gobs of glue on coils inside electronic devices. The glue helps reduce the vibration and noise the coils generate during normal operation. It is entirely possible for users to apply glue onto coils using a glue gun to suppress coil whine, and people have successfully done so on their computer parts. However, you generally can't do this easily on small wall chargers of the sort you mentioned without risking damage to the charger or exposure to potentially dangerous voltages.
Ultimately, a whining noise isn't necessarily a sign of trouble in cheaper wall chargers when little or no power is being drawn from them. However, a computer PSU or laptop charger that generates coil noise especially when under load may be defective and you may want to consider replacing it.
More information on coil noise can be found in this Wikipedia article.