When my PC is turned off and I bring my head very close to the PSU, I hear a high frequency noise. Is that normal? I can't hear when I am like 2 metres away from it. And it's only there when turned off. My ethernet port blinks so the pc is using some power even through it's off.
The most common cause of high-pitch audible noise is something loosely known as coil noise. Specifically, it's a vibrating toroidal inductor. As you can see in the picture below, they commonly apply glue at the factory to dampen the vibration. It is harmless to the electronics, but annoying to people and animals.
To answer your question, it's not "normal", just a common flaw. Personally, I find such noises unbearable, and a sign of poor manufacturing. If the computer is under warranty, I would ask for the power supply to be replaced. If it's not under warranty, and it uses a standard form factor, I would replace it with something that has met the criteria for 80 PLUS certification.