You experience this effect as a mix of the hardware and the audio mix.
Your audio connector is most likely a stereo connection. There are two wires in your cable, and they are carrying an audio signal each, which is separated in the connected/plug. Have a look at the black stripe on the plug (or white, if it's an iPod headphone!).
By connecting halfway in, you're only getting one channel of that two channel audio stream. The wiki article on connectors explains the hardware nicely.
Audio is typically mixed in stereo these days. It is still important for an audio engineer to consider the various audio environments, where many radios will be mono. This is curious in 2013 to see a lot of new models of radio units released with a single mono speaker. In this case, a stereo signal is summed as a mono signal. Both streams get squished together, in a fashion.
But largely, our stereo mixes exploit the stereo format to sound interesting. A large part of mixing audio is placement both in the left and right of the perceived audio environment, and forwards and backwards by use of things like reverb. The best mixes "place" audio in this three-dimension space effectively.
A quick example; for most radio-friendly music, the vocals are largely going to be in the center. Bass is as well, this is both another foundation element, and for reasons of phase and some fun legacies of the days of cutting records. But you will notice backing vocals, guitars, and drum kits are panned across the spectrum. Or even how old Beatles records would have really dramatic panning; entire instruments hard panned.
Imagine what happens with your signal when one of these channels drops?
What You're Hearing
When you only semi-patch the connection of a stereo signal, you're only getting part of that mix. Depending on the patching of your audio chain, this might be only that channel coming through, or else summing or inverting. Normalization affects the audio path as this other Wiki article introduces.
But the effect you mention is likely a product of inversion. The following article shows how you might do this with an audio editor, and gives a general idea of what's happening. In your case, the center channel is being inverted.