How they can get your contacts: (just adding a bit).
E-mail (and other passed around contacts) are like any "6 Degrees of separation" http://en.wikipedia.org/wiki/Six_degrees_of_separation
When people blindly or accidentally FWD or CC or BCC or any other easy ways to launch everyone's contacts out into the wild, they end up passing your contacts out. Because your only 6 away from 6 billion other people on the planet, some person who utilizes the web for these purposes, gets it, and it becomes "unlisted" at that point.
That invitation to 25 people for the birthday, distributed obituaries, political cute e-mails you gotta send out to your friends, that web meme that you shot out to 2 friends, who shot that to 4 others, complete with your address still stuck in the thing. When you see your e-mail address plastered to the top of a rack of addresses, you can be assured that it has gone around the world a few times already :-)
In the above cases, there is no malware required, as that is covered here already.
The above scenarios have been applied frequently with My e-mail addresses by customers , friends, family, and even high end tech people. It is not long after my e-mail is distributed using methods like mass FWDs, that a perfectly clean e-mail handed to few people (by me) is found out, and used for something other than it was intended. I can often show where it was handed out by someone trying to mass communicate, even if that was by accident that I was included.
Over a mere few years, my machine Has received about 25 times as many e-mail addresses then my original contacts would be, people I have no idea who they are. I ask people Not to FWD me that stuff, and not to CC me into the lists, but it still happens that I acquire lots of peoples contacts (that I do not want).