Sometimes you have to answer the question by saying you shouldn't answer the question;
I'm going to go out on a limb and say you shouldn't do what you're trying to do. If there is such a large problem with dupes, you should consider figuring out why you have so many dupes in the first place. Entire discographies is one good reason why this is the case.
Do you really like everything by a particular artist so much that you have to have every single release portably/at all times? If that's the case, why is hearing such a good song multiple times so much of a problem?
Here's an active reason why you don't want to do this, at least in my perspective.
I like the "atomic" nature of albums. After listening to it a few times, I begin to like not only the songs, but the order in which they're presented. If you start cutting songs out, you're going to find that you don't like that album as much, all because you saved 3MB on a large storage device. You can save that much space by crunching your library into a slightly lower bitrate/and perhaps a different format. (mp3 -> aac, just don't re-use the aac files anywhere else.)
Surely when you listen on shuffle, you are affected by duplicates such as this even less? On an iPod touch, I know that when you play by artist without shuffle, you'll hear the same song multiple times because of the same title and the alphabetical playback order.
In a nutshell, my suggestions:
- Don't do this.
- Don't do this.
- Don't do this.
- You can't have one physical file exist in multiple albums (to the best of my knowledge), and you'll lose the atomic nature of a given release if you start excluding songs.
- Don't put every single album you have for a given artist on your iPod. Pick the best.
- Listen by album instead of by artist, or;
- Listen by artist on shuffle.