After I download a song from the iTunes Store it will "process" the song, and the processing portion of the ordeal typically takes a lot longer than actually downloading it.
A typical situation will consist of the song downloading in 3 or 4 seconds, and the processing taking about a minute.
I know if you tell iTunes to keep your folder organized (which I do) then any time you download or import a song it has to create a new folder for that artist if one didn't already exist, and the same thing with the album.
Does doing those kinds of things take all that time, or are there other things going on behind the scenes?