When a folder appears to move instantly, it's because the operating system has been able to update the file allocation table* to without moving the actual file data.
It's slower to do this with small files, as for each one a change has to be made in the table. If the files are really small, this may even take a similar amount of time to actually moving their data.
I don't know in exactly what circumstances the file data has to be moved when it's on the same partition as its destination, but I would assume there's nothing you could do to avoid it.
As other commentators and answers have hinted at, copying to a different drive (different disk, different partition, to a usb stick, to a drive over the network, etc) would of course mean you had to copy the full data, so things will be slower. You will be affected a lot more by the bandwidth of what you are copying to.
(*Think of file data as books in a library, and the file allocation table as a set of index cards showing you which section the books are in. On disk editing the index card has the same effect as actually moving the books)