A Virtual Hard Disk is a filesystem inside of another filesystem.
In order to read the contents of the VHD, the operating system has to go through two levels of indirection. Let's look at it starting from the perspective of the hardware.
So the hardware (hard disk, SSD, whatever) reads the "raw" sectors from the disk. From these raw sectors the NTFS filesystem driver tells the operating system the information about the sectors on the disk where the VHD data resides. After all, the VHD is a file containing a filesystem. Before you can read the filesystem within the VHD, you first have to read the VHD itself as a file on the underlying NTFS filesystem.
Now, you've read the bytes of the VHD file. And what do those bytes represent? They're... another NTFS filesystem, with its own list of files and directories, metadata, streams, file fragments, and so on.
One of the original answers to the question mentioned fragmentation. This can become a huge issue when you have filesystems nested within filesystems. The worst case is if both filesystems are fragmented: you can have fragments of files in the nested filesystem which are themselves fragmented on the host filesystem! All of this "scattering" of the data leads to a very random file access pattern, which means the hard disk has to "seek" a lot. Seeking is particularly slow on rotating media (traditional hard disks); on memory-based devices such as SSDs, seeking is much faster, but it's still better to avoid seeks when you can.
So your problem is most likely to be fragmentation, but more specifically, nested fragmentation (fragmentation of both the nested and the outer filesystem).
Imagine you have a plastic ice cube tray for 10 ice cubes on your desk, and within each ice cube hole are 10 rectangular rods. Each rectangular rod represents a piece of a letter. You want to spell the word "APPLEJACKS". How are you going to do this?
Well, first, you have to look in all the ice cube holes and find the correct rods to assemble each letter. Then you have to make sure all the letters are in order. So you have a "problem within a problem" -- the micro-level problem is assembling each letter, and the macro-level problem is assembling the letters into the word. This is how nested filesystems work when fragmented.
Now, if you only had one layer of filesystem, this would be like having a solid wooden block with the letter printed on it in each ice cube hole. Now, even if the letters were scattered in random order, it would only take you a few seconds to re-order them to spell "APPLEJACKS" if you have the correct letters available. It is much simpler than having to assemble each letter itself.
Hope the real world analogy helped.