A combination of 3 factors:
- Seek Time
- Transfer Time
- SIZE OF CACHE
In my experience, all of these come into play... But often SIZE OF CACHE is the biggest issue.
For example, in most of the computers today that are being sold, Internet Explorer is still set to use 5% or 10% of your hard drive... For a system with a 500 GB hard drive that's 25 GB (@ 5%), which means that IE will spend more time going through the cache, than it would have if you just re-downloaded the page. Even if it's been changed to 1%, that's still 5GB of data...
Now, Firefox & I assume Chrome may have better standard defaults, but generally on any high speed connection, I change the cache size to be at most 10-20 MB (Often I use 5 MB). I would not suggest anything higher than 50 MB, and even then I would seriously question the choice of going that high...
After that, I would suggest Seek speed, and then transfer speed.