up vote 6 down vote favorite
1
share [g+] share [fb]

Estimation

xkcd

I know that the Windows copy dialog (in XP) stores the copy in memory first, and is still copying after the dialog closes, so the time is off, but why is the estimation of the time it will take to make a copy so unaccurate, even when this memory copying has been disabled (in Vista and Windows 7)? It seems so... arbitrary! How does the whole copy procedure work, and why can't Windows estimate it correctly (what's so difficult about it)?

link|improve this question
79% accept rate
Windows isn't the only culprit here, but I'd say out of all the estimation dialogues I've used, it's the worst by far. – Sasha Chedygov Sep 18 '09 at 21:10
Yup! I love windows though! – Maxim Zaslavsky Sep 18 '09 at 21:17
yet another reason to use Total Commander :) – Molly7244 Sep 18 '09 at 22:48
The progress bar shows the # of files completed, not the % time completed, fyi. – Factor Mystic Sep 19 '09 at 0:34
show 3 more comments
feedback

3 Answers

up vote 9 down vote accepted

In short: the poor algorithms and the jumpy estimation is actually an implementation weakness.

Other tools like TeraCopy do a better job. I think it is not worth explaining why their implementation is not good. They will have noticed it and will improve.

What is difficult:

  1. You have to take into account resource fluctuations (CPU/Network bandwidth/HDD speed mainly)
  2. You need to extrapolate the time it'll take by predicting the behavior (what windows file copy definitively does badly right now).
  3. Make adjustments time over time to your original estimation (I mean small adjustments not like in the funny picture above! :-) )

For this not only the amount of bytes but the amount of files to create play a role. If you have a million of 1KB files or thousand 1MB files the situation will be quite different because the former has the overhead of creating many many files. Depending on the filesystem used, this could take more time than actually transferring the data.

This dialog drove me mad also quite a couple of times:

  • On an older WinNT system, if you had a lot of small files to copy, it displayed the name and nice animation for each file slowing down the whole process to be practically unusable.

The modern windows copy stuff is not much better:

  • To compute the amount of data to transfer it seems to make a lookup first (that is what I suppose it does) so it takes ages if you select many directories until it effectively starts to make the job
  • Some built-in timeout impeaches big files to be copied (> about 60GB on my system). The pain is that it tell you that after having copied already more than 30GB over the network and this is lost bandwith and time because you have to restart from scratch!
  • Copy of files from one computer to another is damn slow for some reason??? (I mean compared with the available network bandwidth, using other tools it is faster to it's not a computational limitation)
link|improve this answer
Very interesting! – Maxim Zaslavsky Sep 19 '09 at 9:28
feedback
up vote 2 down vote

Here's the explanation by Raymond Chen, Principal Software Design Engineer at Microsoft:

Why does the copy dialog give such horrible estimates?

Because the copy dialog is just guessing. It can't predict the future, but it is forced to try. And at the very beginning of the copy, when there is very little history to go by, the prediction can be really bad.

Here's an analogy: Suppose somebody tells you, "I am going to count to 100, and you need to give continuous estimates as to when I will be done." They start out, "one, two, three...". You notice they are going at about one number per second, so you estimate 100 seconds. Uh-oh, now they're slowing down. "Four... ... ... five... ... ..." Now you have to change your estimate to maybe 200 seconds. Now they speed up: "six-seven-eight-nine" You have to update your estimate again.

The blog post quoted above has a long discussion of this issue, with some interesting comments.

Raymond Chen is a legendary person, "Microsoft's Chuck Norris", I don't suppose you're going to get a more authoritative answer. I'm sure he had at least seen the code in question.

link|improve this answer
feedback
up vote 0 down vote

The mathematically correct model is to actually do a naïve averaging and extrapolation:

transfer speed = data copied / time elapsed
time remaining = data remaining / transfer speed

The reason is that by the Law of Large Numbers the local fluctuations will cancel out in the averaged transfer speed, and this will give you the most stable result.

What Microsoft seems to do is to calculate the transfer speed at the latest time frame. This means that each local fluctuation changes the result significantly.

link|improve this answer
Your model will not properly handle long-running disturbances, like starting other file transfers in parallel, and will continue to tell me it'll just take 5 more minutes even though the same amount of data had just taken 20 minutes. A weighted moving average might be more accurate. – Daniel Beck May 11 at 16:48
@DanielBeck: Not exactly correct. The expected time will gradually increase. The question is how fast will it increase? Well, it depends on the elapsed time. If it was a long operation, e.g. it was already copying for 5 hours, then it won't increase the expectation much. But does the 15 minute inaccuracy matter for 5 hour operation? No. The point is that it gives you the best approximation in terms of relative error. Also you can't do something that will work much better in every scenario. – ybungalobill May 11 at 17:15
The problem of your model is that it absolutely does not react to transfer rate changes midway through the transfer. This will be just as insufferable as the fast-reacting Windows file transfer Example: 60GB transfer at 10MB/s at first. Time left at start: 100min. Transfer 54GB and drop to 2MB/s. After 90 minutes: Estimated time left at 54GB: 10min. Real time left at 54GB: 50min. After 115 minutes: Estimated time left at 57GB: 6min. Real time left at 57GB: 25min. After 131.67 minutes: Estimated time left at 59GB: 2.23 minutes. Real time left at 59GB: 8.33 minutes. – Daniel Beck May 11 at 17:44
@DanielBeck: the whole transfer lasts 150 minutes, so the maximal relative error is 50% in the beginning of the transfer where you can't do any better. At the 54th GB it's just ~14% off of total. (if it takes you 150 minutes, why 20 minutes matter?) Actually a very good estimate...That said, I understand your point. The way to improve this is not weighted moving average because you can't know what size of the window it should be (does this operation expected to take minutes like copying a file, – ybungalobill May 11 at 18:53
or hours through a p2p file sharing protocol where you get 10 minutes of 10 MB/s and 10 minutes of 0 MB/s). The way to improve this is to take the average weighted by time, not by size. – ybungalobill May 11 at 18:54
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.