edit: now taking recommendations on how to format this better
There are three deficiencies in current estimation algorithms.
Contrary to popular belief they are not nearly difficult enough to throw our hands up over.
The reason most people writing the blogs, and people here aren't aware of the possibility is as best as I can tell due to field of study and schooling breadth. A modest yet also very comfortable remedy should be possible for [a graduate with more recent training than the blog writers] [a multibillion dollar company] Microsoft.
I will attempt to roughly explain why.
The points of failure are as follows. The kernel:
1. cannot reliably predict future IO load due to circumstances outside scope of the kernel
- nothing should be done about this as it's a very unbounded P=NP problem.
2. does not track IO heuristics in any useful level of detail. Utilization is a much broader concept than disk/network read/write speed.
3. were they tracked, would not have use for the heuristics
- little has been done here, where we do most of the work
- this is where we put the data from #2 to use
- rough statistical analysis of file weights and locations to determine how much hopping we're going to do. The weight + location gives us a prediction
- combine with current disk load weights and locations
- to estimate what we think average read / write speed of the number of files F will be
- which we compare to fine tune our model
- which will let us quite accurately estimate the progress bar and time to completion
- the method of analyzing for purpose of predicting... here is patentable
The point of all this is our model is only 2a = F*(b x c) + d complex
Where a, b, and c have 3 states each: the file manager peaks at the files or just the metadata before copying, and F*(b x c) + d is not an expensive computation; if you want it more accurate use a lookup table with more states and there's hardly any calculation at all.
note: dimensions here are for a platter, would be different with an SSD-- beginning/middle/end wouldn't matter
The key difference between what I described and previous implementations that we've seen so far would be, in short, observing filesize and file distrubtion/entropy on the disk and thereby accurately accounting for the time element.