3

On Windows on a HDD partition with Total Commander, if I make a file search on the file system root e.g. C:\, it will take considerable time: it display as it walks the entire directory structure (seemingly depth-first).

However if I do another search again, after I finished the first one, from the same root, it is so much faster. Based on the missing (physical) noise, I think the HDD is not involved this time, so the directory/file structure got cached.

I assume it is not Total Commander that is caching the directory structure, or is it? It seems like an OS feature to me, or a hard drive cache.

Where and how exactly is it cached then? (ie. By which component's which mechanism, in which structure?) How do the system know that the cache is up-to-date or not? If it can be cached entirely, why is it not cached by default?

2 Answers 2

3

I assume it is not Total Commander that is caching the directory structure, or is it?

Probably not. If you have a large number of files, you can see the same effect if you open a command prompt and type something like dir c:\abc.def /s You aren't likely to have any files with that name, but it takes a lot longer to finish the first time than it would take if you run it a second time immediately after.

It seems like an OS feature to me, or a hard drive cache. Where and how exactly is it cached then? (ie. By which component's which mechanism, in which structure?)

Windows/NTFS would be caching the items and there will also be a cache in your hard-disk controller.

How do the system know that the cache is up-to-date or not?

Because the only way of writing to the hard-disk is by using routines that know and therefore update or invalidate the cache. For windows the routines are probably the NTFS device driver, whereas for the hard-disk, it is the disk-controller.

If it can be cached entirely, why is it not cached by default?

Because the cache is better used for other things. To cache it by default would mean

  1. Every time the computer starts, it would have to read and store information on every file on your hard-disk regardless of whether it needed it. This can add a few seconds to your startup time. It is also quite common for people to have millions of files, which means the time could be significantly more.

  2. That memory is not available for caching anything else especially the files you are really working on.

  3. What happens if your profile/documents are stored on a server. Do you want the system to wait while it pulls the information on all your files across the network?

Also remember that just because everything can be cached, doesn't mean it should be cached. It can sometimes take longer to access information in an extremely large cache that in a smaller cache, which means that caching too much can actually slow you down. (By this I mean it is slower than when using smaller cache, but it would still be faster than using no cache)

2
  • I suppose a disk controller would cache binary data based on last access of "some position" on disk, right? As it should not understand file system structures. Also I expect the HDD cache be relatively smaller and thus the main|"most significant" cache space is somewhere in the kernel memory space, right?
    – n611x007
    Apr 2, 2013 at 13:49
  • 1
    Correct. The HDD cache would normally be a lot smaller than windows (it is normally in the order of 16 or 32 Mb), so if you have a million files, the HDD cache would not be big enough to cache them all. But the cache in the disk controller would be more complicated that just remembering stuff at "some position" on the disk, rather it would be caching the most active sectors or blocks, so if the directory you are accessing happens to be stored in a sector/block it had previous cached, it would be able to return it faster.
    – sgmoore
    Apr 2, 2013 at 15:00
0

What you are referring to is called indexing. Most search anythings have an index that is built over time to speed up searches. Start -> Search has an index, TotalCommander appears to use its own, Google has one for all the webpages you can find through Google.

There are various ways to speed up an index, and searching a file system. The index will be faster than re-searching and new recursive calls to dig through a directory though.

3
  • 1
    I don't think it's an index in Total Commander. If I close Total Commander and then reopen it, the search is still fast. If I close it, run CCleaner and reopen TC, the search is still fast. However if I restart the computer, it is slow again. It seems like in-memory cache to me.
    – n611x007
    Apr 2, 2013 at 12:04
  • There may be a release built in, but keeping all of that in memory would not be reasonable for large drives, especially considering the age of the application. When TC was first introduced every bit counted in RAM, seeing as it has a 16 bit version too. If, the cache / index is resetting after a reboot I would say a temp file is the index, or ghisler has built another check in. But! I see no discernible difference in first search and second, third etc search on my machine. i7, 4GB RAM, SSD... Apr 2, 2013 at 13:47
  • On SSD (I also possess some) it is not possible to reproduce impressing results because it needs no seek time and I imagine it may even be handled differently as a drive. But to eliminate TC I've made a python script with os.walk to benchmark this performance and between the first and the subsequent searches, there was a 2 order of magnitude difference in time (ie. 0.9 vs 90 s).
    – n611x007
    Apr 4, 2013 at 9:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .