Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have an internal HDD which contains a directory, which contains a series of folders, which contain a total of about about 10 million small text files. Whenever I open this directory in Windows Explorer, the hard drive goes absolutely nuts and sounds like it's having a heart attack.

Resource Monitor shows that it's due to a roughly 11.5 MB/sec access to the drive's NTFS Master File Table (see below screenshot):

enter image description here

Indexing has been disabled for this directory, and all its subdirectories, as it is known that drive indexing and large numbers of small files are a volatile combination. Likewise, the directory only contains a couple folders (each of which house a large number of files), so none of the files in question are ever being displayed onscreen.

As such, I am a bit baffled as to why this intense reading occurs. It slows down any other processes which are working with the HDD to a near-standstill whenever it happens, so I have recently started navigating the drive and its contents via Command Prompt/PowerShell, which, for whatever reason, does not seem to trigger the NTFS reading-frenzy.

I'm not really a tech person, and so I don't know the details of how Windows 7 accesses drives to display files and folders, so I have a couple questions:

  • Why is the NTFS Master File Table being accessed, even though none of the files in question ever appear on screen, and none of them are ever being opened?

  • Is it absolutely necessary?

  • If not, is there a way to disable it? If so, what negative consequences would result?

  • Is there a more efficient (in terms of reducing disk wear and unnecessary file table access) way to set up a directory whose sole purpose is to house a large number of files?

Additional info: The drive is healthy (no S.M.A.R.T errors, and CrystalDiskInfo says there have been no problems), and is not a boot drive.

share|improve this question
    
have you ever defragged this hdd? –  cybernard Jul 17 at 23:09
    
@cybernard: No, but I don't see why that would be a problem, considering that 1) the files have all been added recently, and not many deletions or modifications have occurred, and more importantly 2) none of the files are ever displayed or opened during the operation (only the 10 or 11 folders containing them appear onscreen). –  DumpsterDoofus Jul 17 at 23:17
    
have you ever done a chkdsk /r To see if the drive has gained any bad sectors over time? –  cybernard Jul 17 at 23:21
    
if the files are so small that they fit in the MFT , that is where they will be stored. I think the size is around 600bytes or smaller. The Idea is you already have a specific space allotted for each file record, if the data also will fit in the file record , that is where it will be put. There are better explainations on the sites that explain the file systems. With that many file items, your MFT may become fragmented. In most normal situations the MFT is actually way to big. You have a rare use case here where it might actually be too small –  Psycogeek Jul 18 at 1:35
    
Windows 7 now parses much more of the file information than XP ever did when hopping into the folder/directory for the files. It probably follows along with other "Pre-Do" stuff that attempts to speed up the system, and can instead slow the system , when you were not needing to Do what it pre-doed. It does not matter how the explorer is set to view the info it still bumbles through it all. Therefore a good answer is the one Tamay provided, have it in smaller groups better organised. –  Psycogeek Jul 18 at 2:22

1 Answer 1

A good way to store this many files is by adding some directory structure to it.

For example see the "algorithm" described here: http://serverfault.com/a/95454/5109

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.