Tell me more ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.
up vote 2 down vote favorite
1

I have a very heavy I/O application that is constantly receiving data through the network and writing to a specific directory on the HDD. Then, the application need to load the files from that specific place. One of my clients has been experiencing slowness and when I try to access the directory it takes quite long before I can see the content.

My gut feeling is the HDD is degrading due to high I/O for a couple of years and I'm thinking of changing the HDD. Is there a benefit to write to multiple directories instead of using always the same?

BTW he is using Windows XP.

share|improve this question

3 Answers

up vote 9 down vote accepted

Not likely. A directory is not a physical location on the disk. New contents for the directory are placed on disk based on their size and the availability of free space on the disk. The actual file for the directory is just a list of those locations.

However, it is very possible to have too many items in a directory. Once you get above a few hundred or thousand items (depending on the OS and type of access), access to the directory can cause a huge slowdown. If your directory has more than a few hundred items consider adding sub-folders to minimize the number in any given folder.

One final thought is that read errors could cause a large slow down if they are not so bad as to cause total failure. If you suspect this, run a diagnostic on the drive and get your data off ASAP if confirmed.

[Edit] You mentioned above that these are radiology images. I assume with some form of patient ID or serial number/datestamp as the filename. Suggest you create folders for the first 2-3 letters of whatever denotes the ID and move the items into those. You can do this multiple folders deep and it will make finding the items no harder than it is now and greatly speed up directory indexing.

oldfolder
  123456
  123457
  123558
  ...
  55123
  55124
  ...
  99345
  99346

newfolder
  12
    34
      123456
      123457
    35
      123558
  ...
  55
    12
      55123
      55124
  ...
  99
     34
       99345
       99346
share|improve this answer
I had 60,000 sub-directories in the same directory root. In there, countless files. – code-gijoe Dec 3 '12 at 16:45
1  
That's way to many. I bet just doing ls/dir takes many seconds or even minutes. – Chris Nava Dec 3 '12 at 16:46
It took 10 minutes :) – code-gijoe Dec 3 '12 at 16:47
2  
Note that the same number of items will sit happily in a tree structure with a few dozen sub-folders in the first few branches. – Chris Nava Dec 3 '12 at 16:48
1  
A directory is not a physical location on the disk - The data within a directory doesn't have a fixed location, but the blocks used to store the information about what is within the directory doesn't move around. – Zoredache Dec 3 '12 at 17:34
show 5 more comments
up vote 3 down vote

The hard drive does not care. What you are probably experiencing is the filesystem letting the large directory become badly fragmented, as Windows is apt to do. You might try creating a new directory, moving all of the files there, and replacing the original directory with the new one.

share|improve this answer
moving files between locations on the same drive does nothing regarding fragmentation, because the file does not get copied or written. the filesystem just updates the index to show the file being part of the new directory. – Frank Thomas Dec 3 '12 at 16:40
Degrag would be ok then, it is a dedicated HDD, so I wouldn't mind to launch a full defrag. They are radiology images so there are literally hundred of thousand of small files. – code-gijoe Dec 3 '12 at 16:42
@FrankThomas, it isn't the files but the directory that is the problem. Starting with a clean, fresh directory eliminates the fragmentation. – psusi Dec 3 '12 at 16:44
1  
keep in mind, if files never get changed (once they are created they are only ever read) and the disk has sufficient free space, there shouldn't be much fragmentation regarding many small files. fragmentation occurs when a file is updated (changed and saved) but there is not enough contiguous space following the previous end of the file to hold the added data and place it in one contiguous block. if can also occure at file creation if the drive is already so populated that there isn't a contigious block large enough for the file, but if they are all small, that is unlikely. – Frank Thomas Dec 3 '12 at 16:46
1  
@Psusi, I think we must be talking about different phenomena when using the word 'fragmentation'. directories do not become fragmented (or more precisely file scattering algorithms cause files stored on disk to have no direct relation to the file system index structure). Free space fragmentation can cause file fragmentation at file create, but thats solved by increasing volume size or doing a free-space optimization. – Frank Thomas Dec 3 '12 at 16:55
show 5 more comments
up vote 2 down vote

Short answer no, or at least not because of the folder. File locations on disk have little or no relationship the the file system heirarchy, so two files in the same directory may be in consecutive addresses or may be on opposite ends of the disk.

The non-stop IO may be slowly damaging your drive (but HDDs are constantly dying every second of their lives) but spreading out over multiple directories will make no difference.

share|improve this answer
I'll check for fragmentation, but also, having an almost full HDD might just increase fragmentation probability doesn't it? – code-gijoe Dec 3 '12 at 16:43
definitely, though the smaller the file, the less likely. I think being almost full is likely the issue. – Frank Thomas Dec 3 '12 at 16:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.