3

My site requires 10,000+ 1-2 seconds mp3 clips, saved as individual files. They are each about 12Kb in size.

Are there any potential issues with storing them all in one huge folder as word_1.mp3, word_2.mp3 etc...? Are there other approaches?

Many thanks, Patrick


EDIT

Sorry for not being clearer. The files will be uploaded to the folder via FTP but a javascript function (SoundManager2) picks the mp3 files to play. Will there be speed issues with crawling the folder for the right file?

1
  • 1
    What operating system, file system, and physical storage medium are you going to be using?
    – shadit
    Dec 22, 2009 at 16:11

5 Answers 5

2

Can you sort and group your mp3 into themes? If so, then I would put them into separate folders, all from a root folder name mp3, clips, or whatever suits your use.

Maybe it's just me, but having 1000s in one folder can be hard to manage, plus some FS might not like it.

2

From personal experience some file systems will grind to a halt with 1k+ files in a folder. I remember NTFS being one of these about 3-3.5 years ago on win2003/XP.

Reiser fs is supposed to be designed for this sort of thing, Other UNIX file systems might do well as well. To back this up I will say that in the UNIX world everything is a file, and heavy mail daemons rely on this and often stuff folders with thousands of emails (qmail).

2

There are a couple of reasons I would suggest organising them a bit more.

Firstly - and this may not be an issue for you - if you were to access that folder via FTP, it will be a very slow folder to browse. By organising the files into a number of folders, each folder will be easier to browse.

Secondly, I have experienced a problem on Windows where there were too many files in a folder and it all ground to a halt - that was with 2,000 files - apologies as I can't remember the exact details surrounding this issue.

1

a nice piece of advice regarding large folders on NTFS volumes:

Here's some advice from someone with an environment where we have folders containing tens of millions of files.

  1. A folder stores the index information (links to child files & child folder) in an index file. This file will get very large when you have a lot of children. Note that it doesn't distinguish between a child that's a folder and a child that's a file. The only difference really is the content of that child is either the child's folder index or the child's file data. Note: I am simplifying this somewhat but this gets the point across.
  2. The index file will get fragmented. When it gets too fragmented, you will be unable to add files to that folder. This is because there is a limit on the # of fragments that's allowed. It's by design. I've confirmed it with Microsoft in a support incident call. So although the theorical limit is several billions, good luck when you start hitting tens of million of files.
  3. It's not all bad however. You can use the tool: contig.exe to defragment this index. It will not reduce the size of the index (which can reach up to several Gigs for tens of million of files) but you can reduce the # of fragments. Note: The Disk Defragment tool will NOT defrag the folder's index. It will defrag file data. Only the contig.exe tool will defrag the index. FYI: You can also use that to defrag an individual file's data.
  4. If you DO defrag, don't wait until you hit the max # of fragment limit. I have a folder where I cannot defrag because I've waited until it's too late. My next test is to try to move some files out of that folder into another folder to see if I could defrag it then. If this fails, then what I would have to do is 1) create a new folder. 2) move a batch of files to the new folder. 3) defrag the new folder. repeat #2 & #3 until this is done and then 4) remove the old folder and rename the new folder to match the old.

To answer your question more directly: If you're looking at 100K entries, no worries. Go knock yourself out. If you're looking at tens of millions of entries, then either:

a) Make plans to sub-divide them into sub-folders (e.g., lets say you have 100M files. It's better to store them in 1000 folders so that you only have 100,000 files per folder than to store them into 1 big folder. This will create 1000 folder indices instead of a single big one that's more likely to hit the max # of fragments limit or

b) Make plans to run contig.exe on a regular basis to keep your big folder's index defragmented.

Source

Contig is freeware.

if you're not comfy with the command line, you may use Power Defragmenter, a GUI written for Contig.

1
  • Thanks for this answer. Does the above apply for a normal folder online, and does NTFS only apply for PC folders? Dec 22, 2009 at 18:15
0

10000+ files are not that much.

Every file system that not used a linked list for dentry storage should be fine, e.g. even ext3 with dir_index option. The inodes and dentries of 10000+ file are easily cached in RAM.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .