Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have read in the past that is it good practice/performance to limit the amount of files you put in each folder, in this case it is user uploaded photos,

My site has around 1,000,000 user photos so far and growing daily so I structure the folders that photos are saved in to something like 9/8/5 so it puts X amount of photos per folder then starts putting into a new folder.

I know it is not a huge performance issue but I believe too many files does hurt seeking time

I am wanting to know what is a good amount of files to limit a folder to?

share|improve this question

migrated from stackoverflow.com Aug 6 '09 at 19:14

This question came from our site for professional and enthusiast programmers.

    
That greatly depends on the file system. –  Joey Aug 6 '09 at 14:23
2  
What filesystem? What operating system? –  Rob Aug 6 '09 at 14:23
    
It would be on a LAMP setup –  jasondavis Aug 6 '09 at 14:27
    
Depends on the filesystem. On an MS-DOS system with FAT fileystem I ran into performance problems at around 500-1000 files per directory. –  Karl Voigtland Aug 6 '09 at 14:33

5 Answers 5

up vote 1 down vote accepted

Similar to this question.

share|improve this answer
    
thanks thats helpful –  jasondavis Aug 6 '09 at 14:31

I'd personally not worry too much about it but I'd personally change the file system to be stored in a date format.

Such as 2009/08/06.

That'll cut down on your daily amount at least by keeping it organized but your annual amount wouldn't change.

I've never really bothered with the amount I put in a folder, I just keep things organized.

share|improve this answer

In ext3 filesystems you can have approx 32000 subdirectories in a directory:

For ext2 and ext3 the hard limit is 31998 due to EXT[23]_LINK_MAX being 32000 and each of those subdirectories (folders) has a link back to its parent, which began life with a link count of 2.

The limit for decent performance would likely be quite a bit less.

Also, structuring your data to avoid hitting such limits has other benefits, for example it makes easier to split a large data store across multiple volumes or even servers.

share|improve this answer

At some point, you'll want to consider looking at MogileFS to store your files. Then you don't need to worry about file system limits, you get almost automatic replication, and it's fast. I can't say for sure if you are at the point with a million pictures, but it's worth looking at. May be easier, in any case, than rolling your own solution just to deal with file system limits.

share|improve this answer

I usually split so there's 10000 files per folder. It's fairly easy if you have have some sort of numeric identifier for your files.

As an example, if you're using a file with identifier = 9876543, you could use the following process:

  • Formats number with 12 digits (000009876543)
  • Uses the first eight digits as a folder identifier (0000/0987)
  • Save the file as (0000/0987/myfile.jpg)

While most file systems allow you to keep more files in a single folder, this makes things more organized, it'll be easier if you eventually need to split in multiple volumes, and if you're using Windows it also makes sure Explorer won't freeze trying to read 1M files. :)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.