What is the maximum amount of files to put in 1 folder for performance?

Question

I have read in the past that is it good practice/performance to limit the amount of files you put in each folder, in this case it is user uploaded photos,

My site has around 1,000,000 user photos so far and growing daily so I structure the folders that photos are saved in to something like 9/8/5 so it puts X amount of photos per folder then starts putting into a new folder.

I know it is not a huge performance issue but I believe too many files does hurt seeking time

I am wanting to know what is a good amount of files to limit a folder to?

Answer 1

Similar to this question.

Answer 2

I'd personally not worry too much about it but I'd personally change the file system to be stored in a date format.

Such as 2009/08/06.

That'll cut down on your daily amount at least by keeping it organized but your annual amount wouldn't change.

I've never really bothered with the amount I put in a folder, I just keep things organized.

Answer 3

In ext3 filesystems you can have approx 32000 subdirectories in a directory:

For ext2 and ext3 the hard limit is 31998 due to EXT[23]_LINK_MAX being 32000 and each of those subdirectories (folders) has a link back to its parent, which began life with a link count of 2.

The limit for decent performance would likely be quite a bit less.

Also, structuring your data to avoid hitting such limits has other benefits, for example it makes easier to split a large data store across multiple volumes or even servers.

Answer 4

At some point, you'll want to consider looking at MogileFS to store your files. Then you don't need to worry about file system limits, you get almost automatic replication, and it's fast. I can't say for sure if you are at the point with a million pictures, but it's worth looking at. May be easier, in any case, than rolling your own solution just to deal with file system limits.

Answer 5

I usually split so there's 10000 files per folder. It's fairly easy if you have have some sort of numeric identifier for your files.

As an example, if you're using a file with identifier = 9876543, you could use the following process:

• Formats number with 12 digits (000009876543)
• Uses the first eight digits as a folder identifier (0000/0987)
• Save the file as (0000/0987/myfile.jpg)

While most file systems allow you to keep more files in a single folder, this makes things more organized, it'll be easier if you eventually need to split in multiple volumes, and if you're using Windows it also makes sure Explorer won't freeze trying to read 1M files. :)