1

I have read in the past that is it good practice/performance to limit the amount of files you put in each folder, in this case it is user uploaded photos,

My site has around 1,000,000 user photos so far and growing daily so I structure the folders that photos are saved in to something like 9/8/5 so it puts X amount of photos per folder then starts putting into a new folder.

I know it is not a huge performance issue but I believe too many files does hurt seeking time

I am wanting to know what is a good amount of files to limit a folder to?

migrated from stackoverflow.com Aug 6 '09 at 19:14

This question came from our site for professional and enthusiast programmers.

  • That greatly depends on the file system. – Joey Aug 6 '09 at 14:23
  • 2
    What filesystem? What operating system? – Rob Aug 6 '09 at 14:23
  • It would be on a LAMP setup – JasonDavis Aug 6 '09 at 14:27
  • Depends on the filesystem. On an MS-DOS system with FAT fileystem I ran into performance problems at around 500-1000 files per directory. – Karl Voigtland Aug 6 '09 at 14:33
1

Similar to this question.

3

I'd personally not worry too much about it but I'd personally change the file system to be stored in a date format.

Such as 2009/08/06.

That'll cut down on your daily amount at least by keeping it organized but your annual amount wouldn't change.

I've never really bothered with the amount I put in a folder, I just keep things organized.

1

In ext3 filesystems you can have approx 32000 subdirectories in a directory:

For ext2 and ext3 the hard limit is 31998 due to EXT[23]_LINK_MAX being 32000 and each of those subdirectories (folders) has a link back to its parent, which began life with a link count of 2.

The limit for decent performance would likely be quite a bit less.

Also, structuring your data to avoid hitting such limits has other benefits, for example it makes easier to split a large data store across multiple volumes or even servers.

0

At some point, you'll want to consider looking at MogileFS to store your files. Then you don't need to worry about file system limits, you get almost automatic replication, and it's fast. I can't say for sure if you are at the point with a million pictures, but it's worth looking at. May be easier, in any case, than rolling your own solution just to deal with file system limits.

0

I usually split so there's 10000 files per folder. It's fairly easy if you have have some sort of numeric identifier for your files.

As an example, if you're using a file with identifier = 9876543, you could use the following process:

  • Formats number with 12 digits (000009876543)
  • Uses the first eight digits as a folder identifier (0000/0987)
  • Save the file as (0000/0987/myfile.jpg)

While most file systems allow you to keep more files in a single folder, this makes things more organized, it'll be easier if you eventually need to split in multiple volumes, and if you're using Windows it also makes sure Explorer won't freeze trying to read 1M files. :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.