Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have an application that creates 16^5 files (1048576). I can configure it to create all them in the same folder or separate them in whatever way I want (having 1, 2, 3 or 4 subfolders on each folder). Example:










All the files have less than 4KB. I am using Ubuntu 12.10 64bits and an ext4 partition to store this. What folder structure would be the best structure for this case? Maybe other filesystem would be best suitable for this case, any ideas?

Anyway I am trying to run this algorithm, I should be able to open 9999999 files:

user@pc$ ulimit

user@pc$ cat /proc/sys/fs/file-max

user@pc$ cat /etc/sysctl.conf
fs.file-max = 9999999

However, when I run it saving everything in one single folder the fopen call fails around 999999 files:

user@pc$ ls database/ | wc -l

Strangely this 999999 was my previous value for file-max at system files. I have of course rebooted my machine before updated the value, maybe it is too big and then it keeps the last one. What can be wrong?

share|improve this question
Whomever that didn't like my question, please comment so I can fix it – Frederico Schardong Mar 2 '13 at 6:52
Why in the world would you need to create so many files, can't you group some of the information into a single large file? – terdon Mar 2 '13 at 11:10
Try dividing so that you don't have more than 10^4-10^5 files per directory. Use reiserfs for small files. – Ярослав Рахматуллин Mar 5 '13 at 6:53
up vote 1 down vote accepted

If you look at proc(5), /proc/sys/fs/file-max "defines a system-wide limit on the number of open files for all processes". In particular, it doesn't say that a single process can open that many files.

You may want to refer to sysconf(3), which describes OPEN_MAX as "The maximum number of files that a process can have open at any time". You can retrieve this value by running getconf OPEN_MAX.

I actually don't know offhand how large you can make OPEN_MAX, and I'm not inclined to investigate further at this hour, but feel free to experiment and report back to us.

Incidentally, I would also run ulimit -a to show all the limits. Running ulimit alone in bash implies ulimit -f, which only shows the maximum size of files written by the shell and its children.

P.S. If your application needs to hold a million files open at the same time, I would highly suggest re-evaluating your design.

share|improve this answer
Yes I am going to change it, thanks for your answer. – Frederico Schardong Mar 3 '13 at 21:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .