Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have an application that creates 16^5 files (1048576). I can configure it to create all them in the same folder or separate them in whatever way I want (having 1, 2, 3 or 4 subfolders on each folder). Example:

/*

or

/a/*
.
.
.
/f/*

or

/a/a/*
/a/b/*
.
.
.
/f/f/*

or

/a/a/a/*
.
.
.
/f/f/f/*

or

/a/a/a/a/*
/a/a/a/b/*
.
.
.
/f/f/f/f/*

All the files have less than 4KB. I am using Ubuntu 12.10 64bits and an ext4 partition to store this. What folder structure would be the best structure for this case? Maybe other filesystem would be best suitable for this case, any ideas?

Anyway I am trying to run this algorithm, I should be able to open 9999999 files:

user@pc$ ulimit
unlimited

user@pc$ cat /proc/sys/fs/file-max
9999999

user@pc$ cat /etc/sysctl.conf
fs.file-max = 9999999

However, when I run it saving everything in one single folder the fopen call fails around 999999 files:

user@pc$ ls database/ | wc -l
999958

Strangely this 999999 was my previous value for file-max at system files. I have of course rebooted my machine before updated the value, maybe it is too big and then it keeps the last one. What can be wrong?

share|improve this question
    
Whomever that didn't like my question, please comment so I can fix it –  Frederico Schardong Mar 2 '13 at 6:52
    
Why in the world would you need to create so many files, can't you group some of the information into a single large file? –  terdon Mar 2 '13 at 11:10
    
Try dividing so that you don't have more than 10^4-10^5 files per directory. Use reiserfs for small files. –  Ярослав Рахматуллин Mar 5 '13 at 6:53

1 Answer 1

up vote 1 down vote accepted

If you look at proc(5), /proc/sys/fs/file-max "defines a system-wide limit on the number of open files for all processes". In particular, it doesn't say that a single process can open that many files.

You may want to refer to sysconf(3), which describes OPEN_MAX as "The maximum number of files that a process can have open at any time". You can retrieve this value by running getconf OPEN_MAX.

I actually don't know offhand how large you can make OPEN_MAX, and I'm not inclined to investigate further at this hour, but feel free to experiment and report back to us.

Incidentally, I would also run ulimit -a to show all the limits. Running ulimit alone in bash implies ulimit -f, which only shows the maximum size of files written by the shell and its children.

P.S. If your application needs to hold a million files open at the same time, I would highly suggest re-evaluating your design.

share|improve this answer
    
Yes I am going to change it, thanks for your answer. –  Frederico Schardong Mar 3 '13 at 21:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.