Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm working on a bit of software for which it would be useful for us to write betweeen 100 and 200 files a second to disk. The files vary in size from 1k to 3k. Ubuntu is the operating system, using ext4 file system.

So far I have seen some odd behaviour. At first the inodes ran out, causing us to get "out of disk space" errors. But to combat this we've moved to a large partition and massively increased the number of inodes.

However, I am still getting "No space left on device" errors. On different machines. With df -i and df -h reporting plenty of free space and plenty of free inodes.

From speaking to people I've had answers varying from "it should be fine" to "that's far too many files to be writing to disk a second". I've experienced this behaviour on 3 different Ubuntu machines.

Is there a definitive answer to this? For this many writes should I just be using a database or should the operating system be able to handle this?

share|improve this question
1  
What are you doing? –  Aaron Miller Aug 10 '13 at 8:58
    
Caching messages from from a data feed for debugging purposes –  Andy Smith Aug 10 '13 at 10:49
2  
@AndySmith Why does that necessarily involve writing several hundred files per second to disk? Make one big one; keep track of its size and rotate at some set limit if you don't want it to grow too much. After all, you're talking about at most 600 KB written per second. Write an index at the end if you want quick lookups, or just write some marker between each record and do a sequential scan/split if you can. –  Michael Kjörling Aug 10 '13 at 15:13
    
We needed to break up the messages and index them on various keys, individual file names with the keys in the names seemed like a good idea –  Andy Smith Aug 10 '13 at 17:37
2  
You should indeed use a DB instead of 200 create/write/close calls per second. –  ott-- Aug 10 '13 at 17:55

1 Answer 1

There might be a problem with number of open file descriptors; you can check your system's current value via ulimit -n.

If that's not enough, increase it by executing ulimit -n 50000 (for example).

share|improve this answer
    
Sounds about right! Cheers, i'll try it out –  Andy Smith Aug 10 '13 at 17:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.