Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a program that is very heavily hitting the file system, reading and writing randomly to a set of working files. The files total several gigabytes in size, but I can spare the RAM to keep them all mostly in memory. The machines this program runs on are typically Ubuntu Linux boxes.

Is there a way to configure the file system to have a very very large cache, and even to cache writes so they hit the disk later? I understand the issues with power loss or such, and am prepared to accept that. Crashing aside, in normal operation the writes should eventually reach the disk!

Or is there a way to create a RAM disk that writes-through to real disk?

share|improve this question

7 Answers 7

Are you seeing high numbers of IO waits, indicating that the read and write requests are not being satisfied via existing buffers? As others have noted, Linux is very good about giving spare RAM to buffers, so you should check this first.

If you're not seeing IO waits, then it's possible that your performance problems (do you even have problems? your question doesn't say) are due to kernel context switches from lots of small program-initiated IO operations. In this case you can gain a significant performance boost by rewriting your application to use memory-mapped files. But that's more of a question for StackOverflow.

share|improve this answer
    
hmm ... last answer before mine was 3 months ago, OP hasn't been seen since posting the question, no other obvious activity, yet it appeared on the front page ... I guess the system is trying to get answers to questions –  Anon May 20 '10 at 21:01
    
That is precisely what user account Community is for - bumping unanswered questions. stackoverflow.com/users/-1/community –  Corey May 20 '10 at 22:41
    
:shrug: the OP is long gone, and nobody else seems to care –  Anon May 21 '10 at 13:19
    
I do care, and I'm back; it will take me a while to digest your question and answer it fairly, please be patient –  Will May 27 '10 at 13:07

Consider using an ext4 filesystem using fast-and-loose mount options:

noatime,data=writeback,nobh,barrier=0,commit=300

to postpone writing data from cache back to physical disk.

Other than that, you could use aufs to union-mount a tmpfs-filesystem on top of your regular filesystem, do all the writing and then merge the tmpfs back down to the real filesystem afterwards.

share|improve this answer
2  
meaning of the options: mjmwired.net/kernel/Documentation/filesystems/ext3.txt –  netvope Feb 12 '11 at 5:15

Linux by default uses any spare RAM as a file cache, so no configuration is necessary for that.

You may want to consider using ext4 as the filesystem. It uses quite a number of techniques to speed up disk access, including delayed allocation which:

This has the effect of batching together allocations into larger runs. Such delayed processing reduces CPU usage, and tends to reduce disk fragmentation, especially for files which grow slowly. It can also help in keeping allocations contiguous when there are several files growing at the same time.

Data loss is pretty rare due to the use of journaling.

Ext4 is now the default filesystem in recent releases of Linux, though you will probably want to make sure the kernel you use is at least 2.6.30

share|improve this answer

Use mmap on those files - it uses the kernel's superior caching, making it much faster.

share|improve this answer

You can see my answer here: Reserve RAM for cache and buffer. If you want to reserve memory for cache and buffer: echo 10 > /proc/sys/vm/vfs_cache_pressure when 100 is the default value. Then you can limit the max ram used by each app: echo 8192 > /proc/sys/vm/max_map_count.

share|improve this answer

Linux uses your RAM very efficiently.
I don't think you really need to work on getting things cached with large files.
To check this out for yourself, try these things.

The Linux disk cache is very unobtrusive. It uses spare memory to greatly increase disk access speeds, and without taking any memory away from applications.

Check how much RAM your system has and if you can scale it up.
There is no need for a RAM-disk for this purpose,
the disk cache will work well with all available RAM space.

share|improve this answer
    
I don't think this is about Linux using RAM efficiently but rather about disk caching, like a Linux equivalent to FlashFire: flashfire.org/xe –  Molly7244 Feb 17 '10 at 17:56

RapidCache provides a RAM disk with write-through, but is currently "in beta".

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.