1

We have hundreds of millions of small files on our server, totally about 500 GB. I'm looking for a way to speed up their creation. Would there be any difference if I get more RAM for that purpose? Say, 32 GB instead of 8 GB. I know the OS uses some RAM space as the FS cache (default 50%), but will it increase the number of writes per second?

Linux, EXT4.

Thanks!

2 Answers 2

0
  1. Organize your files if they are in the same folder, or there are many in the same folder (try to limit to 4-5.000 files/folder).
  2. Try to avoid filling HDs (have ~20% free space on disk) to avoid fragmentation
  3. RAM helps with calculations, working with big chunk of data at any one time so if more RAM helps it really depends on how you use those files... For I/O and file operations one has to focus more on FSB and HDD speed ("RPM"), as well as HDD buffer size and access speed.
2
  • thanks @Mark. So the simple answer is 'no'? If other details are the same, the files organized into subfolders, there are no big files (the biggest is 50MB) - then the extra RAM is useless, isn't it?
    – Spaceman
    Apr 1, 2014 at 11:21
  • @Spaceman indeed. It won't help, except the application you mention "creating files" is not very well written and more RAM helps it run faster. But if it is OK, and what happens really is just file/IO operation and not more, the only thing helps is what I wrote above
    – user221741
    Apr 2, 2014 at 6:41
0

Simple answer is NO, RAM will not affect speed of small files being written.

Complex answer is: DEPENDS on how much data is being written at any given time. If more than ~ 4GB? (estimate based on system taking up 4 GB) at any given moment (high I/O count) then you will benefit from the increased RAM due to the write buffer being used more intensely. http://en.wikipedia.org/wiki/Data_buffer

Computer Science: Data is written to the RAM while a write to disk is requested. File copy may finish, but the operating system will continue flushing data from RAM to disk.

This answer is written with Linux in mind.

A faster speed will be achieved in your case with:

  • SATA III or SATA II motherboard connection to SSD, HDD won't benefit much unless it is above 7200rpm (+150MB/s speed) (this is a motherboard upgrade)
  • A better CPU, thus process write I/O faster
  • SSD + at least SATA II. The most inflential factor in file write speed.

As always, any upgrade will be hindered by the weakest link in the system, keep that in mind if you purchase a SSD, but have a Intel Pentium CPU running.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .