No performance increase on I/O intensive operation after symlink to ramdisk

Question

We have a system that transforms a set of several thousand multi-MB files. After every step in the transform, it writes the intermediate file to disk. The concept is similar to a scratch disk.

We have high-performance SAS drives running in RAID 6, but as neither the RAM nor CPU take much of a beating, I suspect one of our bottlenecks is in fact writing the intermediate files to the scratch directory.

We have a lot of RAM on these systems, and RHEL comes out of the box with a ram drive /dev/shm. So as a quick test, I removed the scratch directory, and symlinked it to a directory in the ram drive. The performance was the same or slightly worse? How?

Does going to disk to get the symlink somehow negate the performance benefit of the ramdisk?

I tested this in some scenarios where I know that we touched more than 10GB of data (the max of the java vm) and so it shouldn't be caching.

What am I missing here?

Answer 1

What you're missing is that file writes are almost always non-blocking. So long as the OS has sufficient memory, a file write will just write to memory. The OS will flush those writes to disk as it has the resources to do so. Since you have lots of RAM, are writing small files, and have fast disk, your application probably almost never has to wait for writes to complete.

As a complete guess, I'd think that you're actually application architecture limited. You say the CPU doesn't take much of a beating, but that likely means that your application is too crude to take advantage of much of your CPU's resources. For example, if your CPU has 8 cores but your application architecture is single-process, single-thread, it can only use one core.