We have a linux cluster with worker nodes sharing a common remote file system. We want to optimize the I/O, in particular when the workers read from the remote fs.
The starting point is to measure the I/O performance. The cluster is used by many user and their jobs are doing intensive reading.
I would like to monitor the performance many times during a day to see how they change (night/day/workdays/before and after hardware upgrade,...).
The question is how to measure the performance. If possible I would like to decouple the network performance from the disk performance. I want to test many different kind of reading, big files, small, sequential, random, access, ... avoid caching, ...
Do you have experience with this kind of test? Do you know some free tools, library, easy to interface with other utilities (visualization, ...)
By now the most smart thing I can do is to use