I'm building the code on remote linux machines, and it takes very long time. Since all the filesystem is on NFS, I suspect the NFS is the bottleneck. Can I profile the build process (make all) down to system read/write calls? Or simply put, what tool do I need to find the bottleneck. Is strace gonna help?
Most probably strace is not going to help you in this case, as it do not provide meaningful timing information over time. You can get summaries using command
Output shows how much time were used for I/O (read and write calls), but it is hard to make difference between normal use and excessive waits.
If you can, easiest way to check whether NFS is the bottleneck is to time compiling over NFS and then on local disk, which should be faster. If that is not possible, one useful and trivial pointer is wait time, shown by for example top. From man top:
Shows that processor is mainly using time for waiting I/O to complete. In this case culprit is another virtual machine using disk heavily, but case is almost the same with waiting for NFS.
This requires monitoring top output during compilation process (at least sometimes), for example sar (man page) collects system statistics automatically.
Hopefully this helps.