Is there a way to file the cause (or target file) of disk write operations in linux (RHEL6.1)?

Question

I have a virtual machine with a php based application (in apache) on it, when looking at disk activity there is a constant 170kbps write when apache is running.

When we apply load then the performance of the server is cpu bound but this seems to be directly related to disk write IO jumps of 20-30x. The CPU usage graph mirrors the write disk access graph.

Any ideas?

Answer 1

Two recommendations:

iotop is a handy program that shows you real-time I/O usage in a format like the top command. This should help you identify the script/binary thats doing all the I/O. I think this may already be installed on newer kernels, but yum may provide it on RHEL if you don't see it.

lsof is also helpful with I/O troubleshooting; it lists any open files.

UPDATE

If there is "constant" disk activity, then iotop should tell you which process(es) are the culprits. Regarding I/O transaction history: I don't believe IO history is logged anywhere in most Linux distros, but you can add a cron job.

As root, open up crontab with crontab -e and add these two lines:

* * * * * /bin/date >> /var/log/iotop; /usr/local/bin/iotop -bot --iter=3 -q >> /var/log/iotop
* * * * * /bin/date >> /var/log/lsof; /usr/bin/lsof -b -w >> /var/log/lsof

That dumps iotop and lsof output to a log every minute. Add a logrotate script for each to keep the files from taking up all your space, e.g.:

someuser@myhost:~> cat /etc/logrotate.d/lsof
/var/log/lsof {
  rotate 3
  weekly
  compress
  missingok
  notifempty
}

Answer 2

It appears that an unrelated background task was deleting the contents of the /tmp directory where php was being pre-compiled to. As a result the php pre-compile process started again resulting in constant disk activity.