Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to copy a 200GB file from an NFS mount to a local disk. The local disk is an XFS filesystem on a LVM on top of a RAID 5 system (hardware RAID controller).

I'm using rsync to monitor the transfer speed. At the beginning, the IO speed is about 200MB/s, stable for the first 18GB. But then the performance drops by a factor of 10-20 and never recovers to the initial rate. Sometimes it reaches about 50-100MB/s, but just for a few seconds, and then the process seems to hang for a bit.

At the same time all file-stat operations on the target filesystem are blocking for a long time (minutes). Also, interrupting the copy process blocks for several minutes. A subsequent delete of the partly copied file takes also several minutes.

Any ideas what could be causing this?

share|improve this question
looks like you saturated some write disk cache on the local disk. Have you checked the speed if you copy the file local to local with say file bigger than 18GB – jet Jun 5 '12 at 22:18
up vote 0 down vote accepted

Thanks to jet's comment, I looked into the caching of disk IO under Linux. It turns out, as the system has a lot of RAM (48GB) which is almost free, a lot is used for the I/O cache.

I monitored /proc/meminfo and looked for the fields 'Dirty' (dirty I/O cache pages) and 'Writeback' (written dirty pages to disk).

watch -n1 -- "grep -E Dirty|Writeback /proc/meminfo"

Shows that Dirty increases to about 18GB, than goes down again, while Writeback increases, once Dirty pages have been written, it grows again etc. Monitoring the disk io using iostat one sees that during caching the disk does not write, only while 'Writeback' decreases data is written to the disk. The write speed is about 15MB/s.

So the first 18GB are fast, written to the cache but than, when the actual writing occurs it slows down and while writing the disk I/O is saturated and blocking until all dirty pages are send to the disk.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .