Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm trying to copy a 200GB file from an NFS mount to a local disk. The local disk is an XFS filesystem on a LVM on top of a RAID 5 system (hardware RAID controler).

I'm using rsync to monitor the transfer speed. At the beginning the IO speed is about 200MB/s, stable for the first 18GB. But then the performance drops by a factor of 10-20 and never recovers to the initial rate. Sometimes it reaches about 50-100MB/s but just for a few seconds and then the process seems to hang for a bit.

At the same time all file-stat operations on the target filesystem are blocking for a long time (minutes). Also interrupting the copy process blocks for several minutes, a sub-sequent delete of the partly copied file takes also several minutes.

Any ideas what could be causing this?

share|improve this question
1  
looks like you saturated some write disk cache on the local disk. Have you checked the speed if you copy the file local to local with say file bigger than 18GB –  jet Jun 5 '12 at 22:18

1 Answer 1

up vote 0 down vote accepted

Thanks to jet's comment, I looked into the caching of disk IO under Linux. It turns out, as the system has a lot of RAM (48GB) which is almost free, a lot is used for the I/O cache.

I monitored /proc/meminfo and looked for the fields 'Dirty' (dirty I/O cache pages) and 'Writeback' (written dirty pages to disk).

watch -n1 -- "grep -E Dirty|Writeback /proc/meminfo"

Shows that Dirty increases to about 18GB, than goes down again, while Writeback increases, once Dirty pages have been written, it grows again etc. Monitoring the disk io using iostat one sees that during caching the disk does not write, only while 'Writeback' decreases data is written to the disk. The write speed is about 15MB/s.

So the first 18GB are fast, written to the cache but than, when the actual writing occurs it slows down and while writing the disk I/O is saturated and blocking until all dirty pages are send to the disk.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.