I'm trying to copy a 200GB file from an NFS mount to a local disk. The local disk is an XFS filesystem on a LVM on top of a RAID 5 system (hardware RAID controller).
rsync to monitor the transfer speed. At the beginning, the IO speed is about 200MB/s, stable for the first 18GB. But then the performance drops by a factor of 10-20 and never recovers to the initial rate. Sometimes it reaches about 50-100MB/s, but just for a few seconds, and then the process seems to hang for a bit.
At the same time all file-stat operations on the target filesystem are blocking for a long time (minutes). Also, interrupting the copy process blocks for several minutes. A subsequent delete of the partly copied file takes also several minutes.
Any ideas what could be causing this?