Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm currently copying a large number of files over a network. To monitor progress, I tried running watch du. However, the output never changed (or not much, not sure). find . -type f | wc -l always gives me the same number of files as does ls -R.

It seems, these programs use caching, which is, in general, a good thing. Does anyone know, though, how cache usage could be controlled?

I'm on an Archlinux system and I'm working on an ext4 fs on an encrypted hd.

Thanks

share|improve this question

migrated from stackoverflow.com Jul 11 '10 at 13:46

This question came from our site for professional and enthusiast programmers.

    
Forget the buffer cache, that's overkill. What are you using to copy, does that have a verbose or progress option? Would netstat -i give you the information that you need? –  msw Jul 11 '10 at 15:24

5 Answers 5

ls and friends do not cache their results. Instead, I suspect the output of du just changes slowly. (Due to the way your transferring software plays with open file handles). You can also try dropping the filesystem cache altogether, although I don't think it makes much sense here: echo 1 | sudo tee /proc/sys/vm/drop_caches

For a better way to track the progress, you should use rsync for transfer and give it the -P flag, which makes it print progress in standard output.

share|improve this answer

You can try the unbuffer script which comes with expect.

Here's a CVS view of the unbuffer script itself. Deceptively simple.

Another option for monitoring data transfer is pv.

share|improve this answer

The caching you are referring to is not in userspace; it's the Linux kernel inode and dentry cache. You can control the rate at which the kernel reclaims memory used in this cache by adjusting the vm.vfs_cache_pressure sysctl parameter. This article suggested a value of 10000. As for Linux cache [coherency] itself, I couldn't give you a definitive answer but it's likely not the issue.

Also, unbuffer is for buffering (unbuffered, block, or line-buffering) of the standard streams. It is likely (but not necessarily) not relevant here, unless you are passing tarballs over the net and untarring them locally, or redirecting output from a net-connected process to the filesystem. Even so, it shouldn't matter unless these programs are using block buffering with a huge size, or line-buffering for binary data and you're unlucky enough to not have had a single 0x0d or 0x0a in your data for some arbitrarily large amount of data. Also, this caching is a library feature, not a kernel or program feature. The latter may set parameters.

Netstat would also just list outstanding network connections, not whatever is going on therein.

You can try strace -e trace=fileing the process to see what it's doing.

share|improve this answer

To monitor activity you could use iotop in conjunction with bwm-ng. iotop as the name suggests monitors input/output. bwm-ng monitors network interfaces.

If you want a more 'raw' approach, just look at the numbers in /proc/net/dev

Both aforementionned tools are available on the AUR, since you mentionned you are a ArchLinux user.

share|improve this answer

Am not sure about caching.

But am pretty sure that, you wont get wrong data at any time. Kindly recheck what you are doing. In all the cases, these find and ls gave me very correct data, even while handing tons of files.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.