How do you prevent find, du, ls -R from caching?

Question

I'm currently copying a large number of files over a network. To monitor progress, I tried running watch du. However, the output never changed (or not much, not sure). find . -type f | wc -l always gives me the same number of files as does ls -R.

It seems, these programs use caching, which is, in general, a good thing. Does anyone know, though, how cache usage could be controlled?

I'm on an Archlinux system and I'm working on an ext4 fs on an encrypted hd.

Thanks

Answer 1

ls and friends do not cache their results. Instead, I suspect the output of du just changes slowly. (Due to the way your transferring software plays with open file handles). You can also try dropping the filesystem cache altogether, although I don't think it makes much sense here: echo 1 | sudo tee /proc/sys/vm/drop_caches

For a better way to track the progress, you should use rsync for transfer and give it the -P flag, which makes it print progress in standard output.

Answer 2

You can try the unbuffer script which comes with expect.

Here's a CVS view of the unbuffer script itself. Deceptively simple.

Another option for monitoring data transfer is pv.

Answer 3

To monitor activity you could use iotop in conjunction with bwm-ng. iotop as the name suggests monitors input/output. bwm-ng monitors network interfaces.

If you want a more 'raw' approach, just look at the numbers in /proc/net/dev

Both aforementionned tools are available on the AUR, since you mentionned you are a ArchLinux user.

Answer 4

Am not sure about caching.

But am pretty sure that, you wont get wrong data at any time. Kindly recheck what you are doing. In all the cases, these find and ls gave me very correct data, even while handing tons of files.