3

I'm currently copying a large number of files over a network. To monitor progress, I tried running watch du. However, the output never changed (or not much, not sure). find . -type f | wc -l always gives me the same number of files as does ls -R.

It seems, these programs use caching, which is, in general, a good thing. Does anyone know, though, how cache usage could be controlled?

I'm on an Archlinux system and I'm working on an ext4 fs on an encrypted hd.

Thanks

Improve this question
1
  • Forget the buffer cache, that's overkill. What are you using to copy, does that have a verbose or progress option? Would netstat -i give you the information that you need?
    – msw
    Commented Jul 11, 2010 at 15:24

6 Answers 6

Reset to default
3

The caching you are referring to is not in userspace; it's the Linux kernel inode and dentry cache. You can control the rate at which the kernel reclaims memory used in this cache by adjusting the vm.vfs_cache_pressure sysctl parameter. This article suggested a value of 10000. As for Linux cache [coherency] itself, I couldn't give you a definitive answer but it's likely not the issue.

Also, unbuffer is for buffering (unbuffered, block, or line-buffering) of the standard streams. It is likely (but not necessarily) not relevant here, unless you are passing tarballs over the net and untarring them locally, or redirecting output from a net-connected process to the filesystem. Even so, it shouldn't matter unless these programs are using block buffering with a huge size, or line-buffering for binary data and you're unlucky enough to not have had a single 0x0d or 0x0a in your data for some arbitrarily large amount of data. Also, this caching is a library feature, not a kernel or program feature. The latter may set parameters.

Netstat would also just list outstanding network connections, not whatever is going on therein.

You can try strace -e trace=fileing the process to see what it's doing.

Improve this answer
2

ls and friends do not cache their results. Instead, I suspect the output of du just changes slowly. (Due to the way your transferring software plays with open file handles). You can also try dropping the filesystem cache altogether, although I don't think it makes much sense here: echo 1 | sudo tee /proc/sys/vm/drop_caches

For a better way to track the progress, you should use rsync for transfer and give it the -P flag, which makes it print progress in standard output.

Improve this answer
2
  • I have made different oberservations than "ls and friends do not cache their results". It's seven years later now, but at this time and in this broadness your statement isn't true anymore. hoov@floyd:~# time find > /dev/null real 0m8.359s user 0m0.079s sys 0m0.430s hoov@floyd:~# time find > /dev/null real 0m0.095s user 0m0.029s sys 0m0.065s
    – Hoov
    Commented Aug 3, 2017 at 13:42
  • @Hoov That doesn't prove that any results were cached and, in fact, they aren't. It just proves the primary data source was easier to access later.
    – David Schwartz
    Commented May 24, 2018 at 15:56
1

You can try the unbuffer script which comes with expect.

Here's a CVS view of the unbuffer script itself. Deceptively simple.

Another option for monitoring data transfer is pv.

Improve this answer
0

To monitor activity you could use iotop in conjunction with bwm-ng. iotop as the name suggests monitors input/output. bwm-ng monitors network interfaces.

If you want a more 'raw' approach, just look at the numbers in /proc/net/dev

Both aforementionned tools are available on the AUR, since you mentionned you are a ArchLinux user.

Improve this answer
0

(Context) I am using a Docker / SilverStripe environment and I am frequently running into this exception when trying to call http://example.com/dev/build/?flush=all (which rebuilds the database and clears the cache).

rm: cannot remove '.nfs00000001004173880000002c': Device or resource busy

I learned that I can navigate to that directory in the terminal (outside Docker) and delete the file manually. But doing this repeatedly leads me to this same caching issue where ls -hal will show me an empty directory even though the file exists (and cannot be deleted).

Long story short: in my case, I can use cd . to update the terminal … or delete its cache or whatever.

Improve this answer
-1

Am not sure about caching.

But am pretty sure that, you wont get wrong data at any time. Kindly recheck what you are doing. In all the cases, these find and ls gave me very correct data, even while handing tons of files.

Improve this answer

You must log in to answer this question.