I found myself in a situation I am copying an entire filesystem across two different HD partitions (different sizes and filesystem types). The problem so far is taht both cp and rsync will read a single file at the source, and write it on the destination - doing that for every small file. Therefore, I have only some small OS buffering and the HD cache itself to prevent the HD head to move back and forth for each small file - but it feels it is quite close to that.

The machine is got plenty of RAM. What could I do to say, read several megabytes (or even GB) worth of data on the same partition before comiting that on the other partition?

If it matters I am copying from BTRFS to EXt4 - data includes several git trees, source files and such (few large video and audio files).

  • Copy is over with rsync by now :-) but still, a clever way to do this would be welcome. – jsbueno Sep 14 '13 at 5:40

You might try playing with cpio or tar piping, for example using cpio's --io-size

That said, I don't believe you would get better performance than you can achieve leaving cp or rsync do their jobs. It's actually the job of the filesystem and the underlying OS to deal with hdd driver behaviour and help it decide whether to seek or not.

You might get a better insight on this with this SO question. Particularly:

EDIT: Ah, "native Linux" may be improving performance by interleaving reads and writes with asynchronous I/O. Letting commands pile up can help the disk driver decide when is best to seek. You might try Boost Asio or pthreads for comparison. As for "can't beat POSIX file descriptors"… well that's true if you're doing anything with the data, not just blindly copying.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.