28

I have a big iso image which is currently being downloaded by a torrent client with space-reservation turned on: that means, file size is not changing while some chunks in in (4 Mib) are constantly changing because of a download.

At 90% download I do the initial rsync to save time later:

$ rsync -Ph DVD.iso /media/another-hdd/
sending incremental file list

DVD.iso
       2.60G 100%   40.23MB/s    0:01:01 (xfer#1, to-check=0/1)

sent 2.60G  bytes  received 73 bytes  34.59M bytes/sec
total size is 2.60G   speedup is 1.00

Then, when the file's fully downloaded, I rsync again:

total size is 2.60G   speedup is 1.00

Speedup=1 says delta-transfer was not used, although 90% of the file has not changed, target dir is on another FS and copying takes several minutes. Why doen't it try to speedup the transfer?! How can I force rsync to use delta-transfer?

Improve this question
4
  • 6
    What you are doing does not make any sense. The purpose of rsync is to speed up transferring files over the network, not locally. In order to find the differences, it has to read both the source and destination. In the time it takes to read the destination locally to find the differences, you may as well just do a normal copy. Just download the file to the destination in the first place instead of copying it.
    – psusi
    Jan 17, 2011 at 15:35
  • 1
    So it just does not use delta-xfer because, working locally, it's faster to copy than to calculate hashes? If yes — post the answer plz :)
    – kolypto
    Jan 17, 2011 at 16:37
  • 13
    Reading can be faster than writing to a local disk in certain circumstances. It can also reduce the wear on an SSD. This is certainly a valid question and the answer is quite valuable to me.
    – HRJ
    May 8, 2013 at 11:56
  • 2
    @psusi apart from HRJ's comment above, also consider the case when the target file has been reflinked (eg on btrfs or ocfs2). Minimizing writes during the sync can make an enormous difference to the overall space usage.
    – user84636
    Aug 23, 2013 at 21:13

4 Answers 4

Reset to default
23

According to the manpage, psusi is right:

-W, --whole-file: The transfer may be faster if this option is used when the bandwidth between the source and destination machines is higher than the bandwidth to disk (especially when the "disk" is actually a networked filesystem). This is the default when both the source and destination are specified as local paths, but only if no batch-writing option is in effect.

Improve this answer
4
  • 12
    Oh, thank you! I miseed this line :) To turn delta-trasfer on, use -no-W
    – kolypto
    Jan 18, 2011 at 12:35
  • 1
    On my system -no-W doesn't work only the long option -no-whole-file. My reason for needing this switch is when I'm setting up a backup and have large files (eg images) that don't have the same modification time. It is MUCH faster, speedup is 163.26, to sync these files using the delta-transfer on my local filesystem.
    – Jesse the Wind Wanderer
    Sep 21, 2015 at 17:05
  • 9
    @JessetheWindWanderer, the long option is --no-whole-file (please, note the double -- at the beginning).
    – Eddie C.
    Dec 31, 2016 at 12:42
  • Thanks Eddie C. I'd edit my comment if I could figure our how :-(
    – Jesse the Wind Wanderer
    Jan 1, 2017 at 16:00
19

The straight-forward answer to this question is:

Use the --no-W flag to force delta compression, no matter local or remote.

Update: It looks like there is more to the story. The delta compression seems to be enabled only between receive and transmit process of rsync. When outputting the file to the file-system, rsync may still write out the whole file(s), even with delta compression on.

See "Wakan Tanka's" investigation here.

Update 2: The --inplace option writes only the changed parts of the file. However note that it conflicts with --sparse and is not recommended, by the manual, when the destination file is being used.

Improve this answer
2
5

By default, rsync first creates a new copy of the target file and then replaces it, for various safety reasons. You can override this by specifying --inplace along with --no-whole-file. This tells rsync to do an in-place-edit of the target file, accepting the various risks (typically minor for this situation) as documented in the man page.

Improve this answer
2

By default rsync creates a copy of the file at the destination and then atomically replaces the original with the new copy. This is done for safety reasons. What you're looking for is the --inplace option, which will cause rsync to modify only the portions of the destination file which have changed relative to the source.

For the O.P's use-case, I recommend turning off pre-allocation as well, so that a sparse copy can be synced, which will be much faster. For downloads, don't worry about fragmentation unless you're using a very ancient filesystem like VFAT. Media files in particular are not read at the maximum performance of the storage media, so defragmenting them is a wasted effort.

To copy your downloads directory sparsely to the destination volume, I recommend these flags and operations, in this order:

rsync --ignore-existing -vxaHAXS /source /destination
rsync --inplace -vxaHAX /source /destination

The first pass will copy new files sparsely to the destination The second pass will update existing files in-place, copying only the changes

Since it's doing sparse and in-place delta copies, you can run this repeatedly without incurring much extra IO. Even if you have 20 torrents running simultaneously it's won't amplify the writes at the destination, or thrash the source/dest volumes.

Improve this answer
2
  • What do you mean by "sparsely" here, Wil ? It doesn't really reflect the actual meaning of the word, as far as I can tell.
    – Julius
    Oct 6, 2019 at 15:12
  • @Julius: it means exactly what it implies - copy the files with full support for sparse allocation, so for example your 40GB HDR movies will not take up any more space at the destination than they do at the source. Same with VirtualBox disk images. As stated the O.P. would need to disable pre-allocation for that to work.
    – Wil
    Nov 18, 2019 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.