I have one server hosting virtual machines and one older NAS Synology DS1512+ used as backup target for those virtual machines. The server uses ZFS, creates snapshots and transfers the files of the snapshots to the NAS. The NAS uses BTRFS with enabled compression and supports snapshots as well. the ultimate goal would be that the server really only sends DELTAs using RSYNC to minimize the amount of changed data received by the NAS and make efficient use of snapshots on that as well.
Using RSYNC with DELTAs doesn't work in my case, though, because transferring the data simply takes too much time. When RSYNC is used with
--inplace --whole-file, the data takes ~2 hours to transfer. When removing
--whole-file to make use of DELTAs, the same backup process takes far longer, I often killed the process after already running 12+ hours. For historical reasons I need to fit different backups into far smaller time windows.
The only bottleneck which makes sense is the NAS, because the server is far more powerful and idles around most of the time. The NAS OTOH has a pretty high load on CPU and I/O during backup. Though, the numbers aren't too bad at all as well, it's only that they are badder than when using
--whole-file. With that, the NAS pretty much simply writes ~100+ MiB/s, while with DELTAs it reads slower most of the time, spanning from ~50 to 100 MiB/s. I thought that the amount of data to NOT write because of DELTAs would easily outperform the fact of the slower NAS, but that doesn't seem to be the case. And the changed amount of data on the VMs isn't too high mostly.
What I recognized on the NAS was that RSYNC seems to process two files the same time at some point. This looks like some read-ahead or alike:
root@amds1512-01:~# lsof | grep [d]asi_ rsync 6883 root cwd DIR 0,33 290 259 /volume1/[...] rsync 6883 root 0r REG 0,33 2142633984 580 /volume1/[...]/[...]-s024.vmdk rsync 6884 root cwd DIR 0,33 290 259 /volume1/[...] rsync 6884 root 1r REG 0,33 2143748096 579 /volume1/[...]/[...]-s023.vmdk rsync 6884 root 3w REG 0,33 2143748096 579 /volume1/[...]/[...]-s023.vmdk
HTOP clearly shows that both instances of RSYNC do read. Just ignore the other RSYNC-processes, those are unrelated and the problem still persists even when one backup runs exclusively.
So what's the purpose of those two running RSYNCs with different files on the backup target? Is there any way to tell RSYNC to only process one file after another?
That might increase overall processing time with less concurrent load. I couldn't find anythign like read ahead or alike in the man page. If it makes any difference, the following are the used options:
--owner \ --numeric-ids \ --compress-level=0 \ --group \ --perms \ --rsh=rsh \ --devices \ --hard-links \ --inplace \ --links \ --recursive \ --times \ --delete \ --delete-during \ --delete-excluded \ --rsync-path=[...] \ --specials