Why does RSYNC use two reading processes with different files on my backup target?

Question

Background

I have one server hosting virtual machines and one older NAS Synology DS1512+ used as backup target for those virtual machines. The server uses ZFS, creates snapshots and transfers the files of the snapshots to the NAS. The NAS uses BTRFS with enabled compression and supports snapshots as well. the ultimate goal would be that the server really only sends DELTAs using RSYNC to minimize the amount of changed data received by the NAS and make efficient use of snapshots on that as well.

Problem

Using RSYNC with DELTAs doesn't work in my case, though, because transferring the data simply takes too much time. When RSYNC is used with --inplace --whole-file, the data takes ~2 hours to transfer. When removing --whole-file to make use of DELTAs, the same backup process takes far longer, I often killed the process after already running 12+ hours. For historical reasons I need to fit different backups into far smaller time windows.

The only bottleneck which makes sense is the NAS, because the server is far more powerful and idles around most of the time. The NAS OTOH has a pretty high load on CPU and I/O during backup. Though, the numbers aren't too bad at all as well, it's only that they are badder than when using --whole-file. With that, the NAS pretty much simply writes ~100+ MiB/s, while with DELTAs it reads slower most of the time, spanning from ~50 to 100 MiB/s. I thought that the amount of data to NOT write because of DELTAs would easily outperform the fact of the slower NAS, but that doesn't seem to be the case. And the changed amount of data on the VMs isn't too high mostly.

Observation

What I recognized on the NAS was that RSYNC seems to process two files the same time at some point. This looks like some read-ahead or alike:

root@amds1512-01:~# lsof | grep [d]asi_
rsync   6883   root  cwd    DIR   0,33        290   259 /volume1/[...]
rsync   6883   root    0r   REG   0,33 2142633984   580 /volume1/[...]/[...]-s024.vmdk
rsync   6884   root  cwd    DIR   0,33        290   259 /volume1/[...]
rsync   6884   root    1r   REG   0,33 2143748096   579 /volume1/[...]/[...]-s023.vmdk
rsync   6884   root    3w   REG   0,33 2143748096   579 /volume1/[...]/[...]-s023.vmdk

HTOP clearly shows that both instances of RSYNC do read. Just ignore the other RSYNC-processes, those are unrelated and the problem still persists even when one backup runs exclusively.

Questions

So what's the purpose of those two running RSYNCs with different files on the backup target? Is there any way to tell RSYNC to only process one file after another?

That might increase overall processing time with less concurrent load. I couldn't find anythign like read ahead or alike in the man page. If it makes any difference, the following are the used options:

--owner \
--numeric-ids \
--compress-level=0 \
--group \
--perms \
--rsh=rsh \
--devices \
--hard-links \
--inplace \
--links \
--recursive \
--times \
--delete \
--delete-during \
--delete-excluded \
--rsync-path=[...] \
--specials

Thanks!

Answer 1

Take a look at How Rsync Works. Specifically, there is a generator process and a sender process that operate as a pipeline. The sender reads the file to send to the remote. The generator is responsible for generating the list of files to send, and also "block checksums are created for the basis file and sent to the sender immediately following the file's index number."

This definitely sounds like it has the potential to cause filesystem thrashing if you're using --inplace to send multiple large files and don't have enough RAM available for the kernel to hold two consecutive files in cache.

As a test, you could trying transferring individual files with rsync --inpace and see if the performance is significantly better. (Something like for i in *.vmdk; do rsync [...]; done.) That should help determine if having two separate readers is actually causing your performance problem.

If multiple readers is causing the performance problem, then one possible route would be to improve the kernel's ability to cache the reads, either by making more RAM available to the host kernel or by making your individual vmdk files smaller.

Unfortunately I don't see any obvious way to change the generator/sender pipeline behavior in rsync, short of writing your own script to call rsync once for each file. You might want to ask about this on the rsync mailing list.