4

I'm backing up files to a remote webdav share. I'm using rsync so that it's at least incremental. However it's not going very fast at all, and so I'm wondering if I've configured it correctly.

Will the following command do an incremental backup if run multiple times?

# Backup
sudo rsync -av --delete --progress \
--exclude '*@SynoResource' --exclude '@eaDir' \
--exclude '.TemporaryItems' \
--exclude '*.vsmeta' --exclude '.DS_Store' \
/mnt/configs/ /mnt/remote/configs

If it does, how can I then speed it up otherwise?

1

Yes, I believe your command will achieve what you're aiming for.

As you are using both verbose mode and the progress option can you give some indication of where the delay is occurring? If the delay is before the files start transferring this is likely down to all files being indexed for comparison, particularly if there are lots of files or the files are large in size.

If the lack of speed is during actual transfers then that could be a lot of things; bandwidth is the obvious one, syncing big changes to big files is another one. Once the command has completed it should give you an idea of how much data has actually been transferred compared to the total size of the destination files.

Note that some file types (particularly compressed files like jpegs and MP3s) don't lend themselves very well to syncing as a small change to the file will change the layout of the entire file meaning almost the entire file would need to be synced again.

A tip for improving your sync speed, depending on the nature of your data and the kinds of changes, would be to use the fuzzy option (-y) in conjunction with --delete-after. Then, if you rename or copy a file, rather than that file having to be synced in its entirety (as a new file), rsync will use the file already present on the destination to create the new file.

Rsync does not work perfectly with some file systems. For example, mtimes on FAT32 are only accurate to +/-2 seconds so rsync could be triggered into syncing a file that is not actually newer than the source, even though it appears to be. On occasions, you can use the --size and --modify-window arguments to make rsync ignore discrepancies between the file modification times.

4
  • The delay doesn't occur before, it's more that the handling of each file just isn't that quick. With this command it should initially only be comparing the last-modified date right? I just thought that that would be extremely fast. But it takes along a couple of seconds per file (even for files < 1MB). Which seems long to me.
    – user703165
    Mar 5 '17 at 17:20
  • Ah, just realised it's using WebDAV so I would assume every file is being 100% synced each time. Again, that should be obvious in the final stats. You might want to look up --size-only and --modify-window to make it only compare basic file attributes for comparison.
    – Darren
    Mar 5 '17 at 17:31
  • Ok, --size-only is a lot faster. Seems like that's doing what I want. Thanks!
    – user703165
    Mar 6 '17 at 13:06
  • 1
    OK. I added a bit more to my answer to cover it.
    – Darren
    Mar 6 '17 at 13:12
1

In this command both source and target are "local" to rsync. It only gets really fast if the target is accessed via rsync or ssh protocol e.g.

rsync -av /my/local/dir root@somehost:/my/remote/dir

In this case rsync will spawn an rsync process on somehost and will access the files in the target dir locally (fast, as not via network).

In your configuration rsync will read the files from configs locally and will read the files in /mnt/remote/configs via network. In order to compare the contents it will have to read all files and hash the contents blockwise so there is no real speed benefit.

Does that help?

2
  • So the command you suggested should be a lot faster? The remote location only supports webdav, will the command you supplied work with that as well?
    – user703165
    Mar 5 '17 at 17:25
  • No, for fast rsync via network, the target has to support rsync.
    – smakks
    Mar 10 '17 at 2:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy