From my understanding, rsync over SMB tries to build a file list each time on source/destination to check any changes have been modified before a copy takes please.

I’m trying to understand the differences using rsync on OSX over SMB when mounting network storage and using rsync on OSX over SSH to a NFS mounted storage on Linux. During testing when rsyncing to the mounted storage on Linux, it identified new/deleted files immediately! When rsyncing over SMB builds a complete file list and creates a list of each folder before any copy takes place.

To the users perception, this building file list (providing they have hundreds/thousands of folders) takes some time to run through each directory. Once the rsync completes the initial copy, on the second run it provides me with the sent and size information, extremely quick.

The problem occurs when we unmount the SMB share. If we run rsync again it builds the file list and runs through every folder even if there is nothing to copy. Rsyncing to the storage via ssh doesn’t show a building file list and immediately informs us if files are copying etc.

The commands we have played around are;-

Over smb rsync -uvaz --delete /source /destination rsync -aHEXAx -v --delete --progress --stats --timeout=999 /source /destination

Over ssh/nfs rsync -nuvaz --delete /source/ root@nfsServer.domain.co.uk:

Any ideas?

rsync always has to build the list of files to sync, but "Over SSH" is the key difference here.

  • When you point rsync at a SSH server, it can ask the server to create such a list, by running another copy of rsync on the server.

    That way, both the client & the server only need to read / rescan their own local copies, and only exchange a short list of updates.

  • Meanwhile, when you point rsync directly at a mounted share, it cannot run anything on the server, neither over SMB nor NFS nor whatever else. It can't even distinguish mounted network shares from mounted local disks – all it sees is files.

    This means that your rsync has to rescan the entire share, which involves sending quite a lot of "read directory" commands over the net (latency) and downloading whole files just to determine the changed parts (bandwidth).

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.