rsync on Windows not transferring large files

Question

I'm noticing a strange issue where rsync isn't transferring large (approx >3GB) files. I'm running MSYS rsync and I'm transferring on my LAN from my Windows box to my Linux based NAS over ssh. Most of the files transfer without issue, but with large files I see the file name on the destination, but it is listed as 0KB in size and the files won't open.

I don't see any errors when I enable verbose output except a note about a couple long file paths for unrelated files. Here's my command (added line breaks for legibility):

rsync -avv -e 'ssh' --hard-links --inplace --no-inc-recursive
    --modify-window=2 --delete --delete-excluded --exclude=".svn*"
    "/d/All Files" user@local_ip:"/mnt/All Files"

Has anyone seen something like this before? What can I do to debug this?

Update: Here's the verbose output from rsync for the transfer with --progress and --stats enabled. The file "Uncompressed 1080 vs 720.avi" is the problem file. It is almost 7GB. The strange thing to me is the negative file size rsync is reporting. What could be causing that?

building file list ...
3 files to consider
delta-transmission enabled
Uncompressed 1080 24p vs 24pa 29 97.avi is uptodate
Uncompressed 1080 vs 720.avi
 -1546369996 100%    0.00kB/s    0:00:00 (xfer#1, to-check=0/3)
total: matches=0  hash_hits=0  false_alarms=0 data=-1546369996

Number of files: 3
Number of files transferred: 1
Total file size: 4868647526 bytes
Total transferred file size: 2748597300 bytes
Literal data: -1546369996 bytes
Matched data: 0 bytes
File list size: 124
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 180
Total bytes received: 37

sent 180 bytes  received 37 bytes  39.45 bytes/sec
total size is 4868647526  speedup is 22436163.71

Answer 1

It sounds like rsync is using a signed 32 bit integer for storing the file size, and your file is so large that the value looks negative.

If you are on a 64bit box, see if you can find a 64bit version of rsync. If not, try other rsync implementations (the two that come to mind are DeltaCopy and cwRsync). I suspect all these rsync implementations are just ports of the same code, but it's worth a try. The company that provides DeltaCopy has a supported commercial product that might solve your problem.

There are lots of file copy programs out there, both free and commercial, so there has to be one that can solve your problem. One example is SyncBack (available in free and commercial versions).

Answer 2

I used to have this problem all the time. I thought they might have fixed it by now.

The problem used to be that rsync ran out of memory when trying to deal with large files. I gave up using it some years ago because of that and switched to other backup/sync tools instead.

Not sure of the status of rsync for Windows, it might be worth trying to source an alternative binary?