1

I'm noticing a strange issue where rsync isn't transferring large (approx >3GB) files. I'm running MSYS rsync and I'm transferring on my LAN from my Windows box to my Linux based NAS over ssh. Most of the files transfer without issue, but with large files I see the file name on the destination, but it is listed as 0KB in size and the files won't open.

I don't see any errors when I enable verbose output except a note about a couple long file paths for unrelated files. Here's my command (added line breaks for legibility):

rsync -avv -e 'ssh' --hard-links --inplace --no-inc-recursive
    --modify-window=2 --delete --delete-excluded --exclude=".svn*"
    "/d/All Files" user@local_ip:"/mnt/All Files"

Has anyone seen something like this before? What can I do to debug this?

Update: Here's the verbose output from rsync for the transfer with --progress and --stats enabled. The file "Uncompressed 1080 vs 720.avi" is the problem file. It is almost 7GB. The strange thing to me is the negative file size rsync is reporting. What could be causing that?

building file list ...
3 files to consider
delta-transmission enabled
Uncompressed 1080 24p vs 24pa 29 97.avi is uptodate
Uncompressed 1080 vs 720.avi
 -1546369996 100%    0.00kB/s    0:00:00 (xfer#1, to-check=0/3)
total: matches=0  hash_hits=0  false_alarms=0 data=-1546369996

Number of files: 3
Number of files transferred: 1
Total file size: 4868647526 bytes
Total transferred file size: 2748597300 bytes
Literal data: -1546369996 bytes
Matched data: 0 bytes
File list size: 124
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 180
Total bytes received: 37

sent 180 bytes  received 37 bytes  39.45 bytes/sec
total size is 4868647526  speedup is 22436163.71
2
  • Try it without '--inplace' as a test
    – Ram
    Commented Mar 12, 2014 at 22:18
  • @Ram, thanks for the suggestion. I tried that as well as omitting --no-inc-recursive with no discernible effect.
    – Dominic P
    Commented Mar 12, 2014 at 22:23

2 Answers 2

3

It sounds like rsync is using a signed 32 bit integer for storing the file size, and your file is so large that the value looks negative.

If you are on a 64bit box, see if you can find a 64bit version of rsync. If not, try other rsync implementations (the two that come to mind are DeltaCopy and cwRsync). I suspect all these rsync implementations are just ports of the same code, but it's worth a try. The company that provides DeltaCopy has a supported commercial product that might solve your problem.

There are lots of file copy programs out there, both free and commercial, so there has to be one that can solve your problem. One example is SyncBack (available in free and commercial versions).

2
  • After reading your answer, I finally bit the bullet and got rid of MSYS in favor of Cygwin. The rsync binary provided with Cygwin seems to be handling my test large file without issue. Thanks for pointing me in the right direction.
    – Dominic P
    Commented Mar 12, 2014 at 22:54
  • Useful to know that Cygwin may have cracked that issue. Unfortunately, it can be a bit slow to start up. I use Cygwin for some Photo processing scripts I use for auto renaming based on meta-data. Commented Mar 12, 2014 at 23:36
1

I used to have this problem all the time. I thought they might have fixed it by now.

The problem used to be that rsync ran out of memory when trying to deal with large files. I gave up using it some years ago because of that and switched to other backup/sync tools instead.

Not sure of the status of rsync for Windows, it might be worth trying to source an alternative binary?

3
  • Hmm, not the answer I was hoping to hear. :) I used to use Robocopy and I liked it, but it can't preserve hard links so I'm trying to use rsync. I'm also experimenting with ln.exe, but I can't seem to make that work. Any other suggestions?
    – Dominic P
    Commented Mar 12, 2014 at 20:16
  • Personally, I use LinkShellExtension for handling links from Explorer. It handles Hard/Soft links and junctions nicely. I used to us SyncBack for backup and sync of files and folders (there is a free version though I used the paid version). Though I stopped using it since I moved to CrashPlan for backups and Dropbox/OneDrive/Copy.com for cloud file sync. About to get a Synology NAS so I might start using it again or will use the Synology tools. Commented Mar 12, 2014 at 23:34
  • If I'm not mistaken, LinkShellExtension is essentially a GUI for ln.exe, which seems to be a great tool. But for some reason, it's painful slow when copying to my NAS. I blame the samba overhead, but who knows. I've used it on local drives and it was great. Thanks for the help and the options.
    – Dominic P
    Commented Mar 13, 2014 at 0:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .