I'm using rsync to do a backup to a NAS on my local network over SSH. However when running rsync I'm finding that it's getting stuck on certain files. Rsync will completely freeze and refuse to transfer any further files. I then have to force a SIGKILL which causes the whole rsync job to restart and get stuck on the same file that caused it to hung last time I ran it.
I've tried various fixes but so far none have worked. I originally thought its happening because of some illegal character issue between my local system (OS X 10.11.3 with OS Extended FS and my NAS running Ubuntu Linux 14.04.1 with ext4 drive for the backup). I've noticed when rsync gets stuck on a file it usually has, the filename or path usually has a '&' in it 9/10 times.
However after watching the rsync processes with
htop on the server it looks like rsync (most times but not all) crashes around the same point that the rsync file from the client hangs. I have noticed though that even when the rsync hangs on the client side, I still get output appearing in
lsof showing that the files on the server side are being accessed.
This is the rsync command that I’m using.
/usr/bin/rsync --bwlimit=1000 --verbose --rsync-path="sudo rsync" --archive --recursive --numeric-ids --human-readable --partial --progress --relative --itemize-changes --stats --files-from=/Users/user/Dropbox/Flex/Scripts/mac/rysnc-backup-to-cp/config/backup_files --exclude-from=/Users/user/Dropbox/Flex/Scripts/mac/rysnc-backup-to-cp/config/exclude -e "ssh -q -p 22 -i /Users/enwhat/.ssh/user" / email@example.com:/media/Backup/_Backup/Machine/
Example of where rsync will typically get stuck:
17.33M 63% 994.25kB/s 0:00:10
<f+++++++ Volumes/Data/Users/user1/Documents/docs/Work/_Sort from USB backup drive/Drive/JOB/CD Album/AAA1834__Album&flyer_15_Years/2-Design/1-D-Visuals/stage 05/AAA_album_12_c.psd
96.40M 50% 1.55MB/s 0:01:00
I’ve tried removing
--verbose --rsync-path="sudo rsync” --delete-during all individually. When I remove these argument flags the rsync process will get to a given file and then hang.
Is there something else at play here or is it quite likely that an illegal character in the filename is causing an issue between FS types?
I did think that Crashplan, which is running on the sever may have been eating up too much resource and causing rsync to crash. But when I stop the CrashPlan service on the server, resources do free up but rsync still crashes on the same files. This is a side note and a outside the scope of the question, but I do wonder if I should ditch Crashplan and switch to Amazon Glacier as a backup service as Crashplan sucks up a lot of CPU and memory.