I'm using rsync to do a backup to a NAS on my local network over SSH. However when running rsync I'm finding that it's getting stuck on certain files. Rsync will completely freeze and refuse to transfer any further files. I then have to force a SIGKILL which causes the whole rsync job to restart and get stuck on the same file that caused it to hung last time I ran it.

I've tried various fixes but so far none have worked. I originally thought its happening because of some illegal character issue between my local system (OS X 10.11.3 with OS Extended FS and my NAS running Ubuntu Linux 14.04.1 with ext4 drive for the backup). I've noticed when rsync gets stuck on a file it usually has, the filename or path usually has a '&' in it 9/10 times.

However after watching the rsync processes with lsof and htop on the server it looks like rsync (most times but not all) crashes around the same point that the rsync file from the client hangs. I have noticed though that even when the rsync hangs on the client side, I still get output appearing in lsof showing that the files on the server side are being accessed.

This is the rsync command that I’m using.

/usr/bin/rsync --bwlimit=1000 --verbose --rsync-path="sudo rsync" --archive --recursive --numeric-ids --human-readable --partial --progress --relative --itemize-changes --stats --files-from=/Users/user/Dropbox/Flex/Scripts/mac/rysnc-backup-to-cp/config/backup_files --exclude-from=/Users/user/Dropbox/Flex/Scripts/mac/rysnc-backup-to-cp/config/exclude -e "ssh -q -p 22 -i /Users/enwhat/.ssh/user" / user@192.168.0.21:/media/Backup/_Backup/Machine/

Example of where rsync will typically get stuck:

<f+++++++ Volumes/Data/Users/user1/Pictures/2013_12_iPhone_Archive/IMG_6993.m4v 17.33M 63% 994.25kB/s 0:00:10

or

<f+++++++ Volumes/Data/Users/user1/Documents/docs/Work/_Sort from USB backup drive/Drive/JOB/CD Album/AAA1834__Album&flyer_15_Years/2-Design/1-D-Visuals/stage 05/AAA_album_12_c.psd 96.40M 50% 1.55MB/s 0:01:00

I’ve tried removing --verbose --rsync-path="sudo rsync” --delete-during all individually. When I remove these argument flags the rsync process will get to a given file and then hang.

Is there something else at play here or is it quite likely that an illegal character in the filename is causing an issue between FS types?

I did think that Crashplan, which is running on the sever may have been eating up too much resource and causing rsync to crash. But when I stop the CrashPlan service on the server, resources do free up but rsync still crashes on the same files. This is a side note and a outside the scope of the question, but I do wonder if I should ditch Crashplan and switch to Amazon Glacier as a backup service as Crashplan sucks up a lot of CPU and memory.

  • I just hit what appears the exact same issue. I haven't notice that files with ampersands in them cause particular trouble, but I'm trying to rsync several terabytes of files from a USB drive to a NAS. I ran rsync overnight and this morning it finished but had an error that some files were not copied. I checked and many were indeed missing, so I tried again, and now it consistently gets stuck after the first couple files. rsync is still running on the server, but not doing anything. – Mitch Lindgren Sep 22 '17 at 5:45
  • @MitchLindgren I didn’t find a solution to this problem. In the end I changed to a different backup system. I suspect the issue may have been a number of things working against my setup. Ie invalid characters in file names. A low powered CPU and I/O bus on the backup device. And I was copying from a local HFS+ file system to a remote ext3 file system. I hope you’re able to find a workable solution. – juliushibert Sep 22 '17 at 8:15

I don't know why rsync might be hanging. You might try running a regular copy across that directory tree, and see if you have an I/O error of some kind. Running a file-system check wouldn't be a bad idea.

As for Amazon Glacier, yes you can use that. Duplicity supports S3 as a backup destination, and S3's life-cycle rules allow you to move the files to Glacier automatically (configured via S3 web console). You need to keep the signature and manifest files in S3 (or else duplicity won't be able to retrieve them), but the volume files are only needed if you need to extract backed up files, so they can be safely stored in Glacier. The life-cycle rules require a known prefix, which isn't duplicity's default behaviour, so be sure to check the parameter settings.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.