Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have had an rsync backup script running on a Centos 6 box that has been syncing ~410gb of data from a local (LAN) server over a gigabit connection. This script runs hourly, and if there are no new files to copy then it usually takes about 20 mins to complete.

I have implemented a purge script that runs on the Centos box and deletes all files from the local backup copy older than 1 year (after some checks) and adds these files to an exclusion list for rsync to parse. It all works ok, but it's taking about 10 hours to complete (with no new files). After the initial purge the source data is now about 80gb and the exclusion list is about 90mb.

Here's my rsync line from the script:

rsync -pad $wholefiles  --exclude-from=${exclfile}.combined /mnt/${smbdir}/${drive}/${foldername} /backup/rob/${smbdir}/${drive} >> /var/log/rob/robso.log 2>&1

Does rsync (or bash) have size limitations for file lists? Can you see anything in my syntax that would cause such a huge amount of processing time? Thanks in advance.

EDIT: The $wholefiles variable in the rsync line above passes the --whole-file option in this case.

share|improve this question
Have you disabled the cron job before testing? Check if you have multiple rsync processes running in the background. Check your netIO. – Kaurin Aug 23 '13 at 10:50
Thanks, but yes I have, @Kaurin. I also read that rsync 3.0+ offers better speed with large file numbers so I upgraded from 2.68 --> 3.06. Ran last night and it's no faster. – kiwisan Aug 23 '13 at 14:56
How are you mounting samba? Fuse = bad, Cifs = good. Fuse has horrible performance. – Kaurin Aug 23 '13 at 19:08
I had an issue with long rsync delays with very large folder structures (100k folders, 1mil files). Setting up unison to do a one-direction sync over ssh runs in seconds without updates. The problem for me was that rsync sends the whole file list over the network, regardless if a file has changed. Unison only sends the list of changed files. – Clayton Stanley Aug 24 '13 at 0:23

You must log in to answer this question.

Browse other questions tagged .