Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a huge chunk of data (11G) in a subversion repository that I'm using rsync to migrate to Alfresco, which lucene indexes new files as they hit the file system. I'm using a dav mount as a proxy to allow me to rsync.

The issue I'm having is the indexing post-rsync is quite an expensive operation for such a huge chunk of data, so I was wondering whether there's a way I could logically separate the rsync into identically-sized batches (say 500MB each) so I could schedule them in cron.

At the moment, I'm traversing the top level folders and taking the smallest ones across first, but once I'm done with those, the much larger sub-directories are going to be quite troublesome.

Please let me know if you need any further info.

Thanks in advance.

share|improve this question

migrated from Nov 11 '11 at 16:45

This question came from our site for professional and enthusiast programmers.

Before I try an answer, an excursion: VCS are pretty good at syncing data between computers. Maybe a better solution would be to checkout the subversion repo to migrate it. If Subversion isn't suitable for some reason, have a look at Git. Git is extremely efficient when it comes to syncing because it can calculate the diff once. Syncing itself is little more than transferring the minimal amount of data.

That said, rsync has no "limit" (either by number of files or the amount of data transferred).

You have two options:

  1. Run rsync though a small pipe command which kills the connection after a certain amount.

  2. Run rsync several times with different filters (once for each subdir or once for all small files and then again for each huge file)

#1 has the obvious disadvantage that it can cause copying of a huge file twice (say you had a small file + 90% big file -> hit limit -> big file is transferred 1.9 times)

#2 has the disadvantage that the filter might be too coarse for your purposes.

share|improve this answer
Thanks Aaron, appreciate the quick response. Will reply below. – Chris Jul 12 '11 at 12:01
Sorry, I should've clarified: I have the SVN repository checked out to an ephemeral server in Amazon EC2. I've then mounted Alfresco as davfs on the same server. Are you suggesting that I checkout the repo directly into the davfs mount? I take your two points on board; and I appreciate the caveats in using either as I've considered them before. However, if there isn't a cleaner way of doing this, I'd be more likely to go with the second option of the two. Thanks again. – Chris Jul 12 '11 at 12:09
That is not accurate, if you kill rsync when you are transfering a file at 90%, when you start it again it will resume it. – useless Dec 9 '14 at 20:58
I don't know much about davfs and how it responds to various operations but maybe you can try to checkout the subversion inside of the davfs mount. The problem here is that svn will do several read/write operations per file so this may be bad for large files (not probably not so bad if you have just many files). After the checkout/update, Alfresco should see some files with changes dates. If this change can trigger the reindex, then this should be very efficient. – Aaron Digulla Dec 10 '14 at 12:57
Generally, my approach is to run the indexing on the same computer (i.e. not transfer those files again to some remote service). In this case, I'd try to checkout the data on the server which runs Alfresco instead of the davfs mount. – Aaron Digulla Dec 10 '14 at 12:58

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.