Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a huge chunk of data (11G) in a subversion repository that I'm using rsync to migrate to Alfresco, which lucene indexes new files as they hit the file system. I'm using a dav mount as a proxy to allow me to rsync.

The issue I'm having is the indexing post-rsync is quite an expensive operation for such a huge chunk of data, so I was wondering whether there's a way I could logically separate the rsync into identically-sized batches (say 500MB each) so I could schedule them in cron.

At the moment, I'm traversing the top level folders and taking the smallest ones across first, but once I'm done with those, the much larger sub-directories are going to be quite troublesome.

Please let me know if you need any further info.

Thanks in advance.

share|improve this question

migrated from stackoverflow.com Nov 11 '11 at 16:45

This question came from our site for professional and enthusiast programmers.

add comment

1 Answer 1

Before I try an answer, an excursion: VCS are pretty good at syncing data between computers. Maybe a better solution would be to checkout the subversion repo to migrate it. If Subversion isn't suitable for some reason, have a look at Git. Git is extremely efficient when it comes to syncing because it can calculate the diff once. Syncing itself is little more than transferring the minimal amount of data.

That said, rsync has no "limit" (either by number of files or the amount of data transferred).

You have two options:

  1. Run rsync though a small pipe command which kills the connection after a certain amount.

  2. Run rsync several times with different filters (once for each subdir or once for all small files and then again for each huge file)

#1 has the obvious disadvantage that it can cause copying of a huge file twice (say you had a small file + 90% big file -> hit limit -> big file is transferred 1.9 times)

#2 has the disadvantage that the filter might be too coarse for your purposes.

share|improve this answer
    
Thanks Aaron, appreciate the quick response. Will reply below. –  Chris Jul 12 '11 at 12:01
    
Sorry, I should've clarified: I have the SVN repository checked out to an ephemeral server in Amazon EC2. I've then mounted Alfresco as davfs on the same server. Are you suggesting that I checkout the repo directly into the davfs mount? I take your two points on board; and I appreciate the caveats in using either as I've considered them before. However, if there isn't a cleaner way of doing this, I'd be more likely to go with the second option of the two. Thanks again. –  Chris Jul 12 '11 at 12:09
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.