Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need to synchronize often two large directories (with lots of sub directories and files) between a Windows XP and a Unix Server. (I currently do it with the nice WinSCP, but have tried a bunch of others.)

My problem is that every time I synchronize, the software checks every file to see if it has been updated. That takes about 1 minute.

I dream of a software that would keep track on both systems of which directory was updated, and wouldn't visit a directory unless needed.

Since I usually change just a few files, that should speed up my sync time from 1 minute to at most 1 sec.

Is there any software that does that, free or not?

One solution would be to synchronize both systems to some remote thing like DropBox. There is a number of reasons why I do not want to do that. It slows everything, it costs, and also I do not need my files in any other place.

Thanks.

share|improve this question
    
I'd like to hear some solutions for this as well. Right now I've got an rsync setup via cygwin, but over the years the folders have come to contain many millions of tiny JPGs and it takes around 24 hours to complete a sync -- even when only when new JPG has been added -- because of all the time spent verifying the existing JPGs haven't changed. –  Uninspired Feb 19 '11 at 18:39
    
So your issue is not that it checks every file, but actually how long it takes? –  Daniel Beck Feb 19 '11 at 18:45
    
why doesn't it store hashes of files and then check if the hashes have changed, it should be faster. –  Jonathan. Feb 19 '11 at 18:53

1 Answer 1

up vote 0 down vote accepted

Is this a one-directional transfer (i.e. back up new and changed files on the Windows XP side to the Unix server, nothing is making changed from the Unix end)? That might make it much easier to find a solution.

In order to avoid scanning all the existing unchanged files, you'll need something that checks a list of changes. On XP, there's the NTFS Change Journal. Unix/Linux systems have inotify and also journaling filesystems. But finding a single piece of software that does BOTH could be difficult. That's why I'm hoping this is a uni-directional incremental mirror and not a true "sync".

Oh... I should mention that another method is with a filesystem (filter) driver. On Linux the "fuse" framework makes this fairly simple, but this approach is less used because it's so much more complicated than processing the journal.

share|improve this answer
    
It is a true sync. I edit the files both from the unix side (to which I actually connect via a windows folder through a local network) and remotely from windows. Thanks for your input though. Conceptually this is a very simple task of course, and I am always amazed at how difficult it is to implement such things ;-) –  Emanuele Viola Feb 19 '11 at 20:41
    
@Emanuele: Like I said, the information is there on both Windows and Unix, it's a matter of finding software which is optimized on both platforms. But now you have some keywords to use when looking for programs/reading feature lists. –  Ben Voigt Feb 19 '11 at 20:44
    
@Ben: Since I don't manage the unix server, I wonder: Is the information supposed to be on a standard server (as opposed to requiring installing/modifying the filesystem), and is it going to be accessible by me? If so, do you think it would be complicated to write such an app? I'd guess many people would benefit from it. BTW: I appreciate your pointers, if it turns out there isn't anything else, I'll accept your answer. –  Emanuele Viola Feb 20 '11 at 0:19
    
@Emanuele: Some of the most common filesystems do journaling by default, but several do not, so you'd have to find out from your provider what the filesystem is. And reading the filesystem journal of a physical partition is probably not possible from an unprivileged account. OTOH inotify should work fine from userspace (just make sure you always have the observer running in the background, unlike the journal, inotify doesn't tell you what happened when you weren't looking), or a filesystem mounted on loopback (instead of a partition) would give you full read access to the journal data. –  Ben Voigt Feb 20 '11 at 0:31
    
I would love it if you could provide more details about your approach: is there a good place to read about these journaling files, where are they stored, how can I parse them? If it wasn't a true sync, do you know an app? Also, what language would you use to code up such an app? –  Emanuele Viola Feb 24 '11 at 14:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.