up vote 3 down vote favorite
share [g+] share [fb]

Does anyone know of tool for Windows (or ideally platform independent) that is capable of syncing directories on two machines that contain several hundred of thousands of files in a fast manner? Fast meaning a few seconds / under a minute in case of only changes in a few hundred files and a few MB have occurred.

The ones I tried are fast with regard that they use the obvious minimal approach (check file mod date and only copy based on this), but even this takes 20 or more minutes with my current directories, even when only about 500 files and maybe 50 MB of data have changed.

Is there a tool that employs some clever scheme to do it faster? Some ideas that come to mind is the use of the MFT on NTFS file systems (or the equivalent on other FS), monitoring of file accesses in the background and thus having always an up to date list of changed files, etc.

link|improve this question
Do you really mean platform dependant – Nifle Feb 4 '10 at 23:23
Does it matter to you whether the syncing happens passively? And are LAN transfers like PeerSync more useful than internet-only tools? – Miss Cellanie Feb 4 '10 at 23:36
Of course, "platform *in*dependent", fixed it. Syncing should be on demand, because the files are partly in use and only when all the accessing applications are closed they are in a clean state. – Elmar Weber Feb 5 '10 at 0:27
feedback

3 Answers

up vote 1 down vote accepted

Dropbox is cross-platform and seems very fast to me, once you have done the initial sync. Its speed will depend on your network connection.

It's only fast and easy to use if you install an application, though; its web interface probably won't accomplish what you want.

link|improve this answer
Also, I vaguely know that at some point, filesystems themselves will slow down from huge numbers of files in a given directory, so that's something to watch out for... – Miss Cellanie Feb 4 '10 at 22:58
1  
To clarify, the number of files is a total spread over a large directory structure, not all in one directory. The performance itself on the FS is "normal". – Elmar Weber Feb 4 '10 at 23:03
I'm giving DropBox a try, I have known of it, but I haven't considered it yet for this. – Elmar Weber Feb 4 '10 at 23:05
I tried DropBox as well and did the same actions as with SyncToy (it's an Eclipse workspace), resulting in about 1500 file changes and 30 MB changed data. The DropBox daemon took about 4 minutes to notice all file changes after Eclipse was closed and then some more time to upload. Since most of this was done in the background it's not noticeable and with LAN sync it will probably go a lot faster between the boxes in question. I think I can life with this, escpecially since DropBox can be used for other purposes as well. – Elmar Weber Feb 5 '10 at 16:50
feedback
up vote 3 down vote

I would recommend a piece of software called PeerSync.

Open Files Processing (i.e. Outlook .pst, or other file that is locked while in use)
Intelligent file transfer ensuring minimal resource usage
Byte-Level replication (minimizing bandwidth utilization)
Continuous file replication (TRUE real-time processing)

It works on two or more computers connected via the LAN or over the internet.

It has real-time sync functionality as files are changed, and it uses byte-level replication to minimize bandwidth and speed up transfers.

They offer a free trial and I can vouch that it works as we use it at a few clients who are spread over multiple geographical areas.

Because it has the ability to work realtime, it's quick. If you want to go crazy, they also offer a high-volume server edition which has multi-threading. You would need to contact them directly for more information.

They also offer additional software called PeerLock which integrates in with PeerSync. It basically replicates "locks" on files that are open. So, if a file is opened on one end, it cannot be opened the other end at the same time. Clever stuff and very useful.

link|improve this answer
PeerSync looks good, I'll give it a try to see if it works for me. – Elmar Weber Feb 5 '10 at 0:30
I didn't get around to checking it because DropBox worked and is cheaper =) But I'm guessing PeerSync will work even better than DropBox. If it bugs me too much I'll definitly give PeerSync a real try. – Elmar Weber Feb 5 '10 at 16:52
deletes previous comment haha, you know it's been a long day when you go to downvote your own answer! I'm tempted to delete - instead I'll comment and say give the software a thorough test before spending any money on it. We've recently had a ton of clients experiencing endless problems, getting absolutely nowhere with their support guys. Maybe it's just us but yeah, have to mention it. – kez Mar 2 '11 at 23:41
feedback
up vote 0 down vote

Have you tried Microsoft's SyncToy 2.0?

link|improve this answer
Just tried it, on a test directory structure with 50,000 files it took 8 minutes for a sync that just had 1500 files with a total size of 30 MB changed. And that even though most of the files still were in some kind of OS cache (8 GB RAM). – Elmar Weber Feb 5 '10 at 16:22
feedback

Your Answer

 
or
required, but never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.