up vote 1 down vote favorite
1
share [g+] share [fb]

Are there any tools out there that will echo one-way changes from one large directory (5+ million files) to another across a Windows network? I also need it to be a fairly quick sync after the initial build up. I've played around with a few tools like Microsoft SyncToy and Cobian backup but they don't seem to be built for this kinda of size.

Less than 1% of the files change every 1 hour but the problem with the design of the directory system I've inherited is that the changes can be at any position in the directory tree and there is no log of the changes so a full scan was being preformed in the past which would take about 4-5 hours.

I built a custom tool to do the sync much faster than these others but I just don't have the time to build in all the bells and whistles so I'm looking for something to do the trick in the mean time. I do plan on changing the system but at this point I'm just looking to get them synced with less lag time.

[Update] -My files are pretty small ranging from 10-30kb each -The directory tree is pretty shallow.

link|improve this question
100% accept rate
feedback

2 Answers

up vote 2 down vote accepted

When you say 'windows network', do you mean you have a Windows network with an Active directory? One possibility that you should be looking at DFS-R. Newer versions of DFS-R watch the filesystem and only transmit changes. If you do have 5 million files, you may be getting close to the DFS limits though.

link|improve this answer
Yeah we have a few Windows Storage Server edition boxes and we tried turning on DFS yesterday and the box didn't seem to enjoy swallowing that much data and we had to stop it because clients started to call. Does it seem like a better idea to start with one folder at a time than the entire beast? – Michael J. Lee Jul 13 '11 at 19:42
DFS-R may be the way to go, but the initial sync is sure to take a very long time. This MS Blog covers how to pre-seed for DFS – Zoredache Jul 13 '11 at 20:05
+1 allow me to look into this a little more – Michael J. Lee Jul 13 '11 at 20:08
feedback
up vote 0 down vote

You could try one of these:

cwrsync - Rsync for Windows

Robocopy

link|improve this answer
Neither of these tools avoid a full filesystem scan. With millions of files these will take a long time. – Zoredache Jul 13 '11 at 19:57
yeah both take a long time to scan! I'm surprised no one has built parallel scanning and copying system. My files are small (10-30 kb) so the copying should go pretty quick – Michael J. Lee Jul 13 '11 at 20:03
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.