up vote 1 down vote favorite
share [g+] share [fb]

The situation: 2 computers, each has a large section with data on it (the OS and programs are in other sections on the computers and can be ignored). The data should be the same on both computers, so when parts change on one computer the change has to be mirrored by the other computer (within a generous time-frame, say a week). It is not possible to create a network connection between the 2 computers, but I can attach an external drive.

Is there a tool to keep the 2 data sections synchronized using an external drive, where the external drive is much smaller than the data section (e.g. the external drive has only 10% of the size that the data sections have)? It can be assumed that the actual changes easily fit on the external drive, even in the sense that all files that changed (not only the changes) easily fit on the external drive.

If it makes things easier master-slave synchronization would be sufficient, ie only one computer can change data.

I could imagine an approach where the transfer drive stores a DB of time-stamps / checksums of the data files and only stores those files on the external drive that actually changed and need to be updated on the other computer; but I don't know of an exsiting tool that would do this, or solve the problem with a different approach.

[Edit] Note also that the computers are not at the same location, but several kilometers apart.

link|improve this question
67% accept rate
feedback

3 Answers

up vote 1 down vote

If the files are text based you could use a version control system to manage the changes. Just generate a patch of the changes since the last checkin and move that to the target system.

This is easier with a master/slave setup. If changes happen on both systems you would need to have version control repositories on both sides and patch both directions. UGH!

link|improve this answer
The files are mostly binary, but this might still work. However, the data is also too large to have twice on any single system, which I think I'd need to using a version control system? – user12889 Jan 21 '10 at 5:40
git can do binary diffs, but you'd still need to have an extra copy of everything in the repo. a workaround might be to keep the repo on the external drive (unfortunately using more space than you've specified) and to map it as .git under the data store. By "map" I mean symlinking under linux or "mapping a network drive to a directory" under Windows. But there are some tricky complications with this, like the fact that the data sections won't remember which revisions they're at. I think you'd have to create a branch for each and manually switch the repo between them by modifying .git/HEAD. – intuited Jun 3 '10 at 17:21
feedback
up vote 1 down vote

I once had to do this for systems on different unconnected networks. The solution I used was to generate a index file containing the file names and MD5 hash of each file on the target and server systems. Then I copied the target hash file to the server and generated a difference list. Then I compressed the changed files and transferred them to the target system.

link|improve this answer
Did you do this by hand or is there a tool that does this? – user12889 Jan 21 '10 at 5:35
I wrote a perl based web interface (my job at the time). Unfortunately, I don't own the code or have access to the source any more. So I can't supply examples. – Chris Nava Jan 22 '10 at 5:52
feedback
up vote 0 down vote

Is this a situation that could be solved using a DAS or similar shared-storage device?

link|improve this answer
No, the computers are several kilometers apart. – user12889 Jan 21 '10 at 4:10
Ah, thanks for the clarification. – phoebus Jan 21 '10 at 5:42
feedback

Your Answer

 
or
required, but never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.