How to combine two sets of files

Question

I have several very unorganized backups of home directories (same user) that I burned throughout the years. Every once in a while, I just copied the entire home directory to a new directory on a huge hard drive. The format of the home directory changed much throughout the years and some large, important files were simply removed to save space (I knew there was a backup). Even worse some files were moved from something like myfile to old/myfile.

Now, I want to combine these backups into one comprehensive directory that has one and only one copy of everything (only the latest copy). I would like tools that use stuff like filename, modified date, and SHA1Checksums to combine these files. What are some good tools that can do this?

Please no advice on how I should have used backup software. However, I wouldn't mind hearing some software features that help convert this current messy model into a centralized backup model.

Thanks

Answer 1

The simplest method I can think of is to copy each folder one at a time (in order of archive date) into a master folder. Then use de-duplication software to scan the master folder and remove duplicate items.

Answer 2

I have this problem myself. I'm thinking about using version control software (e.g. git or Subversion (See also: TortoiseGit and TortoiseSVN)) to do this for me. Assuming you generally have your backups clustered into progressive timestamps - that is, you have e.g. /backup1/ from 2010, /backup2/ from 2012, /backup3/ from 2015, etc... (as opposed to the backup dates being intermingled in each directory tree) - then this will work perfectly. It will allow you to:

1. Combine all of your backup files into one repository, whereby you can access all files in an organized fashion.
2. Commit each backup directory tree in chronological order and maintain it that way.
3. Easily compare versions of files (text files especially, using a diff utility) with each other.
4. Continue using this system for future commits, so you never have to do this again.

---

The only problem with this is, normal version control software doesn't scale well with loads of folders and especially with large and/or binary (executable) files. There are however version control solutions that are built for this. I've found 3 seemingly good ones (I haven't tested any yet) that all will work on both Windows and Linux:

• Plastic SCM
• Boar VCS
• rdiff-backup

---

Of course, if you've moved entire sub-directories around within your backups over time, as I have, these version control solutions almost certainly won't pick up on that. For that, there is de-duplication software, which you could run on your final backup directory tree commit, which should have all of the files merged together (since moved directories will simply be committed, ignoring their duplicity). :)

Hope that helps...