Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have 6 backups of my PC wasting a lot of space, on an external hard disk, because all of them are full backups (not incremental) of my PC, meaning that the majority of the contents are duplicated across the backups.

My idea is to trim those backups, keeping only files MISSING from the original backup source, REGARDLESS the folder location.

As backups are "mountable units" in such a way I can use common file/folders utilities to find duplicates & missing files between two compared sets, I have tried some tools such as Windiff and Comparator pro, but both of those tools will show as missing files those files moved in other folders, when comparing with the current file set.

What I need is a tool the will list missing files from the backup source that are present in the backup, wherever they are, even though they have been moved elsewhere.

Any hint?

share|improve this question
    
For the right price, I'll write one for you. . . –  surfasb Oct 17 '11 at 21:32
    
What OS are you running and what file system does the external drive use? –  Scott McClenning Oct 18 '11 at 0:16
    
Windows 7. @surfasb, thanks but it would be easier to use a duplicate file finder and delete all duplicates found in the backup :-) –  Riccardo Oct 18 '11 at 8:25
add comment

3 Answers

up vote 1 down vote accepted

I too have been looking for space efficient backups and here are some applications I've found:

Dupemerge is a command line program what will dedupe directories using hard links. This program will look at a directory or directories you point it to and if there are duplicate files, it will hard link them. If you backup on a regular basis, you can schedule this to run after the backup and free up the space.

CloneSpy is a GUI program that will display lists of duplicate files. You can manually dedupe some files or have it automatically dudupe the files for you. Hard linking needs to be turned on in the options, this program started out as a program to remove duplicate files and they have since added the hard link capability. Si if you want the file hard linked, you must turn that on. I don’t know CloneSpy knows the NTFS hard link limitations, but Dupemerge does.

There are some Windows Explorer extensions to also create hard links and to visually see which files are hard linked from within Windows Explorer, Hard Link Shell Extension will put a red “shortcut” arrow overlay over the files that have been hard linked. Also, on local drives in the file properties it will display which files are hard linked together. It is nice to see which files are duplicates and which are unique. Also if there is a chance of editing the files, this will act as a warning to remind you that if you edit a file with a red arrow that you are actually editing all the hard linked files at the same time. The website http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html has a ton of information about Hard Links, Junctions, Symbolic Links.

Using hard links like this is nice because each backup folder looks like a complete full backup, but common files within the backup and across backups are only using space up once, usually. Usually because NTFS has a limitation of 1023 hard links to one file, and dupemerge will only hard link 1022 files to one file, so if you have more than 1022 copies, a second copy of the data will have to be stored for another 1022 hard links to link to.

There are some programs intended for space efficient backups, duplicati (a Windows port of the Linux duplicity backup program). This GUI program, from what I understand, will dedupe because it hashes the data. It reminds me of using rsync for backup. The current version of duplicati are much improved, and I would recommend it. It can manage your backups by setting how many backups to keep, how much space to use, max age of backups and so on. I use this program for long term backups.

Hardlink Backup (formerly RsyncBackup) (a GUI program) dedupes by hard links. (I hadn't used this program since it was rebranded.) Because I was going to use my program in a commercial environment I didn't test the program out much more. However it did appear to work well.

Rdiff-Backup (a command line program) also dedupes by hard links. The thing I didn't care for was it put a directory with all the revision history in the backup directory. If that was necessary I wish they would have hidden it and made it a dot directory. This program is similar to an rsync backup program.

Just to warn you, with Windows it isn't always clear how much space hard linked files are using. I believe the overall drive statistics are correct and show actual space used. However, if you did a properties over the backup directories it would look like no space was saved by hard linking. Eventually, you should be able to store what looks to be over 100% of the capacity of the drive. This is because Explorer counts the space each file entry takes up, but doesn't check to see if two files entries are pointing to the same piece of data on the disk.

--Added--

If you only want to keep files not in the original backup regardless of their location. (e.g. FolderA has File1 in Backup1, and in Backup2 File1 is in FolderB. Because File1 was not changed, but relocated you want a program to remove File1.) Then the program I would suggest is CloneSpy. This was what the program was originally designed for. CloneSpy has many options. I've used it for a similar task when I would compare FolderA to FolderB, FolderC & FolderD. Then compare FolderA & FolderB to FolderC & FolderD. Lastly compare FolderA, FolderB & FolderC to FolderD. That way among all the folders only unique files exist.

--Added 2--

Actually a simpler way to do the above is to have all the directories in one group and tell it to delete newer files. That would leave the oldest copy of the dupe set. This way you can tell when a version of the file was first created. This would also dedupe files within the directories as well as across the directories, resulting in only one copy of a unique file.

share|improve this answer
    
very nice post, very helpful. Thanks :-) –  Riccardo Oct 19 '11 at 7:00
    
Sorry, I got hardlink backup and rdiff-backup mixed up. It was about 18 months ago when I evaluated them. –  Scott McClenning Oct 19 '11 at 7:39
    
Scott, I have noticed that the same files in the backups have strange differences: 1) exactly 1 hour time shift in "modified date", 2) Some of them have some 3/6kb size difference. Could this depend on cluster size on different partition sizes? –  Riccardo Oct 20 '11 at 16:38
    
That is interesting. I've seen 1 hour time differences from daylight savings being in effect or not. As far as cluster size, I hadn't seen that change the size of a file, but I have seen it change the amount of space a file takes. The unused part of the cluster is slack space. It would be interesting to take a diff tool over the two files to see what differences there are. I've had files contain the same data but were different because of meta data (ex. word doc records when and how many saves even if you don't change the content of the document each save will be a unique file) –  Scott McClenning Oct 21 '11 at 2:16
add comment

This is what I would do:

  1. For each file in backups, calculate hash code by the file content (may be md5sum) and redirect the results and full name to a text file (file1).
  2. Sort the text file by hash code and the path.
  3. Sort the text file by hash code with -u option and redirect the result to another text file (file2)
  4. Now file2 has all unique files (if backups have multiple versions of a file, all versions will be preserved). You can join file1 and file2 to find out the redundant files in file1.
share|improve this answer
    
what tools to achieve this? Isn't there a tool wit a user interface? Because once I find the missing files from the sets, I'd like to copy them somewhere, and delete the duplicates... –  Riccardo Oct 17 '11 at 16:55
    
It may be easier to delete duplicates and obtain missing files... –  Riccardo Oct 17 '11 at 16:57
    
All tools are included in unix text tools, which is available in most unix/linux releases; also available to windows through cygwin. The programs you need are: md5sum, sort, join. If you have no experience with them, it takes some efforts to learn but I hope you already see they are very worth learning. –  Codism Oct 17 '11 at 18:16
    
Thanks, could be an idea, however instead of script programming, I'd rather stick with a tool that will offer duplicates finding with a "base" set; a sort of compare "base set" with "other set", then clear duplicates only from "other set"... –  Riccardo Oct 18 '11 at 8:27
add comment

It all ended using Duplicate Finder, a tool by Ashisoft.

Pros: it does what I was looking for.

Cons:
- Expensive, IMO, 29USD
- While purchasing, the ecommerce site "dropped" in my basket to other tools I didn't asked for, riseing the price to 50USD, had to remove manually from the basket.
- the price was converted in Euros, then back again in dollars!Can you believe this?

Very unfair, just bought it because on a hurry, I believe there are tons of source codes out there so just copying this code and rearranging it asking for so much money, just sucks!

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.