Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

We need a Windows 7 program to remove/check the duplicates but our situation is somewhat different than the standard one for which there are enough programs.

We have a fairly large static archive (collection) of photos spread on several disks. Let's call them Disk A..M. We have also some disks (let's call them Disk 1..9) which contain some duplicates which are to be found on disks A..M.

We want to add to our collection new disks (N, O, P... aso.) which will contain the photos from disks 1..9 but, of course, we don't want to have any photos two (or more) times.

Of course, theoretically, the task can be solved with a regular file duplicate remover but the time needed will be very big.

Ideally, AFAIS now, the real solution would be a program which will scan the disks A..M, store the file sizes/hashes of the photos in an indexed database/file(s) and will check the new disks (1..9) against this database.

However I have hard time to find such a program (if exists).

Other things to note:

  • we consider that the Disks A..M (the collection) doesn't have any duplicates on them
  • the file names might be changed
  • we aren't interested in approximated (fuzzy) comparison which can be found in some photo comparing programs. We hunt for exact duplicate files.
  • we aren't afraid of command line. :-)
  • we need to work on Win7/XP
  • we prefer (of course) to be freeware
share|improve this question
    
How many files and bytes are on disks A..M? –  Dennis Jul 9 '12 at 13:33
    
@Dennis: Do you need an exact number? An estimate would be: 400-500k files taking ~ 4-5 TB –  John Thomas Jul 10 '12 at 6:48
    
I tried a few tools (most already listed here), but none seems to do exactly what you want. A command-line based approach is probably the best solution, but I need additional information to suggest one. 1) Can you access the disks simultaneously? 2) Approximately how many disks A..M and 1..9 are there? 3) Is this a one-time thing or do you want to store the database for future uses? –  Dennis Jul 10 '12 at 15:28
    
Answering: 1.) No, I cannot access the disks simultaneously. But, well, perhaps a kludge can be used. 2.) Now we have 5 (five) disks which are ok (the disks A..M in my example) but an "unknown" number of disks which needs checking. However, I estimate the number of these disks (IOW the 0..9 disks in my example) to 5-6. 3.) Yes, I want to store the db for future use. However I think (ok, I hope) that the "main cleaning" will happen only once. –  John Thomas Jul 10 '12 at 17:49
    
> However I have hard time to find such a program (if exists). Probably because simply moving or renaming files will render the cache useless. The file still exists, but now must be re-cached. I suppose it would still be better for all the files that have not been renamed or moved, but unfortunately nobody seems to have bothered to implement a cache (though I did consider it for my own DFF program which I abandoned after finding AllDup). –  Synetech Jul 19 '12 at 20:04
show 2 more comments

3 Answers

Aproach

  1. Choose a collision-free hash function.

    My example uses SHA1, since the bottleneck is going to be the hard drive anyway.

    If that takes too long, it would be possible to compare only the first megabyte of the files. That should be enough for images.

  2. Read the files of interest on the disks A..M, compute their hashes and store them in a file specific to that disk (so you can add/remove disks later).

  3. Read the files of interest on the disks 1..9 and compute their hashes.

    If a file's hash is already known, perform action (list or delete).

Setup

  1. Download and install Cygwin, a collection of tools which provide a Linux look and feel environment for Windows.

  2. In Windows Explorer, open the folder %ProgramFiles(x86)%\Cygwin\home\%USERNAME%.

  3. Edit the file .bashrc and append the following line:

    export PATH=~:$PATH
    
  4. Create a file called hashdrive and save the following code into it:

    #!/bin/bash
    
    DRIVELETTER=$(echo $1 | tr '[:upper:]' '[:lower:]')
    EXTENSIONS=$(echo $2 | sed 's/,/\\|/g')
    DRIVENAME=$(echo $3 | tr '[:upper:]' '[:lower:]')
    
    set -e
    [ -d /cygdrive/$DRIVELETTER ] || (echo "Drive $DRIVELETTER: does not exist." ; exit 1)
    [ -f ~/drives/$DRIVENAME ] && (echo "Hashfile for drive $DRIVENAME already exists." ; exit 1)
    set +e
    
    mkdir ~/drives 2>/dev/null
    find /cygdrive/$DRIVELETTER -type f -iregex ".*\.\($EXTENSIONS\)" -exec sha1sum {} \; | cut -b -40 > ~/drives/$DRIVENAME
    
  5. Create a file called checkdrive and save the following code into it:

    #!/bin/bash
    
    DRIVELETTER=$(echo $1 | tr '[:upper:]' '[:lower:]')
    EXTENSIONS=$(echo $2 | sed 's/,/\\|/g')
    ACTION=$(echo $3 | tr '[:upper:]' '[:lower:]')
    
    set -e
    [ -d /cygdrive/$DRIVELETTER ] || (echo "Drive $DRIVELETTER: does not exist." ; exit 1)
    set +e
    
    IFS=":" ; for FILE in `find /cygdrive/$DRIVELETTER -type f -iregex ".*\.\($EXTENSIONS\)" -printf %p:`; do
        [ "$(grep -m 1 $(sha1sum "$FILE" | cut -b -40) ~/drives/*)" ] && $ACTION "$FILE"
    done
    

Usage

  • To save the hashes of all images of a certain disk to a file, start Cygwin and execute the following command:

    hashdrive DRIVELETTER EXTENSIONS DRIVENAME
    

    For example, if DiskA is mounted as drive D: and you want to hash all images with extensions jpg and png, use the following command:

    hashdrive d jpg,png diska
    

    There must be no space in jpg,png.

  • To check a disk for duplicate images, start Cygwin and execute the following command:

    hashdrive DRIVELETTER EXTENSIONS ACTION
    

    For example, if Disk1 is mounted as drive E: and you want to list all duplicate images with extensions jpg and png, use the following command:

    checkdrive e jpg,png echo
    

    If you want to remove the files directly, use rm instead of echo.

  • To remove a disk from the database, just delete the file DRIVENAME in the folder %ProgramFiles(x86)%\Cygwin\home\%USERNAME%\drive.

Caution

The rm command does not move files to the Recycle Bin; it deletes them directly.

While it should be possible to recover the files anyway, be careful when using the rm action and try echo before you use rm.

share|improve this answer
    
I'm getting mixed results. On my Linux machine, things run 9 times faster than on my Win7 machine. I hope it's due to the fact that the hard drives of the Linux machine are faster, but it's also possible that things run smoother with a real Linux. Possible improvements: 1) If hashdrive runs too slow (I have no way to check this from here), it could be made faster by hashing only the first megabyte of each file. That should be enough for images. 2) checkdrive can be made faster by grouping hashes in buckets. I'll test tommorow if it has a mayor impact. –  Dennis Jul 11 '12 at 4:30
    
Thanks a lot - Following loosely your solution, I've posted a full/native Windows solution (ok, it is cross-platform actually, because hashdeep runs also on *nixes). Also, I chose not to encapsulate the commands in .bat files because the commands are fairly simple. Thanks a lot, anyway. Upvoted. –  John Thomas Jul 11 '12 at 9:23
    
Btw, Dennis SHA-1 isn't collision free. See en.wikipedia.org/wiki/Sha-1 ...but for day-by-day files work is sufficient. I chose tiger for speed gain. –  John Thomas Jul 11 '12 at 9:27
    
SHA-1 is broken (its initial security claim of 80 bits has been reduced to 53 bits), but no actual collision has been found so far. –  Dennis Jul 11 '12 at 12:48
add comment
up vote 1 down vote accepted

Based on Dennis solution, we decided to use the hashdeep suite which is also available on Windows.

Basic usage:

Step 1. Generate the hashes (this should be done only once)

hashdeep64 -c tiger -r "D:\*" > Disk_D.hash

We use tiger as a hash function - faster and better than SHA-1 (no collisions).

Step 2: Hunt for duplicates (this must be executed for each drive / dir to check)

hashdeep64 -k Disk_D.hash -m "E:\My-Dir-To-Check\*" > Dupes.txt

Now all the duplicates are stored in Dupes.txt

You can use MsWord, LibreOffice or Notepad++ (or any other way you know) to insert del (and/or any other options) in this text file in order to delete the files. You have here enough variants, including a simple .bat file which scans the file list in order to delete all the entries.

Also, you have the choice to review the file list and do the processing manually.

share|improve this answer
add comment

Loosely following your solutions here, I've posted a full/native Windows solution – ok, it is cross-platform actually, because hashdeep runs also on *nixes.

Also, I chose not to encapsulate the commands in .bat files because the commands are fairly simple.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.