I currently have a couple of scripts and Android apps that together do the following for a set of member devices (smartphones, PCs, digital cameras):

  • all pictures taken by all member devices are automatically synced with Dropbox
    • for smartphones, this is done using the Dropsync app
    • for digital cameras, a script is run as soon as the camera connects via USB
  • once a week, all pictures from all members synced this way are moved to a different directory on Dropbox, for long-term storage.
  • After the move, a deduplication takes place — this archive directory (let's call it Dropbox/PicsArchive/) is scanned, and all duplicates are detected and removed. Currently, I use fdupes to detect the duplicates, but to my knowledge, this only detects exact duplicates, i.e., files that have identical checksums.

The problem

There is however nontrivial linkage between all the members.

For example, when connecting a specific kind of digital camera to the USB of the PC running these scripts, the pictures on its memory card are moved to Dropbox, and downsized copies are generated and then sent to a subset of the smartphone members (using the brilliant Autoremote app). These resized copies can very easily end up in a location on the smartphone that is also being synced by Dropsync. Therefore, the camera's pictures, as well as these resized copies, are then both eventually synced into Dropbox/PicsArchive/.

Another example is taking a picture with the smartphone's camera (high resolution) and sharing it to a WhatsApp contact -- often, WhatsApp reduces the resolution of that image. But I want both locations synced (the camera and the relevant WhatsApp media directory), meaning, Dropsync will sync two pictures (one with high and the other with lower resolution) to dropbox, and both will eventually end up in Dropbox/PicsArchive/.

Obviously, I wish to keep only the highest resolution/quality images. Perhaps a better backup strategy is what is needed here, not a more generic tool to clean up mess that is preventable somehow.

Here are a couple of crude pictures of the current setup. Here's the original use case:

I then implemented linkage, to stimulate the users to make higher quality images with the digital camera, while still being able to easily send those pics to WhatsApp users:

Note also that the path Phone camera → WhatsApp creates a duplicate on the phone (both the phone camera directory and the WhatsApp media directory are synced, which is of course necessary to allow pictures not meant for WhatsApp to be synced as well).

So, how can I deduplicate all these pictures?

  • Let us continue this discussion in chat. – Rody Oldenhuis Feb 10 '15 at 18:06
  • Feel free to re-ask this on Software Recommendations. As it stands, this is just a request for list software that does XYZ. A valid alternative would be to just include whatever you've tried and describe the actual problem you're trying to solve. You'll find lots of people who are up for hacking something together (even based on something you started), but I have to agree with Jake here that this looks like a whishlist for a magic program that may or may not exist. Those kinds of questions are not encouraged here (and on most SE sites). – slhck Feb 10 '15 at 18:32
  • @slhck OK, how about this? – Rody Oldenhuis Feb 10 '15 at 19:03
  • I tried to remove the (now irrelevant) part where you're asking for a tool. For me the question is fine that way, you may however 1) try to further reduce it to the bare essentials needed to provide an answer and 2) notify those who already answered that their answers are now no longer valid. Generally it's not so nice to change a question that radically, but given that the answers you have now don't look like a solution to your original question either (and aren't upvoted), I'd let the rewrite pass here. – slhck Feb 10 '15 at 19:18
  • 1
    And this: jhnc.org/findimagedupes/manpage.html – Sun Feb 12 '15 at 16:41

ImageMagick should be able to make some hashes to compare images, but it's more like a toolkit you'd have to spend some time learning how to use & create your own custom answer that does exactly what you want, rather than a "here's your answer pasted in". But even Google's "similar images" is far from perfect, and the best way to really detect duplicates involves looking at them, either created "difference" images, or the actual images themselves.

ImageMagick® is a software suite to create, edit, compose, or convert bitmap images.

ImageMagick is free software delivered as a ready-to-run binary distribution or as source code that you may use, copy, modify, and distribute in both open and proprietary applications. It is distributed under the Apache 2.0 license, approved by the OSI and recommended for use by the OSSCC.

The current release is ImageMagick 6.9.0-5 available from http://www.imagemagick.org/download. It runs on Linux, Windows, Mac Os X, iOS, Android OS, and others.

ImageMagick Examples - Image Comparing - LOTS of info & techniques, but a lot involve creating "difference" images to take a look at.

This clip should solve at least part of the "problem" for identical images with different metadata:

You can have IM generate a 'signature' for each image...

identify -quiet -format "%#" images...

The generates a hash string much like MD5 and SHA256 do. However unlike the latter, it uses the actual image data to generate the signiture, not the images metadata.

Thus, if you have two copies of the same picture but with different creation/modification timestamps, you should get same signature for both files, whereas MD5 and SHA256 will produce two signatures even though the image itself is the same.

WARNING: reading and writing a JPEG image will generate different image data and thus a different signature. This is simply due to the lossy compression JPEG image format uses.

Then compare the hashes with the tool of your choice, liking linux you're probably familiar with if [[ "$string1" == "$string2" ]] ...

Some other possible options:

But, maybe a better backup strategy that only backs up the "master" high-res images, ignoring created thumbnail & non-master folders would be far faster & easier.

| improve this answer | |
  • Thanks, but do you have any experience with any of them? Can you make statements about the reliability of each? Can you show examples of the inputs & outputs? ...or is this just a list of the first couple of Google results I also found? :) – Rody Oldenhuis Feb 10 '15 at 17:31
  • This is not an answer, but just an open-ended list of tools. For example, I know from personal use that ImageMagick can do image comparison, but it is—at it’s core—an intense process to make it work well on a production basis. One has to essentially build some framework—using scripting tools like bash—to make ImageMagick work well for bulk comparison in a truly practical way. – Giacomo1968 Feb 10 '15 at 17:41
  • Added command line answer for "different metadata" images. No personal experience using them, only "statements about the reliability" of ImageMagick would be that it's FOSS, so probably similar to the reliability of other FOSS projects – Xen2050 Feb 10 '15 at 18:06
  • I like the last two sentences...however, I don't see a better strategy; there is often no way for the scripts do distinguish between an image sent via Whatsapp, and the "master" image it comes from (e.g., digital camera or phone camera run from inside or outside Whatsapp, ...) – Rody Oldenhuis Feb 10 '15 at 18:30
  • Maybe the metadata there could help? A camera/phone "original" image usually has the type of camera, settings, sometimes even GPS info... I don't know if whatsapp copies all those, or if it did it might add it's own "use whatsapp!" tagine or something? – Xen2050 Feb 10 '15 at 18:35

Three tools that I’ve used in the past with great results. All are for Windows. None are open source nor for Linux.

  • Picasa: You just load up your images, then go into Experimental features to find the duplicates

  • VisiPics: This tool scans for exact duplicates at its most restrictive setting. You can “loosen” the setting to find similar images. For example, you can match images you might be frowning in one picture, but smiling in the other.

  • Similarity: Works great for audio, but has a feature for images matching as well. It can also detect images that are not oriented correctly, but doesn’t always do a good job leaving the highest quality image behind.

| improve this answer | |
  • Thanks, but...I'm a Linux man :) – Rody Oldenhuis Feb 10 '15 at 17:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.