Deduplicate images synced across devices in various resolutions

Question

I currently have a couple of scripts and Android apps that together do the following for a set of member devices (smartphones, PCs, digital cameras):

• all pictures taken by all member devices are automatically synced with Dropbox
  • for smartphones, this is done using the Dropsync app
  • for digital cameras, a script is run as soon as the camera connects via USB
• once a week, all pictures from all members synced this way are moved to a different directory on Dropbox, for long-term storage.
• After the move, a deduplication takes place — this archive directory (let's call it Dropbox/PicsArchive/) is scanned, and all duplicates are detected and removed. Currently, I use fdupes to detect the duplicates, but to my knowledge, this only detects exact duplicates, i.e., files that have identical checksums.

The problem

There is however nontrivial linkage between all the members.

For example, when connecting a specific kind of digital camera to the USB of the PC running these scripts, the pictures on its memory card are moved to Dropbox, and downsized copies are generated and then sent to a subset of the smartphone members (using the brilliant Autoremote app). These resized copies can very easily end up in a location on the smartphone that is also being synced by Dropsync. Therefore, the camera's pictures, as well as these resized copies, are then both eventually synced into Dropbox/PicsArchive/.

Another example is taking a picture with the smartphone's camera (high resolution) and sharing it to a WhatsApp contact -- often, WhatsApp reduces the resolution of that image. But I want both locations synced (the camera and the relevant WhatsApp media directory), meaning, Dropsync will sync two pictures (one with high and the other with lower resolution) to dropbox, and both will eventually end up in Dropbox/PicsArchive/.

Obviously, I wish to keep only the highest resolution/quality images. Perhaps a better backup strategy is what is needed here, not a more generic tool to clean up mess that is preventable somehow.

Here are a couple of crude pictures of the current setup. Here's the original use case:

I then implemented linkage, to stimulate the users to make higher quality images with the digital camera, while still being able to easily send those pics to WhatsApp users:

Note also that the path Phone camera → WhatsApp creates a duplicate on the phone (both the phone camera directory and the WhatsApp media directory are synced, which is of course necessary to allow pictures not meant for WhatsApp to be synced as well).

So, how can I deduplicate all these pictures?

Answer 1

ImageMagick should be able to make some hashes to compare images, but it's more like a toolkit you'd have to spend some time learning how to use & create your own custom answer that does exactly what you want, rather than a "here's your answer pasted in". But even Google's "similar images" is far from perfect, and the best way to really detect duplicates involves looking at them, either created "difference" images, or the actual images themselves.

ImageMagick® is a software suite to create, edit, compose, or convert bitmap images.

ImageMagick is free software delivered as a ready-to-run binary distribution or as source code that you may use, copy, modify, and distribute in both open and proprietary applications. It is distributed under the Apache 2.0 license, approved by the OSI and recommended for use by the OSSCC.

The current release is ImageMagick 6.9.0-5 available from http://www.imagemagick.org/download. It runs on Linux, Windows, Mac Os X, iOS, Android OS, and others.

ImageMagick Examples - Image Comparing - LOTS of info & techniques, but a lot involve creating "difference" images to take a look at.

This clip should solve at least part of the "problem" for identical images with different metadata:

You can have IM generate a 'signature' for each image...

identify -quiet -format "%#" images...

The generates a hash string much like MD5 and SHA256 do. However unlike the latter, it uses the actual image data to generate the signiture, not the images metadata.

Thus, if you have two copies of the same picture but with different creation/modification timestamps, you should get same signature for both files, whereas MD5 and SHA256 will produce two signatures even though the image itself is the same.

WARNING: reading and writing a JPEG image will generate different image data and thus a different signature. This is simply due to the lossy compression JPEG image format uses.

Then compare the hashes with the tool of your choice, liking linux you're probably familiar with if [[ "$string1" == "$string2" ]] ...

Some other possible options:

• Image Comparer
• different size image compare software on files32.com - Several programs, like:
  • Abonsoft Image Compare
  • Abonsoft Image Compare Command Line
  • Image Compare Before After AS3
• Abonsoft Image Compare (another site)

But, maybe a better backup strategy that only backs up the "master" high-res images, ignoring created thumbnail & non-master folders would be far faster & easier.

Answer 2

Three tools that I’ve used in the past with great results. All are for Windows. None are open source nor for Linux.

• Picasa: You just load up your images, then go into Experimental features to find the duplicates

• VisiPics: This tool scans for exact duplicates at its most restrictive setting. You can “loosen” the setting to find similar images. For example, you can match images you might be frowning in one picture, but smiling in the other.

• Similarity: Works great for audio, but has a feature for images matching as well. It can also detect images that are not oriented correctly, but doesn’t always do a good job leaving the highest quality image behind.