I currently have a couple of scripts and Android apps that together do the following for a set of member devices (smartphones, PCs, digital cameras):
- all pictures taken by all member devices are automatically synced with Dropbox
- for smartphones, this is done using the Dropsync app
- for digital cameras, a script is run as soon as the camera connects via USB
- once a week, all pictures from all members synced this way are moved to a different directory on Dropbox, for long-term storage.
- After the move, a deduplication takes place — this archive directory (let's call it
Dropbox/PicsArchive/) is scanned, and all duplicates are detected and removed. Currently, I use
fdupesto detect the duplicates, but to my knowledge, this only detects exact duplicates, i.e., files that have identical checksums.
There is however nontrivial linkage between all the members.
For example, when connecting a specific kind of digital camera to the USB of the PC running these scripts, the pictures on its memory card are moved to Dropbox, and downsized copies are generated and then sent to a subset of the smartphone members (using the brilliant Autoremote app). These resized copies can very easily end up in a location on the smartphone that is also being synced by Dropsync. Therefore, the camera's pictures, as well as these resized copies, are then both eventually synced into
Another example is taking a picture with the smartphone's camera (high resolution) and sharing it to a WhatsApp contact -- often, WhatsApp reduces the resolution of that image. But I want both locations synced (the camera and the relevant WhatsApp media directory), meaning, Dropsync will sync two pictures (one with high and the other with lower resolution) to dropbox, and both will eventually end up in
Obviously, I wish to keep only the highest resolution/quality images. Perhaps a better backup strategy is what is needed here, not a more generic tool to clean up mess that is preventable somehow.
Here are a couple of crude pictures of the current setup. Here's the original use case:
I then implemented linkage, to stimulate the users to make higher quality images with the digital camera, while still being able to easily send those pics to WhatsApp users:
Note also that the path Phone camera → WhatsApp creates a duplicate on the phone (both the phone camera directory and the WhatsApp media directory are synced, which is of course necessary to allow pictures not meant for WhatsApp to be synced as well).
So, how can I deduplicate all these pictures?