I'm trying to clean up a photo dump folder, in which several files are duplicated but with different filenames or lost in subfolders.

I've looked at tools like rmlint, duff and fdupes, but I can't seem to find a way to have them keep only the file with the most recent timestamp. I suspect I have to postprocess the results, but I don't even know where to start to do this.

Can anyone guide me on how to get the duplicate files list and delete everything but the newest file?

  • How comfortable are you with shell scripting? – Ankur Nov 13 '12 at 6:26
  • Relatively proficient, but I don't know where to start with this task. – pinkie_d_pie_0228 Nov 13 '12 at 6:30

Note that I use the zsh shell.

Try something like the following (untested; based on https://github.com/lipidity/btrfs-fun/blob/master/dedup):

# checksum everything in ${DIR}
find ${DIR} -xdev -type f -print0 | xargs -0 md5sum > $cksums

# loop through each md5 hash found
for hash in $(sort $cksums | uniq -w 32 -d | cut -c 1-32); do
  # list of files with this hash
  files=$(grep $hash $cksums | cut -c 35-)
  unset files
  # $f now contains array of files with the same checksum
  # compare the first file to the rest, deleting any that are older
  for file in $f[2,-1]; do
    # make sure the files are still the same
    cmp $newest $file || continue
    # remove the older file
    if [[ $file -nt $newest ]]; then
      rm $newest
      rm $file

Untested, but should get you most of the way. Let me know if anything needs further explanation.

  • Aah, I didn't know bash could compare files by time like that (just tried, the comparison works well in bash)! Yes, that inner for loop is pretty much what I needed. Thanks a lot! – pinkie_d_pie_0228 Nov 13 '12 at 7:42

I would echo the checksum generated using the sum command and filename of each of the various files, then sort by checksum. You can check that those with the same checksum are indeed duplicates using cmp.

  • All 3 duplicate finders I mentioned already do that. What I want is to only keep the newest file. – pinkie_d_pie_0228 Nov 13 '12 at 6:15
  • ls will sort by date. I hadn't realized that was the stumbling block, not the n-way compare. – Nicole Hamilton Nov 13 '12 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.