Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm trying to clean up a photo dump folder, in which several files are duplicated but with different filenames or lost in subfolders.

I've looked at tools like rmlint, duff and fdupes, but I can't seem to find a way to have them keep only the file with the most recent timestamp. I suspect I have to postprocess the results, but I don't even know where to start to do this.

Can anyone guide me on how to get the duplicate files list and delete everything but the newest file?

share|improve this question
    
How comfortable are you with shell scripting? –  Ankur Nov 13 '12 at 6:26
    
Relatively proficient, but I don't know where to start with this task. –  pinkie_d_pie_0228 Nov 13 '12 at 6:30

2 Answers 2

up vote 1 down vote accepted

Note that I use the zsh shell.

Try something like the following (untested; based on https://github.com/lipidity/btrfs-fun/blob/master/dedup):

# checksum everything in ${DIR}
cksums=$(mktemp)
find ${DIR} -xdev -type f -print0 | xargs -0 md5sum > $cksums

# loop through each md5 hash found
for hash in $(sort $cksums | uniq -w 32 -d | cut -c 1-32); do
  # list of files with this hash
  files=$(grep $hash $cksums | cut -c 35-)
  f=(${(f)files})
  unset files
  # $f now contains array of files with the same checksum
  # compare the first file to the rest, deleting any that are older
  newest=$f[1]
  for file in $f[2,-1]; do
    # make sure the files are still the same
    cmp $newest $file || continue
    # remove the older file
    if [[ $file -nt $newest ]]; then
      rm $newest
      newest=$file
    else
      rm $file
    fi
  done
done

Untested, but should get you most of the way. Let me know if anything needs further explanation.

share|improve this answer
    
Aah, I didn't know bash could compare files by time like that (just tried, the comparison works well in bash)! Yes, that inner for loop is pretty much what I needed. Thanks a lot! –  pinkie_d_pie_0228 Nov 13 '12 at 7:42

I would echo the checksum generated using the sum command and filename of each of the various files, then sort by checksum. You can check that those with the same checksum are indeed duplicates using cmp.

share|improve this answer
    
All 3 duplicate finders I mentioned already do that. What I want is to only keep the newest file. –  pinkie_d_pie_0228 Nov 13 '12 at 6:15
    
ls will sort by date. I hadn't realized that was the stumbling block, not the n-way compare. –  Nicole Hamilton Nov 13 '12 at 20:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.