Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am not searching similar images or photos (imagedupe) nor audio files!

I have a lot of media data here and I want to spot duplicates based on their file name. I know that there is fdupes, etc. but this won't work in my case here (e.g. reencoding of the same data). I can imagine that string similarity is a hard task, but based on the fact that the words "the, and, of" have all less information than e.g. "plant, pirate" I know that it has to be possible to do this stuff. I know I won't spot all duplicates with such an method, as numbers can also be written in words, names in CamelCase and in 1337 may also be hard candidates. But I've wondered if something similar has already be conceived.

My first try is this:

  • first extract all relevant words:

    find . -type f -exec basename {} \; | sort | tr "\n" " " | sed 's/[\._\-]/ /g' | tr " " "\n" | sort | uniq -c

  • second filter out words like: "and, the,..."

  • for each word in the remaining list make a look up like

    find . -iname $word -type f

share|improve this question
2  
It would probably be easier to do this with the metadata in each media file. A lot of it tends to be preserved even when re-encoded to another format. For example date, time, size, camera model and colors for pictures. –  Brian Mar 4 '12 at 22:42
add comment

1 Answer

up vote 2 down vote accepted

glimpse and agrep springs to mind.

share|improve this answer
    
Yes, together with tr (converting all to lower case for instance) this is a great tool in reducing words and improve matches. –  math Mar 5 '12 at 8:48
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.