How to find candidates of file duplicates based on a similarity of their names?

Question

I am not searching similar images or photos (imagedupe) nor audio files!

I have a lot of media data here and I want to spot duplicates based on their file name. I know that there is fdupes, etc. but this won't work in my case here (e.g. reencoding of the same data). I can imagine that string similarity is a hard task, but based on the fact that the words "the, and, of" have all less information than e.g. "plant, pirate" I know that it has to be possible to do this stuff. I know I won't spot all duplicates with such an method, as numbers can also be written in words, names in CamelCase and in 1337 may also be hard candidates. But I've wondered if something similar has already be conceived.

My first try is this:

• first extract all relevant words:

  find . -type f -exec basename {} \; | sort | tr "\n" " " | sed 's/[\._\-]/ /g' | tr " " "\n" | sort | uniq -c

• second filter out words like: "and, the,..."

• for each word in the remaining list make a look up like

  find . -iname $word -type f

Answer 1

glimpse and agrep springs to mind.