I've got a long document, and I suspect that some phrases (one or multiple sentences) are identical or at least similiar. Is there some way to find these repetitive phrases? Some tool or regex?
1 Answer
Finding identical sentences can easily be done in Notepad++:
- Use a regexp to have every sentence on a new line. This can be done (for example) be searching for all
.
and replacing them with.\n
- Remove all whitespaces at the beginning of the line.
- Now go ahead and sort the lines (TextFX -> TextFX Tools)
- Convert the EOL character to UNIX because ot makes things simpler (Edit -> EOL conversion)
- Now search for duplicated lines by searching for
^([^\n]+)\n\1
To find similar sentences you would need something like a word-stemming program or some kind of fuzzy-search.
-
1That will only find two identical lines that are one after the other. Is that what the OP is asking? Apr 3, 2014 at 14:43
-
No, it will find all identical lines; this is what step 3. is for. Since we sort the lines, identical lines will be sorted one after the other.– masgoApr 10, 2014 at 12:58