Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm working on a file containing lines with keywords and some lines contain duplicated keywords.

For example:

dangerous,dangerous,hazardous,perilous

I want to tell Notepad++ that I want to remove every duplicated word per line. For this example dangerous, would be removed:

dangerous,hazardous,perilous

I have a bunch of lines like that and that's why I'm looking for an automated way of doing this.

share|improve this question
    
What about dangerous,hazardous,dangerous,perilous? In other words, are duplicated words always next to each other? –  Daniel Beck Jul 26 '12 at 21:24

1 Answer 1

up vote 4 down vote accepted

You can use a regular expression to remove consecutive duplicated words in a line, however I don't think it's possible to remove duplicated words which are not consecutive (e.g dangerous, hazardous, dangerous).

Use this regex in the replace window in Notepad++, and don't forget to select "Regular expression" as the Search Mode option below:

This regex will remove all consecutive duplicated words - whether it's 2 duplicated words or 10 duplicated words consecutively: \b(\w+)(?:,\s+\1\b)+.

The exact same no-commas regex would be: \b(\w+)(?:\s+\1\b)+ (might be useful for other users).

If you want a regex specifically for only two duplicated words (doubles), use this regex: (\b\w+\b)\W+\1.

Place this regex in the Replace with box to keep one occurrence of the word (otherwise all repeated words will be removed): ${1}.

These regular expressions will fix a situation like the one you described in your question as an example. The first regex will work for every number of duplicated words (e.g dangerous, dangerous, dangerous, dangerous, hazardous), while the second version will only work for two duplicated words (e.g dangerous, dangerous, hazardous).

Note: The regular expression will only apply to the format described in the question, meaning that formats like two words, two words, anotherword, two-words, two-words, anotherword, three words expression, three words expression, anotherword won't be changed because the regex won't apply to them.

share|improve this answer
    
Thanks for the help! However I'm getting 0 occurrences, I tried doing this with separated keywords as you suggested and it didn't work, I also tried as they were before and nothing, please check my screen capture: goo.gl/eZ7Kp –  Gabriel Jul 26 '12 at 20:28
    
This regex should work: (\b\w+\b)\W+\1 for two duplicated words. I'll edit my answer. The commas are why the other regex doesn't work. –  amiregelz Jul 26 '12 at 20:40
    
I tried every possible combination, no commas, only spaces, no space and comma, and yet nothing. Please enlight me, here's the text file: goo.gl/sP20z –  Gabriel Jul 26 '12 at 21:59
    
The problem is that the regular expression I wrote in my answer only applies to the format (I thought) you asked for: word, word, anotherword. However, you have many instances which are a little bit different, like came across, came across and some with 3 or 4 words. Also there are words with ' like don't and it makes things more complicated in the Notepad++ regex system. The Notepad++ regex system is pretty annoying and limited as well, so the solution is to either use regex in python (or another language), or make format-specific regular expressions for the Notepad++. –  amiregelz Jul 26 '12 at 23:48
    
Another problem is that most of the words that are duplicated also appear in the previous line, which makes it difficult to achieve your goal. If you'd want to remove all duplicated words, then it wasn't that difficult. You could do something like this & this. I suggest you use specific regular expressions in Notepad++ (I can help you, just tell me all the formats of the duplicated words) or consider a different approach to your problem. –  amiregelz Jul 27 '12 at 0:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.