Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am looking for a way in VIM to delete all lines that are duplicates and only leave unique lines that exist in the file. I would prefer a macro but a command or function would be great.

Say I have a file that has duplicate lines and some uniques:

1Apple
1Apple
2Peach
2Peach
2Peach
3Beer
4Banana
4Banana
4Banana

I want to delete all lines so that all that is left:

3Beer

The one unique line that I REALLY want.

I use sort u all the time to get a unique list by deleting duplicates but I have times I just need the unique line in the file.

Any ideas how to handle this case in VIM?

share|improve this question
    
Use the right tool to do this type of work....use uniq which is used for this type of thing. –  mdpc Feb 1 '13 at 3:06

4 Answers 4

up vote 1 down vote accepted

You can do this with the substitute command and a backreference:

s/^\(.*\)\(\n\1\)\+//

Note that this only works if the lines are sorted. This works by matching and removing the first group one or more times on the next line(s).

share|improve this answer
    
Since I do not have Uniq, I did a macro in VIM to loop over and delete non uniques. Thanks. –  7null Feb 5 '13 at 19:43

You could use the "uniq" command to do this -

uniq -u filename-to-check.txt
share|improve this answer
    
and you could even run it as a filter inside Vim ... would be cool if your answer would contain that ... –  0xC0000022L Feb 1 '13 at 2:40
  1. Select all the lines you want to filter.

  2. Do :'<,'>!uniq -u<CR>, the '<,'> range is added automatically for you.

If you want to filter the whole buffer, don't select anything and simply do :%!uniq -u<CR>.

You can read about filters in :h filter.

share|improve this answer
    
I take it uniq is not an option in windows 7? –  7null Feb 1 '13 at 23:44
    
I have no idea. Maybe with something like Cygwin? You should have indicated that. All the answers here are useless if you don't have uniq on your machine. –  romainl Feb 2 '13 at 0:00
1  
Yes, I should have mentioned windows. That was one reason I asked about a macro to try and keep it in vim. I was able to write a macro around the recommended s/^(.*)(\n\1)\+// and it worked ok for 50k lines. –  7null Feb 2 '13 at 0:09

You can invoke the uniq tool as a filter directly from Vim:

:!%uniq -u
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.