I am looking for a way in VIM to delete all lines that are duplicates and only leave unique lines that exist in the file. I would prefer a macro but a command or function would be great.

Say I have a file that has duplicate lines and some uniques:


I want to delete all lines so that all that is left:


The one unique line that I REALLY want.

I use sort u all the time to get a unique list by deleting duplicates but I have times I just need the unique line in the file.

Any ideas how to handle this case in VIM?

  • Use the right tool to do this type of work....use uniq which is used for this type of thing. – mdpc Feb 1 '13 at 3:06

You can do this with the substitute command and a backreference:


Note that this only works if the lines are sorted. This works by matching and removing the first group one or more times on the next line(s).

  • Since I do not have Uniq, I did a macro in VIM to loop over and delete non uniques. Thanks. – 7null Feb 5 '13 at 19:43

You could use the "uniq" command to do this -

uniq -u filename-to-check.txt
  • and you could even run it as a filter inside Vim ... would be cool if your answer would contain that ... – 0xC0000022L Feb 1 '13 at 2:40
  • 1
    :%!sort | uniq -u will do just that: sort, remove all lines that are not unique, and leave the result in the file. – andersonvom Jun 12 '17 at 23:36
  1. Select all the lines you want to filter.

  2. Do :'<,'>!uniq -u<CR>, the '<,'> range is added automatically for you.

If you want to filter the whole buffer, don't select anything and simply do :%!uniq -u<CR>.

You can read about filters in :h filter.

  • I take it uniq is not an option in windows 7? – 7null Feb 1 '13 at 23:44
  • I have no idea. Maybe with something like Cygwin? You should have indicated that. All the answers here are useless if you don't have uniq on your machine. – romainl Feb 2 '13 at 0:00
  • 2
    Yes, I should have mentioned windows. That was one reason I asked about a macro to try and keep it in vim. I was able to write a macro around the recommended s/^(.*)(\n\1)\+// and it worked ok for 50k lines. – 7null Feb 2 '13 at 0:09

You can invoke the uniq tool as a filter directly from Vim:

:!%uniq -u

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.