41

I need to perform 2 operations in Sublime Text: extract unique lines and extract duplicate lines. For example for input

a
b
a

Extract duplicates should result in:

a

and Extract unique should result in:

b

Is there a built-in operation or a plugin to do that?

5 Answers 5

57

You can find duplicate lines easily by running a Sort Lines then searching for this regex that uses line boundary markers ^ and $ and the back reference \1.

^(.+)$\n^\1$

Follow that with a Find All, Copy, Paste in a new tab, Permute Lines | Unique and you've extracted them.

4
  • This is amazing: I added a small addition that helped me: 1. Run the regex 2. Replace all the matched values with: \t$1, which will indent all the matched values and keep only 1 instance of them in the file 3. Run another regex: ^\d.*$ and replace the \d with your relevant tag 4. This will keep only the duplicated values
    – Oz Radiano
    Jul 24, 2018 at 6:58
  • Great and elegant regex solution. TBH I was looking for a Sublime Package to do this. But this simple search was simpler and enough for my needs. Thank you.
    – Armando
    May 11, 2021 at 2:22
  • Purely genius...
    – Artfaith
    Jul 21, 2021 at 19:28
  • I sometimes forget to "think outside the box".
    – Kaymaz
    Aug 30, 2021 at 12:47
12

Unfortunately I don't have access to Sublime Text at the moment, so I'm not able to test this, but I believe something like the following might work for you:

  1. Sort the lines via the Edit -> Sort Lines command
  2. Install the Highlight Duplicates plugin, and use it to highlight all the duplicate lines
  3. Cut the highlighted lines to the Clipboard, and paste them into a New File
  4. The lines that remain in the original file are your Extract Unique lines
  5. In the New File, select all the text, and remove duplicate lines via the Edit -> Permute Lines -> Unique command
  6. The lines that remain in the New File are your Extract Duplicates lines

I'm not entirely sure that step #1 is actually necessary, but I included it just in case.

1
  • 1
    I wondered the same thing and just tried it (Sublime 3.0, here)... sorting first is not necessary. (Unlike with Unix 'uniq'.) Nice.
    – Tom Hundt
    Feb 25, 2018 at 18:30
2

Had the same problem (show me the dupes)... didn't find an easy Sublime-based answer and fell back to using Unix commands (my file had the data I wanted to find the duplicates of in columns 11-56):

cut -c 11-56 myfile.dat | sort | uniq -d

Posted here as an FYI to others.

1
2

I found the easiest way to do this with Sublime Text was to just sort lines (f5 on mac), permute lines > unique, then view the diff with git.

0

Slightly modified @MJH answer above to get duplicated lines with Sublime 3 and DiffMerge, without using Highlight Duplicates plugin.

  1. Sort the lines via Sublime 3 Edit -> Sort Lines command
  2. Save original file as sorted_orig.txt
  3. Select all the text, and remove duplicate lines via Sublime 3 Edit -> Permute Lines -> Unique command
  4. Save modified file as no_dup_sorted.txt
  5. Start diff with DiffMerge tool with sorted_orig.txt and no_dup_sorted.txt files.
  6. Use Export -> File Diffs in DiffMerge to get a list of duplicates in clipboard or save to another file.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .