Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have two text files on Linux. One contains a list of valid IDs. E.g:

abcd
efgh
ijkl
etc.

The other contains a list of invalid IDs. But, some of these also appear on the list of valid IDs, in this example "efgh":

mnop
qrst
efgh
etc.

How can I easily construct a text file that contains all the lines from the invalid list that do not appear in the valid list? That is, I want to end up with a text file that has:

mnop
qrst
etc.

I'd like either some Linux commandline magic of some clever Vim trickery. Thanks in advance!

share|improve this question
up vote 9 down vote accepted

I think comm can help you do what you want.

It is a less known command line tool that shall be present in most Linux systems.

share|improve this answer
1  
Just what I needed. I used comm -2 -3 invalid.txt valid.txt. – Sander Marechal Jul 7 '10 at 8:53

Assuming you have files named valid and invalid, you can sort them:

sort valid > valid-sorted; sort invalid > invalid-sorted

then compare:

diff valid-sorted invalid-sorted

The output will be (assuming your example data):

1d0
< abcd
3c2,3
< ijkl
---
> mnop
> qrst

The stuff with '<' is only in the first file, the stuff with '>' only in the second (invalid).

Thus diff valid-sorted invalid-sorted |grep '^>' should give you the strings appearing in invalid but not in valid.

Stripping out the leading '>' is left as an exercise for the reader ;-).

Caveat: This will not work if your input files actually contain '>' at the start of a line. You'll have to do some tricking then...

share|improve this answer

I have my own utility for that, which relies on another; they also work on unsorted input. Basic usage is

except file1 file2 > file2-except-any-lines-in-file1
share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .