Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have two text files on Linux. One contains a list of valid IDs. E.g:

abcd
efgh
ijkl
etc.

The other contains a list of invalid IDs. But, some of these also appear on the list of valid IDs, in this example "efgh":

mnop
qrst
efgh
etc.

How can I easily construct a text file that contains all the lines from the invalid list that do not appear in the valid list? That is, I want to end up with a text file that has:

mnop
qrst
etc.

I'd like either some Linux commandline magic of some clever Vim trickery. Thanks in advance!

share|improve this question
add comment

3 Answers

up vote 8 down vote accepted

I think comm can help you do what you want.

It is a less known command line tool that shall be present in most Linux systems.

share|improve this answer
1  
Just what I needed. I used comm -2 -3 invalid.txt valid.txt. –  Sander Marechal Jul 7 '10 at 8:53
add comment

Assuming you have files named valid and invalid, you can sort them:

sort valid > valid-sorted; sort invalid > invalid-sorted

then compare:

diff valid-sorted invalid-sorted

The output will be (assuming your example data):

1d0
< abcd
3c2,3
< ijkl
---
> mnop
> qrst

The stuff with '<' is only in the first file, the stuff with '>' only in the second (invalid).

Thus diff valid-sorted invalid-sorted |grep '^>' should give you the strings appearing in invalid but not in valid.

Stripping out the leading '>' is left as an exercise for the reader ;-).

Caveat: This will not work if your input files actually contain '>' at the start of a line. You'll have to do some tricking then...

share|improve this answer
add comment

I have my own utility for that, which relies on another; they also work on unsorted input. Basic usage is

except file1 file2 > file2-except-any-lines-in-file1
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.