Substract contents of one textfile from contents of another textfile on Linux

Question

I have two text files on Linux. One contains a list of valid IDs. E.g:

abcd
efgh
ijkl
etc.

The other contains a list of invalid IDs. But, some of these also appear on the list of valid IDs, in this example "efgh":

mnop
qrst
efgh
etc.

How can I easily construct a text file that contains all the lines from the invalid list that do not appear in the valid list? That is, I want to end up with a text file that has:

mnop
qrst
etc.

I'd like either some Linux commandline magic of some clever Vim trickery. Thanks in advance!

Answer 1

I think comm can help you do what you want.

It is a less known command line tool that shall be present in most Linux systems.

Answer 2

Assuming you have files named valid and invalid, you can sort them:

sort valid > valid-sorted; sort invalid > invalid-sorted

then compare:

diff valid-sorted invalid-sorted

The output will be (assuming your example data):

1d0
< abcd
3c2,3
< ijkl
---
> mnop
> qrst

The stuff with '<' is only in the first file, the stuff with '>' only in the second (invalid).

Thus diff valid-sorted invalid-sorted |grep '^>' should give you the strings appearing in invalid but not in valid.

Stripping out the leading '>' is left as an exercise for the reader ;-).

Caveat: This will not work if your input files actually contain '>' at the start of a line. You'll have to do some tricking then...

Answer 3

I have my own utility for that, which relies on another; they also work on unsorted input. Basic usage is

except file1 file2 > file2-except-any-lines-in-file1