up vote 0 down vote favorite
share [g+] share [fb]

Can someone recommend a fast unix-based utility (ideally written in C) for getting efficient, streaming intersection/union of tab-delimited text files? For example, allow queries such as "give me the all the entries that in file A that have a column value K that does not appear in any column K of file B".

e.g., if file A is:

bob sally sue
bob mary john

and file B is:

john sally sue
foo bar quux

then complement of file A relative to B on column 2 would return "bob mary john", since that's the only in file B that has a value in column 2 that does not appear in file B.

I'd prefer not to use a database, but would like a command line based utility. Is awk the answer or is there something simpler? thanks.

link|improve this question
70% accept rate
I suggest python; reading the file, splitting on whitespace to a list, making it a set and doing set operations before writing out. – Eroen Dec 4 '11 at 21:06
@Eroen The user mentions speed and memory efficiency. Python and set operations is anything but. – Daniel Beck Dec 4 '11 at 21:20
1  
This page describes a number of set operations on the unix command line. I'm not sure how you want your columns to be, but if they are not important a quick awk script would merge them to a single col. – Eroen Dec 5 '11 at 0:53
I've read your question, three times by now, even the example you gave does not help me very much. I've done joins and exclusive search on columns multiple times with AWK, but, sorry please help me to understand your problem. E.g. "bob mary john" isn't a column, its a row, or column value K, and column K, makes no sense to me. – math Dec 9 '11 at 11:35
@math That part is quite easily solved by just transposing the contents of file B, and then comparing columns instead. I'm no awk ninja, but I'd be surprised if awk couldn't perform a matrix transposition. – Martin Frost Dec 12 '11 at 13:48
feedback

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
or
required, but never shown
discard

Browse other questions tagged or ask your own question.