Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Is there a fast Unix-based utility (ideally written in C) for getting efficient, streaming intersection/union of tab-delimited text files?

For example, allow queries such as "give me the all the entries that in file A that have a column value K that does not appear in any column K of file B".

For example, if file A is:

bob sally sue
bob mary john

and file B is:

john sally sue
foo bar quux

then the complement of file A relative to B on column 2 would return "bob mary john", since that's the only in file B that has a value in column 2 that does not appear in file B.

I'd prefer not to use a database, but I would like a command-line based utility. Is AWK the answer or is there something simpler?

share|improve this question
    
I suggest python; reading the file, splitting on whitespace to a list, making it a set and doing set operations before writing out. –  Eroen Dec 4 '11 at 21:06
1  
@Eroen The user mentions speed and memory efficiency. Python and set operations is anything but. –  Daniel Beck Dec 4 '11 at 21:20
2  
This page describes a number of set operations on the unix command line. I'm not sure how you want your columns to be, but if they are not important a quick awk script would merge them to a single col. –  Eroen Dec 5 '11 at 0:53
1  
I've read your question, three times by now, even the example you gave does not help me very much. I've done joins and exclusive search on columns multiple times with AWK, but, sorry please help me to understand your problem. E.g. "bob mary john" isn't a column, its a row, or column value K, and column K, makes no sense to me. –  math Dec 9 '11 at 11:35
    
@math That part is quite easily solved by just transposing the contents of file B, and then comparing columns instead. I'm no awk ninja, but I'd be surprised if awk couldn't perform a matrix transposition. –  Martin Frost Dec 12 '11 at 13:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.