Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

On Linux I have two text files with russian UTF8 words sorted with sort -u (actually I used :%sort u in Vim, it produces same results).

One of the files dict.txt contains around 700000 words of my custom dictionary. Another file bad-words.txt contains often mistyped words.

I'd like to remove all words found in bad-words.txt from dict.txt.

I know that a perl script using a hash could do that, but I'm after a Unix one liner.

Ist there please a Unix command (maybe something using diff?) to perform this task? And I hope diff won't be confused by the large number of lines - because "Beyond compare" program on Windows is...

share|improve this question
add comment

1 Answer 1

comm will do this.

comm -3 dict badwords

or to sort on the fly:

comm -3 <(sort dict-input) <(badwords)

share|improve this answer
    
Unfortunately it complains comm: file 1 is not in sorted order (despite me presorting it) and produces strange 2-columns output... –  Alexander Farber Mar 20 '13 at 21:28
1  
comm expects the file to be sorted in ASCIIbetical order, e.g. LANG=C sort. –  grawity Mar 20 '13 at 22:43
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.