0

On Linux I have two text files with russian UTF8 words sorted with sort -u (actually I used :%sort u in Vim, it produces same results).

One of the files dict.txt contains around 700000 words of my custom dictionary. Another file bad-words.txt contains often mistyped words.

I'd like to remove all words found in bad-words.txt from dict.txt.

I know that a perl script using a hash could do that, but I'm after a Unix one liner.

Ist there please a Unix command (maybe something using diff?) to perform this task? And I hope diff won't be confused by the large number of lines - because "Beyond compare" program on Windows is...

3

comm will do this.

comm -3 dict badwords

or to sort on the fly:

comm -3 <(sort dict-input) <(badwords)

  • Unfortunately it complains comm: file 1 is not in sorted order (despite me presorting it) and produces strange 2-columns output... – Alexander Farber Mar 20 '13 at 21:28
  • 2
    comm expects the file to be sorted in ASCIIbetical order, e.g. LANG=C sort. – grawity Mar 20 '13 at 22:43
  • @grawity: That advice saved my day :) Thanks! – Sebastian Łaskawiec Sep 9 '14 at 9:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.