0

On Linux I have two text files with russian UTF8 words sorted with sort -u (actually I used :%sort u in Vim, it produces same results).

One of the files dict.txt contains around 700000 words of my custom dictionary. Another file bad-words.txt contains often mistyped words.

I'd like to remove all words found in bad-words.txt from dict.txt.

I know that a perl script using a hash could do that, but I'm after a Unix one liner.

Ist there please a Unix command (maybe something using diff?) to perform this task? And I hope diff won't be confused by the large number of lines - because "Beyond compare" program on Windows is...

Improve this question

1 Answer 1

Reset to default
3

comm will do this.

comm -3 dict badwords

or to sort on the fly:

comm -3 <(sort dict-input) <(badwords)

Improve this answer
3
  • Unfortunately it complains comm: file 1 is not in sorted order (despite me presorting it) and produces strange 2-columns output...
    – Alexander Farber
    Mar 20, 2013 at 21:28
  • 2
    comm expects the file to be sorted in ASCIIbetical order, e.g. LANG=C sort.
    – u1686_grawity
    Mar 20, 2013 at 22:43
  • @grawity: That advice saved my day :) Thanks!
    – Sebastian Łaskawiec
    Sep 9, 2014 at 9:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .