How can I sort and de-dupe a very long list of words?

Question

I have a 400 megabyte file. The file is a list of words, each word on a single line. End of lines are a mix of cr/lf and lf. Some of the words are unicode.

I'd prefer answers for Linux or Windows please. I'm happy to install software, but I'd prefer open source.

I need to remove all the words that are less than 8 characters. I then need to de-duplicate the list. (Sorting the list is nice, but not need unless it's part of the process.) I don't care about the unicode words - they can stay or go. Stripping them out is fine.

To remove all the words that are 7 characters or less I use awk:

awk "length($0) > 7" wordlist.txt > mynewwordlist.txt

(This is Windows, sorry for the " instead of ')

I then try to use sort, because I can then use awk again to de-dupe lines. Sort didn't like the wordlist. I think (but I'm not sure) that the unicode broke it.

So, how do I de-dupe the wordlist? Or how do I sort the wordlist so that I can use awk to de-dupe?

Answer 1

To dedupe, you need to sort -u – it will only output unique lines. This is an option specified by POSIX, so you'll probably find it on any system.

sort -u wordlist.txt | awk 'length($0) > 7'

Use double quotes for awk if you're on Windows. Note that the sorting step is not optional, as uniq requires duplicate lines to be adjacent in order to remove them.

If you have Unicode problems, this could be a problem of your locale. You can set LC_ALL=C to force the native byte value comparison – the actual sorting order doesn't matter for you anyway. Or, you could use iconv to convert the file from Unicode to another encoding if that's the issue.

Answer 2

In case someone needs to do this without having access to awk, you can also use grep for selecting words of at least a given length:

sort -u   wordlist.txt | grep '........'