I have a 400 megabyte file. The file is a list of words, each word on a single line. End of lines are a mix of cr/lf and lf. Some of the words are unicode.
I'd prefer answers for Linux or Windows please. I'm happy to install software, but I'd prefer open source.
I need to remove all the words that are less than 8 characters. I then need to de-duplicate the list. (Sorting the list is nice, but not need unless it's part of the process.) I don't care about the unicode words - they can stay or go. Stripping them out is fine.
To remove all the words that are 7 characters or less I use awk:
awk "length($0) > 7" wordlist.txt > mynewwordlist.txt
(This is Windows, sorry for the " instead of ')
I then try to use sort, because I can then use awk again to de-dupe lines. Sort didn't like the wordlist. I think (but I'm not sure) that the unicode broke it.
So, how do I de-dupe the wordlist? Or how do I sort the wordlist so that I can use awk to de-dupe?