Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a 400 megabyte file. The file is a list of words, each word on a single line. End of lines are a mix of cr/lf and lf. Some of the words are unicode.

I'd prefer answers for Linux or Windows please. I'm happy to install software, but I'd prefer open source.

I need to remove all the words that are less than 8 characters. I then need to de-duplicate the list. (Sorting the list is nice, but not need unless it's part of the process.) I don't care about the unicode words - they can stay or go. Stripping them out is fine.

To remove all the words that are 7 characters or less I use awk:

awk "length($0) > 7" wordlist.txt > mynewwordlist.txt

(This is Windows, sorry for the " instead of ')

I then try to use sort, because I can then use awk again to de-dupe lines. Sort didn't like the wordlist. I think (but I'm not sure) that the unicode broke it.

So, how do I de-dupe the wordlist? Or how do I sort the wordlist so that I can use awk to de-dupe?

share|improve this question
up vote 6 down vote accepted

To dedupe, you need to sort -u – it will only output unique lines. This is an option specified by POSIX, so you'll probably find it on any system.

sort -u wordlist.txt | awk 'length($0) > 7'

Use double quotes for awk if you're on Windows. Note that the sorting step is not optional, as uniq requires duplicate lines to be adjacent in order to remove them.

If you have Unicode problems, this could be a problem of your locale. You can set LC_ALL=C to force the native byte value comparison – the actual sorting order doesn't matter for you anyway. Or, you could use iconv to convert the file from Unicode to another encoding if that's the issue.

share|improve this answer
sort has a -u ( --unique ) option, at least on cygwin. – Nifle Aug 31 '13 at 14:56
You're right, I thought it wasn't POSIX, will correct my post! – slhck Aug 31 '13 at 15:07

In case someone needs to do this without having access to awk, you can also use grep for selecting words of at least a given length:

sort -u   wordlist.txt | grep '........'
share|improve this answer
Cool hack. I wouldn't have thought of doing something like that. – davidgo Aug 31 '13 at 19:48
Thank you! It's great to see versatility in answers. – DanBeale Sep 2 '13 at 10:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .