Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a sorted file with lines like this

word1  abca
word1  abcb
word1  abcc
word2  abca
word2  abcb
word3  abbb
...........

and i want to have something like this

word1  abca
word2  abca
word3  abbb
...........
share|improve this question
    
You need to provide more details. Do you only need the first entry for each word1, word2...? –  Gh0sT Jun 7 at 9:16
    
it doesn't matter if it's the first entry or the second or the next one and so one, it has to be only one line with word1 or word2 etc –  anon123 Jun 7 at 18:35

3 Answers 3

up vote 1 down vote accepted

This magic incantation is a famous awk idiom:

awk '!seen[$1]++' file

The first time a line with that $1 is seen, the line is printed.

share|improve this answer

An awk solution - using a variable to detect a new word. If a new word is found, print the line and assign the variable to the current word.

As the data file is sorted, only the 1st occurence of each word will print the record.

   awk 'BEGIN{w=""} w!=$1 {print;w=$1}' your-file
share|improve this answer

You could also use the -w flag of uniq which tells it to only compare the first N characters. The details of whether this will work for you depend on your actual data but if the word lengths are set or limited, it should work:

$ sort file.txt | uniq -w 5
word1  abca
word2  abca
word3  abbb

Alternatively, reverse the order of the fields and use uniq -f 1 to skip comparing the 1st field:

$ awk '{print $2,$1}' file.txt | uniq -f 1 | awk '{print $2,$1}'
word1 abca
word2 abca
word3 abbb

Or get the 1st fields and then grep for them, limiting the search to the first match:

$ for i in $(awk '{print $1}' file.txt | sort -u); do grep -m 1 $i file.txt; done
word1  abca
word2  abca
word3  abbb

And, for completion's sake, a Perl one:

$ perl -ane 'print if $k{$F[0]}++<1' file.txt 
word1  abca
word2  abca
word3  abbb
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.