Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a text file which contains about 50MB of text:

valA
valB
valC
valZ
valB

Whats the quickest way to:

  1. cat the file
  2. Display the unique appearences
share|improve this question

migrated from stackoverflow.com Sep 16 '12 at 8:05

This question came from our site for professional and enthusiast programmers.

1  
You don't need to cat the file. You can just do uniq file.txt > uniq.txt which will probably be fastest instead of stdout. –  squiguy Sep 14 '12 at 0:27
    
seems like 'cat text.txt | uniq -c | sort -n ' may be the answer –  Clustermagnet Sep 14 '12 at 0:28
2  
@Clustermagnet: You'd want to do sort first, then call uniq. Useless use of cat. Try: < text.txt sort -n | uniq -c –  Steve Sep 14 '12 at 0:30
1  
@Clustermagnet: UUOC Award? It isn't clear that counts are wanted or needed, so sort -u text.txt? If counts are wanted, then sort text.txt | uniq -c works, optionally followed by sort -n to put the lines into frequency order. –  Jonathan Leffler Sep 14 '12 at 0:31
1  
@squiguy: the file must be sorted before uniq works sanely. The input clearly isn't sorted. –  Jonathan Leffler Sep 14 '12 at 0:32

2 Answers 2

up vote 3 down vote accepted

One way using awk:

awk '!array[$0]++' file.txt

Results:

valA
valB
valC
valZ

Food for thought:

sort -u file.txt

valA
valB
valC
valZ

< file.txt sort | uniq

valA
valB
valC
valZ

< file.txt sort | uniq -u                 # only print unique lines

valA
valC
valZ

< file.txt sort | uniq -d                 # only print duplicate lines

valB
share|improve this answer

sort text.txt | uniq -c | grep "^\s*1 "

  • you have to sort before uniq
  • grep those ones of count=1

notice valB is not unique here.

Edit

How could I miss there is -u for uniq: uniq -u yet use this silly way:)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.