Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a text file which contains about 50MB of text:

valA
valB
valC
valZ
valB

Whats the quickest way to:

  1. cat the file
  2. Display the unique appearences
share|improve this question

migrated from stackoverflow.com Sep 16 '12 at 8:05

This question came from our site for professional and enthusiast programmers.

1  
You don't need to cat the file. You can just do uniq file.txt > uniq.txt which will probably be fastest instead of stdout. – squiguy Sep 14 '12 at 0:27
    
seems like 'cat text.txt | uniq -c | sort -n ' may be the answer – Clustermagnet Sep 14 '12 at 0:28
2  
@Clustermagnet: You'd want to do sort first, then call uniq. Useless use of cat. Try: < text.txt sort -n | uniq -c – Steve Sep 14 '12 at 0:30
1  
@Clustermagnet: UUOC Award? It isn't clear that counts are wanted or needed, so sort -u text.txt? If counts are wanted, then sort text.txt | uniq -c works, optionally followed by sort -n to put the lines into frequency order. – Jonathan Leffler Sep 14 '12 at 0:31
1  
@squiguy: the file must be sorted before uniq works sanely. The input clearly isn't sorted. – Jonathan Leffler Sep 14 '12 at 0:32
up vote 3 down vote accepted

One way using awk:

awk '!array[$0]++' file.txt

Results:

valA
valB
valC
valZ

Food for thought:

sort -u file.txt

valA
valB
valC
valZ

< file.txt sort | uniq

valA
valB
valC
valZ

< file.txt sort | uniq -u                 # only print unique lines

valA
valC
valZ

< file.txt sort | uniq -d                 # only print duplicate lines

valB
share|improve this answer

sort text.txt | uniq -c | grep "^\s*1 "

  • you have to sort before uniq
  • grep those ones of count=1

notice valB is not unique here.

Edit

How could I miss there is -u for uniq: uniq -u yet use this silly way:)

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .