How can I get the unique items in a bash list?

Question

If I have a list in bash

L="11 22 11 33 22 44"

Is there a clean way to get to "11 22 33 44" ?

Answer 1

One way:

$ L="11 22 11 33 22 44"
$ L=$(echo $L | tr ' ' '\n' | sort -nu)    
$ echo $L
11 22 33 44

Using tr, the numbers are split to individual lines, and sort is used to sort the numbers(-n) and remove the duplicate ones(-u).

Answer 2

A common pattern in some other programming languages is to store each value as a 'key' in a hash or dictionary or associative array, and rely on the implementation to keep hash keys unique.

declare -A items=()
for item in $L; do
    items[$item]=1
done
L=${!items[*]}

(Unfortunately, the bash implementation is about 2× slower than sort -u.)

Answer 3

Extending on Gurus answer above, if you have an array you can:

$ L=(11 22 11 33 22 44)
$ echo ${L[@]}
11 22 11 33 22 44
$ L=($(echo ${L[@]} | tr ' ' '\n' | sort -nu))
$ echo ${L[@]}
11 22 33 44

Answer 4

Another one I've found useful for checking the list of unique IPs connecting to your servers, from the 'netstat' command is this:

Netstat Output Example

tcp 0 0 192.168.1.196:8009 192.168.21.124:45921 TIME_WAIT
tcp 0 0 192.168.7.196:8009 192.168.21.124:51579 ESTABLISHED
tcp 0 0 192.168.7.196:8009 192.168.20.12:20213 TIME_WAIT

Command Example

In this example I'm starting by filtering the TCP port, cutting out the local information, sorting the list and then piping it to the 'uniq' command.

netstat -an | grep ":8009" | cut -d " " -f 28 | cut -d ":" -f 1 | sort -u

What I get is a list of the 2 unique foreign IP addresses connecting to my system.

192.168.21.124
192.168.21.12

If you did some fancy monitoring stuff, you can graph the total number of unique nodes connecting, compare the list to your white list of allowed IPs, add non-authorized IPs to your black list, etc.

Answer 5

I guess this version beats all the rest! Mainly because of no need for eval in bash

$ L="11 22 11 33 22 44"
$ readarray -t L < <(printf "%s\n" ${L[@]} | sort -u)
$ printf "%s\n" ${L[@]}
11
22
33
44