Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have huge text files with two fields, the first is a string the second is an integer. The files are sorted by the first field. What I'd like to get in the output is one line per unique string and the sum of the numbers for the identical strings. Some strings appear only once while other appear multiple times. E.g. Given the sample data below, for the string glehnia I'd like to get 10+22=32 in the result.

Any suggestions how to do this either with gnuwin32 command line tools or in linux shell?


glehnia 10
glehnia 22
glehniae 343
glehnii 923
glei 1171
glei 2283
glei 3466
gleib 914
gleiber 652
gleiberg 495
gleiberg 709

share|improve this question
Duplicate question at with similar answers, but the top-voted answer there is more concise than the accepted answer below (both are awk programs). – chbrown Nov 18 '15 at 20:34
up vote 3 down vote accepted

In AWK, you could do something like this:

awk '($1 == last) || (last == "") {sum += $2}
     ($1 != last) && (last != "") {print last " " sum; sum = $2}
                                  {last = $1}
     END                          {print last " " sum}' huge_text_file.txt
share|improve this answer
Thanks so much! I had to tweak a little to use TAB as delimiters but it got me the kickstart! – dnkb May 2 '10 at 18:54
@dnkb: Using a comma in your print statements allows you to set the output field separator and then print will use it automatically. This may be what you did: awk 'BEGIN {OFS="\t"} ($1 == last || last == "") {sum += $2} ($1 != last && last != "") {print last, sum; sum = $2} {last = $1} END {print last, sum}' huge_text_file.txt – Dennis Williamson May 2 '10 at 20:29

You could a use a few lines of Lua to acheive this. Lua is available on a wide range of platforms including Windows and Linux.

-- Quick and dirty - no error checking, unsorted output


results = {}

for line in io.lines() do
    for text, number in string.gmatch(line, '(%w+)%s+(%d+)') do
        results[text] = (results[text] or 0) + number

for text, number in pairs(results) do
    print(text, number)

You can sort the output using any sort utility or a few more lines of Lua.

share|improve this answer
Thanks, I appreciate it. – dnkb May 2 '10 at 18:55

This type of queries are most easily and flexibly done in SQL. You can use SQLet and SQLite for that purpose: -d' ' -A file1.txt 'select a1, sum(a2) from A group by a1' | sqlite3
share|improve this answer

You can use uniq command, for example

cat file | uniq -u -c

where: -c - the number of occurrences -u - print only unique lines

share|improve this answer
did you read the question? – Tomas Jun 30 '14 at 9:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .