Listing all words in a text file and finding the most frequent word

Question

I have a file which contains lines.

How do I write a script that will find and print out every word in the file, one word per line.

Then find and print out the most occurring word (case sensitive) and the number of occurrences of that word in the file.

Answer 1

A shell oneliner:

cat file.txt | sed -r 's/[[:space:]]+/\n/g' | sed '/^$/d' | sort | uniq -c | sort -n | tail -n1

Remove punctuation before counting words and make words lowercase (in English):

cat file.txt | tr 'A-Z' 'a-z' | sed 's/--/ /g' | sed 's/[^a-z ]//g' | tr -s '[[:space:]]' '\n' | sort | uniq -c | sort -n | tail -n1

Answer 2

 sed -e 's/[^[:alpha:]]/ /g' text_to_analize.txt | tr '\n' " " |  tr -s " " | tr " " '\n'| tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr | nl 

This command makes the following:

1. Substitute all non alphanumeric characters with a blank space.
2. All line breaks are converted to spaces also.
3. Reduces all multiple blank spaces to one blank space
4. All spaces are now converted to line breaks. Each word in a line.
5. Translates all words to lower case to avoid 'Hello' and 'hello' to be different words
6. Sorts de text
7. Counts and remove the equal lines
8. Sorts reverse in order to count the most frequent words
9. Add a line number to each word in order to know the word posotion in the whole

For example if I want to analize the first Linus Torvald message:

From: [email protected] (Linus Benedict Torvalds) Newsgroups: comp.os.minix Subject: What would you like to see most in minix? Summary: small poll for my new operating system Message-ID: <[email protected]> Date: 25 Aug 91 20:57:08 GMT Organization: University of Helsinki

Hello everybody out there using minix –

I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I’d like any feedback on things people like/dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things).

I’ve currently ported bash(1.08) and gcc(1.40), and things seem to work. This implies that I’ll get something practical within a few months, and I’d like to know what features most people would want. Any suggestions are welcome, but I won’t promise I’ll implement them 🙂

Linus ([email protected])

PS. Yes – it’s free of any minix code, and it has a multi-threaded fs. It is NOT protable (uses 386 task switching etc), and it probably never will support anything other than AT-harddisks, as that’s all I have :-(.

I create a file named linus.txt, I paste the content and then I write in the console:

sed -e 's/[^[:alpha:]]/ /g' linus.txt | tr '\n' " " |  tr -s " " | tr " " '\n'| tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr | nl 

The out put would be:

 1        7 i
 2        5 to
 3        5 like
 4        5 it
 5        5 and
 6        4 minix
 7        4 a
 8        3 torvalds
 9        3 of
10        3 helsinki
11        3 fi
12        3 any
13        2 would
14        2 won
15        2 what
16        ...

If you want to visualize only the first 20 words:

sed -e 's/[^[:alpha:]]/ /g' text_to_analize.txt | tr '\n' " " |  tr -s " " | tr " " '\n'| tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr | nl | head -n 20

It's important to note that the command tr 'A-Z' 'a-z' doesn't suport UTF-8 yet, so that in foreign languages the word APRÈS would be translated as aprÈs.

Answer 3

#!/bin/bash
cat /dev/stdin
printf "\n"
sort | uniq -c | sort -nr

This simple script will act as a word frequency counter just by using sort and uniq and piping them together. First it prints from the stdin using cat to show the input. Then it prints a newline. Lastly it sorts stdin, counts the number of unique words with uniq -c, then sorts the list again but with the n and r options to order the list numerically and reverse the list so that the most frequent words appear first. Since it reads from the standard input stream call it like this: script < inputfile.

Answer 4

That should give you a start to work with :

#!/usr/bin/perl
use strict;
use warnings;

#Read the file
open my $in, ';
close $in;

#Split the lines of the file into an array of words
my @words;
foreach my $line (@lines)
{
    push @words, (split(/[^\W]/, $line));
}

#Count the occurrences of each word (to evolve into a MapReduce fashion if the file is tremendously big)
my %word_count;
foreach my $word (@words)
{
    $word_count{$word}++;
}

#Find the word with the most occurrences
my $most_frequent_word='';
my $max=0;
foreach $word (keys %word_count)
{
    if ($word_count{$word} -gt $max)
    {
        $max = $word_count{$word};
        $most_frequent_word = $word;
    }
}
#Print results
print "Most frequent word : $most_frequent_word\n";
print "Occurrences : $max\n";

Answer 5

#!/bin/bash
file=$1
declare -A count
for word in $(< "$file"); do
    echo $word
    (( count[$word]++ ))
done

max=0
for word in "${!count[@]}"; do
    if (( ${count[$word]} > $max )); then
        max=${count[$word]}
        max_word=$word
    fi
done
echo "most seen word: '$max_word', seen $max times"

Notes:

• $(<file) is a bash shorthand for $(cat file) -- it returns the contents of the file
• because $(<file) is not itself double-quoted, the shell will split it into words, and the for loop will iterate over the words.
• you need bash version 4 for associative arrays
• <opinion> I don't know why people complain about perl syntax being ugly: do you see how you have to handle arrays in bash? </opinion>