taking certain lines from log files on linux

Question

i have log files in the current structure:

STDAT #668324
CSPEC
--Visitor--
IP2LOC:N/A
Time:0900
DOW:1
OS:Windows NT 5.1
Browser:Firefox 3.5
IP:128.64.55.129

i know linux is probably king of command line tool, so is their a tool such as "getlines" or something so i can extract lines from text file? the only lines i need are "OS" and "Browser", and i want to add them all to 1 single txt file for graphing. what command can i use to take those 2 lines from this txt file? I have thousands of files like this from my website which are created by PHP, but since site is now bigger, I will be using a database, but i do not want these files to go to waste since this information is still useful.

Answer 1

This should just return the OS and Browser lines from the file:

egrep '(OS|Browser)' filename

If you have lots of log files, you can use an *

egrep '(OS|Browser)' /path/to/logs/*

You can redirect this output into a new log file

egrep '(OS|Browser)' /path/to/logs/* > summary.log

If you wanted to strip the OS: and Browser: from the start of each line, you could use awk

egrep '(OS|Browser)' /path/to/logs/* | awk -F: '{print $2}' > summary.log

Answer 2

theotherreceive has very nicely explained the solution.

Just one thing to add,
If you suspect that that OS or Browser terms can occur elsewhere in your log file text,

egrep '(^OS:|^Browser:)' filename
        -  - -       -   # Note: extra parts for catching your keys better

will help you check that.
This is assuming the two terms appear at the start of the line and are followed with ':' character.
The rest as already described in that answer.

Answer 3

Let's say that all your files are under one directory, for example: /var/www/html/logs and it's subdirectory. Let's assume that all log files end with ".log". Let's assume that you want the combined log to /home/username/combinedlog.txt

There is a unix-tool called "find" that searches for files in given directory tree, for example

find /var/www/html/logs -name '*.log' -type f -print

would print names of all files ending with '.log' from directory /var/www/html/logs and it's subdirectories.

As the previous answers show, you can get just the OS with "grep OS:|tail -c +4" and just the Browser with "grep OS:|tail -c +4". The first answer also shows how to make for-loop with results of one command. So the combined answer would be:

for i in `find /var/www/html/logs -name '*.log' -type f -print`; do 
  grep "OS:" $i |tail -c +4 >> /home/username/combinedlog.txt; 
  grep "Browser:" $i|tail -c +9 >> /home/username/combinedlog.txt; 
done;

And this would go through all files at once. Note that ">>" is append to a file, if you run this again without removing the previous combined log file, you will duplicate the contents!

Edit: or you could use egrep + awk from the previous answer and replace the two greps with one egrep + awk. You could also use "cut" instead of "tail" or "awk".

Answer 4

I agree with theotherreceive's answer. And, since maybe you're going to get a lot of duplicate lines and maybe you just want to count them ...

egrep '(OS|Browser)' /path/to/logs/* | sort | uniq -c

Piping the output through sort and uniq -c will give you a count of the number of times each browser or OS is in the output.

Answer 5

You should be able to use the magic abilities of grep and tail!

To return the browser:

 cat file.txt | grep Browser: | tail -c +9

To return the OS:

cat file.txt | grep OS: | tail -c +4

If you want to write them to a text file, this should work:

 cat file.txt | grep OS: >> results.txt && cat file.txt | grep Browser: >> results.txt

_{There are probably other solutions out there, but I am not all that nifty at Bash and the sorts like that, so this is the best I came up with.}