Given a log file, I will usually do something like this:

grep 'marker-1234' filter_log

What is the difference in using '' or "" or nothing in the pattern?

The above grep command will yield many thousands of lines; what I desire. Within those lines, There is usually one chunk of data I am after. Sometimes, I use awk to print out the fields I am after. In this case, the log format changes, I can't rely on position exclusively, not to mention, the actual logged data can push position forward.

To make this understandable, lets say the log line contained an IP address, and that was all I was after, so I can later pipe it to sort and unique and get some tally counts.

An example may be:

2010-04-08 some logged data, indetermineate chars - [marker-1234] ( from: foo@bar.example.com to bar@foo.example.com [stat-xyz9876]

The first grep command will give me many thousands of lines like the above, from there, I want to pipe it to something, probably sed, which can pull out a pattern within, and print only the pattern.

For this example, using an the IP address would suffice. I tried. Is sed not able to understand [0-9]{1,3}. as a pattern? I had to [0-9][0-9][0-9]. which yielded strange results until the entire pattern created.

This is not specific to an IP address, the pattern will change, but I can use that as a learning template.

Thank you all.

  • This sounds very programming related, even simple bash commands like grep and awk are in my opinion better answered on stackoverflow. – Josh K Apr 9 '10 at 1:27
  • @Josh: SU has plenty of linux commandline warriors who can handle this kind of question, and it's welcome here. it may be one of those questions that would fit on either site, so it's really up to the asker. – quack quixote Apr 9 '10 at 5:54
  • I gave it some thought, and to be honest, I did not know where to post it. I went with the name of the sites, figuring SO is more general, and SU is more admin type stuff. I find most quick shell scripting is admin related. Sure, you get into large tcl or bash exclusive projects for one reason or another, in which case I would narrow it to programming related and post to SO. This was more one liner'ish, and SU seemed a good home. Sorry if I posted in the wrong spot, but it seems a grey area in some cases. – user17245 Apr 9 '10 at 7:19
  • @allentown: actually Server Fault is more admin-type stuff; Super User is more end-user stuff. (but power-user end-users.) anyway, this question is probably welcome on any of SO/SF/SU. you've already accepted an answer, so if you're satisfied you can call it done. or if you'd like we can migrate it to SO/SF; just flag it for moderator attention and tell us where to send it. thanks! – quack quixote Apr 9 '10 at 7:31
  • @quack: That's fine, I was just thinking that a better answer might be found there, however it seems that it's already been found. – Josh K Apr 9 '10 at 19:38

I don't know what OS you're on, but on FreeBSD 7.0+ grep has a -o option to return only the part that matches the pattern. So you could
grep "marker-1234" filter_log | grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}"

Returns a list of just IP addresses from the 'filter_log"...

This works on my system, but again, I don't know what your version of grep supports.

  • I think all the answers here are excellent ways of learning and approaching the end result with the same answer. I particularly like this one as it is easily rememberable and is just pipe chaining multiple grep commands. On Mac OS X, I do seem to have the -o option, and of course use the -E option already quite often. Thank you for your reply – user17245 Apr 9 '10 at 7:24

you can do all these in just one awk command. No need to use any other tools

$ awk '/marker-1234/{for(o=1;o<=NF;o++){if($o~/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/)print $o }  }' file
  • Thanks, that does work, awk can hurt your head a little at times, but I am getting used to the for FOO.... one liner aspect of quick stuff in the shell. Very powerful. – user17245 Apr 9 '10 at 7:16

You can shorten the second grep a little like this:

grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}'

To answer your first question, double quotes allow the shell to do various things like variable expansion, but protect some metacharacters from needing to be escaped. Single quotes prevent the shell from doing those expansions. Using no quotes leaves things wide open.

$ empty=""
$ text1="some words"
$ grep $empty some_file
(It seems to hang, but it's just waiting for input since it thinks "some_file" is 
the pattern and no filename was entered, so it thinks input is supposed to come
from standard input. Press Ctrl-d to end it.)
$ grep "$empty" some_file
(The whole file is shown since a null pattern matches everything.)
$ grep $text1 some_file
grep: words: No such file or directory
some_file:some words
(It sees the contents of the variable as two words, the first is seen as the 
pattern, the second as one file and the filename as a second file.)
$ grep "$text1" some_file
some_file:some words
(Expected results.)
$ grep '$text1' some_file
(No results. The variable isn't expanded and the file doesn't contain a
string that consists of literally those characters (a dollar sign followed
by "text1"))

You can learn more in the "QUOTING" section of man bash

  • Great write up, thanks. I need to get a better handle on this with regad to IFS as well, I was pretty hung up on that the other day, but did manage to get IFS to play nice. It's a terrible thing when you have IFS set and forgot about it, wonder wtf is going on for an hour. – user17245 Apr 9 '10 at 7:22
  • It's a good idea to be in the habit to always save the value of IFS and restore it as soon as possible: saveIFS="$IFS"; IFS=","; do_something; IFS="$saveIFS"; do_other_stuff – Dennis Williamson Apr 9 '10 at 11:00
  • +1 Good call on the regex, I don't use it much, so I tend to be a little inefficient. – Chris S Apr 9 '10 at 12:49

Look up the xargs command. You should be able to do something like:

grep 'marker-1234' filter_log|xargs grep "("|cut -c1-15

This may not be it exactly, but xargs is the command you want to use

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.