Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a large amount of log files that I need to remove sensitive data from. The sensitive data is provided to me in a text file and is prone to change.

I had hoped to do the equivalent of this:

#!/usr/bin/bash
pattern=""
for val in 'sed -e 's/.*=//' Client_clean.txt
do
      pattern=$pattern$val"|"
done
#egrep -e $pattern $1
sed -i 's/$pattern/CLIENT/g' $1
exit 0

The commented out egrep works fine, the sed doesn't.

Am I right to use sed for this? Or is there a more apt route to take?

Any help appreciated.

Steve

share|improve this question
up vote 2 down vote accepted

To be able to expand a shell variable in the sed command, you need to use double quotes.

You either need to use the -r option to sed or precede the pipe characters with backslashes.

You also need to avoid adding a pipe character at the end. If you have it there, empty strings will be matched.

#!/usr/bin/bash
pattern=""
delim=""
for val in 'sed -e 's/.*=//' Client_clean.txt
do
      pattern=$pattern$delim$val
      # setting delim after its first use and using it before the new value
      # results in "foo|bar|baz" instead of "foo|bar|baz|"
      delim='|'
done
sed -ri "s/$pattern/CLIENT/g" "$1"
share|improve this answer
    
Thanks so much, worked like a charm. I hadn't realized the problem with the delim character at the end either, so thanks for that too. – Steve Feb 21 '11 at 19:20

Yes, sed is the way to go.

What I see wrong in your sed command are the quotes. the $pattern won't be replaced with it's contents if you use ' for quoting. Instead of using ' you should use " :

sed -ri "s/$pattern/CLIENT/g" $1

You also need "-r" for extended regular expressions.

And pattern=$pattern$val"|" should probably be

pattern=$pattern"|"$val
share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .