Trying to cleanse log files on a Linux machine, currently using sed

Question

I have a large amount of log files that I need to remove sensitive data from. The sensitive data is provided to me in a text file and is prone to change.

I had hoped to do the equivalent of this:

#!/usr/bin/bash
pattern=""
for val in 'sed -e 's/.*=//' Client_clean.txt
do
      pattern=$pattern$val"|"
done
#egrep -e $pattern $1
sed -i 's/$pattern/CLIENT/g' $1
exit 0

The commented out egrep works fine, the sed doesn't.

Am I right to use sed for this? Or is there a more apt route to take?

Any help appreciated.

Steve

Answer 1

To be able to expand a shell variable in the sed command, you need to use double quotes.

You either need to use the -r option to sed or precede the pipe characters with backslashes.

You also need to avoid adding a pipe character at the end. If you have it there, empty strings will be matched.

#!/usr/bin/bash
pattern=""
delim=""
for val in 'sed -e 's/.*=//' Client_clean.txt
do
      pattern=$pattern$delim$val
      # setting delim after its first use and using it before the new value
      # results in "foo|bar|baz" instead of "foo|bar|baz|"
      delim='|'
done
sed -ri "s/$pattern/CLIENT/g" "$1"

Answer 2

Yes, sed is the way to go.

What I see wrong in your sed command are the quotes. the $pattern won't be replaced with it's contents if you use ' for quoting. Instead of using ' you should use " :

sed -ri "s/$pattern/CLIENT/g" $1

You also need "-r" for extended regular expressions.

And pattern=$pattern$val"|" should probably be

pattern=$pattern"|"$val