Using sed to replace a *large* number of variables in a file

Question

I have a large number of log files, on a linux box, I need to cleanse sensitive data from before sending to a third party. I have used the below script on previous occasions to perform this task, and it has worked brilliantly (script was built with some help from here :-) ):

#!/bin/bash

help_text () {
cat <<EOF
Usage: $0 [log_directory] [client_name(s)]
EOF
exit 0
}

CMDLINE=""$0" "$@""
if [ -z "$1" ]; then
        help_text
else

        pattern=""
        delim=""
        n=1

        counter=`find "$1" -name *.gz |sort |wc -l`

        BAKIFS=$IFS
        IFS=$(echo -en "\n\b")
        exec 3<&0
        exec 0<"$2"
        while read -r line
        do
                pattern=$pattern$delim$line
                delim="|"
        done
        exec 0<&3
        IFS=$BAKIFS

        while [ $n -lt $counter ]
        do
                for i in `find "$1" -name *.gz |sort`
                do
                        gunzip "$i"
                        i_unzip=$(echo "$i" |sed 's/\.[^\.]*$//')
                        sed -ri "s/$pattern/CLIENT/g" "$i_unzip"
                        gzip "$i_unzip"
                done
                n=n+1
        done
fi
exit 0

However, now one of our departments has sent me a CLIENT_FILE.txt with 425000+ variables! I think I may have hit some internal limit! If anyone has an idea on how to deal with this many variables I'd really appreciate it.

I have tried splitting the client file into 4 with around 100000 variables in each, this still doesn't work. I'm loathe to keep splitting though as I have 20 directories with up to 190 files in each directory to run through. The more client files I make, the more passes I have to do.

Answer 1

I'd try something like this:

#!/bin/bash

files=()
while read file; do
    gunzip "$file"  &&  files+=( "${file%.gz}" )
done < <(find "$1" -name '*.gz')

awk '
    FILENAME == ARGV[1] {
        client_name[$0]++
        next
    }
    FNR == 1 {
        output = FILENAME ".new"
    }
    {
        for (i=1; i<=NF; i++) {
            if ($i in client_name)
                $i = "CLIENT"
        }
        print > output
    }
' "$2" "${files[@]}"

for file in "${files[@]}"; do
    mv "$file" "$file.old"  &&
    mv "$file.new" "$file"  &&
    gzip "$file"
done

If your log files have anything more than simple space-delimited lines, the awk script may disrupt the formatting.

Answer 2

You should try to write the sed pattern into a file and pass it to sed with option --file=. Command line parameters are not meant to pass along large chunks of data.