Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a large number of log files, on a linux box, I need to cleanse sensitive data from before sending to a third party. I have used the below script on previous occasions to perform this task, and it has worked brilliantly (script was built with some help from here :-) ):

#!/bin/bash

help_text () {
cat <<EOF
Usage: $0 [log_directory] [client_name(s)]
EOF
exit 0
}

CMDLINE=""$0" "$@""
if [ -z "$1" ]; then
        help_text
else

        pattern=""
        delim=""
        n=1

        counter=`find "$1" -name *.gz |sort |wc -l`

        BAKIFS=$IFS
        IFS=$(echo -en "\n\b")
        exec 3<&0
        exec 0<"$2"
        while read -r line
        do
                pattern=$pattern$delim$line
                delim="|"
        done
        exec 0<&3
        IFS=$BAKIFS

        while [ $n -lt $counter ]
        do
                for i in `find "$1" -name *.gz |sort`
                do
                        gunzip "$i"
                        i_unzip=$(echo "$i" |sed 's/\.[^\.]*$//')
                        sed -ri "s/$pattern/CLIENT/g" "$i_unzip"
                        gzip "$i_unzip"
                done
                n=n+1
        done
fi
exit 0

However, now one of our departments has sent me a CLIENT_FILE.txt with 425000+ variables! I think I may have hit some internal limit! If anyone has an idea on how to deal with this many variables I'd really appreciate it.

I have tried splitting the client file into 4 with around 100000 variables in each, this still doesn't work. I'm loathe to keep splitting though as I have 20 directories with up to 190 files in each directory to run through. The more client files I make, the more passes I have to do.

share|improve this question
    
In your log file, can the client name appear anywhere, or do you see it in a particular place/field/column of the log entry? – glenn jackman Jul 28 '11 at 12:09
    
The client name can appear anwhere (or not at all). The log files are a complete mess actually, and I have no idea what the third party is going to get out of them. However, mine is not to question... – Steve Jul 28 '11 at 13:39
up vote 1 down vote accepted

I'd try something like this:

#!/bin/bash

files=()
while read file; do
    gunzip "$file"  &&  files+=( "${file%.gz}" )
done < <(find "$1" -name '*.gz')

awk '
    FILENAME == ARGV[1] {
        client_name[$0]++
        next
    }
    FNR == 1 {
        output = FILENAME ".new"
    }
    {
        for (i=1; i<=NF; i++) {
            if ($i in client_name)
                $i = "CLIENT"
        }
        print > output
    }
' "$2" "${files[@]}"

for file in "${files[@]}"; do
    mv "$file" "$file.old"  &&
    mv "$file.new" "$file"  &&
    gzip "$file"
done

If your log files have anything more than simple space-delimited lines, the awk script may disrupt the formatting.

share|improve this answer
    
Looks good, but it's not liking line #5. I've tried << and a single <, but it complains about an unexpected token near < (or near ( in the case of a single <) – Steve Jul 28 '11 at 13:32
    
What version of bash do you have? The syntax there is correct: redirect stdin (the first "<") from a process substitution (the "<(find ...)") – glenn jackman Jul 28 '11 at 14:20
    
even though both sh and bash both go to bash 3.2.51, running the script with bash explicitely made it work! Thanks for your help. – Steve Jul 28 '11 at 14:57
    
The script also worked extremely fast. Much faster than my previous sed script, so Thanks for that too. – Steve Jul 28 '11 at 14:58
    
Yes, when bash is invoked as "sh", it tries to run in "POSIX mode" so lots of useful bash features will be disabled. Check the man page. – glenn jackman Jul 28 '11 at 15:41

You should try to write the sed pattern into a file and pass it to sed with option --file=. Command line parameters are not meant to pass along large chunks of data.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .