Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a large number of log files, on a linux box, I need to cleanse sensitive data from before sending to a third party. I have used the below script on previous occasions to perform this task, and it has worked brilliantly (script was built with some help from here :-) ):

#!/bin/bash

help_text () {
cat <<EOF
Usage: $0 [log_directory] [client_name(s)]
EOF
exit 0
}

CMDLINE=""$0" "$@""
if [ -z "$1" ]; then
        help_text
else

        pattern=""
        delim=""
        n=1

        counter=`find "$1" -name *.gz |sort |wc -l`

        BAKIFS=$IFS
        IFS=$(echo -en "\n\b")
        exec 3<&0
        exec 0<"$2"
        while read -r line
        do
                pattern=$pattern$delim$line
                delim="|"
        done
        exec 0<&3
        IFS=$BAKIFS

        while [ $n -lt $counter ]
        do
                for i in `find "$1" -name *.gz |sort`
                do
                        gunzip "$i"
                        i_unzip=$(echo "$i" |sed 's/\.[^\.]*$//')
                        sed -ri "s/$pattern/CLIENT/g" "$i_unzip"
                        gzip "$i_unzip"
                done
                n=n+1
        done
fi
exit 0

However, now one of our departments has sent me a CLIENT_FILE.txt with 425000+ variables! I think I may have hit some internal limit! If anyone has an idea on how to deal with this many variables I'd really appreciate it.

I have tried splitting the client file into 4 with around 100000 variables in each, this still doesn't work. I'm loathe to keep splitting though as I have 20 directories with up to 190 files in each directory to run through. The more client files I make, the more passes I have to do.

share|improve this question
    
In your log file, can the client name appear anywhere, or do you see it in a particular place/field/column of the log entry? –  glenn jackman Jul 28 '11 at 12:09
    
The client name can appear anwhere (or not at all). The log files are a complete mess actually, and I have no idea what the third party is going to get out of them. However, mine is not to question... –  Steve Jul 28 '11 at 13:39

2 Answers 2

up vote 1 down vote accepted

I'd try something like this:

#!/bin/bash

files=()
while read file; do
    gunzip "$file"  &&  files+=( "${file%.gz}" )
done < <(find "$1" -name '*.gz')

awk '
    FILENAME == ARGV[1] {
        client_name[$0]++
        next
    }
    FNR == 1 {
        output = FILENAME ".new"
    }
    {
        for (i=1; i<=NF; i++) {
            if ($i in client_name)
                $i = "CLIENT"
        }
        print > output
    }
' "$2" "${files[@]}"

for file in "${files[@]}"; do
    mv "$file" "$file.old"  &&
    mv "$file.new" "$file"  &&
    gzip "$file"
done

If your log files have anything more than simple space-delimited lines, the awk script may disrupt the formatting.

share|improve this answer
    
Looks good, but it's not liking line #5. I've tried << and a single <, but it complains about an unexpected token near < (or near ( in the case of a single <) –  Steve Jul 28 '11 at 13:32
    
What version of bash do you have? The syntax there is correct: redirect stdin (the first "<") from a process substitution (the "<(find ...)") –  glenn jackman Jul 28 '11 at 14:20
    
even though both sh and bash both go to bash 3.2.51, running the script with bash explicitely made it work! Thanks for your help. –  Steve Jul 28 '11 at 14:57
    
The script also worked extremely fast. Much faster than my previous sed script, so Thanks for that too. –  Steve Jul 28 '11 at 14:58
    
Yes, when bash is invoked as "sh", it tries to run in "POSIX mode" so lots of useful bash features will be disabled. Check the man page. –  glenn jackman Jul 28 '11 at 15:41

You should try to write the sed pattern into a file and pass it to sed with option --file=. Command line parameters are not meant to pass along large chunks of data.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.