Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a large number of log files, on a linux box, I need to cleanse sensitive data from before sending to a third party. I have used the below script on previous occasions to perform this task, and it has worked brilliantly (script was built with some help from here :-) ):


help_text () {
cat <<EOF
Usage: $0 [log_directory] [client_name(s)]
exit 0

CMDLINE=""$0" "$@""
if [ -z "$1" ]; then


        counter=`find "$1" -name *.gz |sort |wc -l`

        IFS=$(echo -en "\n\b")
        exec 3<&0
        exec 0<"$2"
        while read -r line
        exec 0<&3

        while [ $n -lt $counter ]
                for i in `find "$1" -name *.gz |sort`
                        gunzip "$i"
                        i_unzip=$(echo "$i" |sed 's/\.[^\.]*$//')
                        sed -ri "s/$pattern/CLIENT/g" "$i_unzip"
                        gzip "$i_unzip"
exit 0

However, now one of our departments has sent me a CLIENT_FILE.txt with 425000+ variables! I think I may have hit some internal limit! If anyone has an idea on how to deal with this many variables I'd really appreciate it.

I have tried splitting the client file into 4 with around 100000 variables in each, this still doesn't work. I'm loathe to keep splitting though as I have 20 directories with up to 190 files in each directory to run through. The more client files I make, the more passes I have to do.

share|improve this question
In your log file, can the client name appear anywhere, or do you see it in a particular place/field/column of the log entry? – glenn jackman Jul 28 '11 at 12:09
The client name can appear anwhere (or not at all). The log files are a complete mess actually, and I have no idea what the third party is going to get out of them. However, mine is not to question... – Steve Jul 28 '11 at 13:39
up vote 1 down vote accepted

I'd try something like this:


while read file; do
    gunzip "$file"  &&  files+=( "${file%.gz}" )
done < <(find "$1" -name '*.gz')

awk '
    FILENAME == ARGV[1] {
    FNR == 1 {
        output = FILENAME ".new"
        for (i=1; i<=NF; i++) {
            if ($i in client_name)
                $i = "CLIENT"
        print > output
' "$2" "${files[@]}"

for file in "${files[@]}"; do
    mv "$file" "$file.old"  &&
    mv "$" "$file"  &&
    gzip "$file"

If your log files have anything more than simple space-delimited lines, the awk script may disrupt the formatting.

share|improve this answer
Looks good, but it's not liking line #5. I've tried << and a single <, but it complains about an unexpected token near < (or near ( in the case of a single <) – Steve Jul 28 '11 at 13:32
What version of bash do you have? The syntax there is correct: redirect stdin (the first "<") from a process substitution (the "<(find ...)") – glenn jackman Jul 28 '11 at 14:20
even though both sh and bash both go to bash 3.2.51, running the script with bash explicitely made it work! Thanks for your help. – Steve Jul 28 '11 at 14:57
The script also worked extremely fast. Much faster than my previous sed script, so Thanks for that too. – Steve Jul 28 '11 at 14:58
Yes, when bash is invoked as "sh", it tries to run in "POSIX mode" so lots of useful bash features will be disabled. Check the man page. – glenn jackman Jul 28 '11 at 15:41

You should try to write the sed pattern into a file and pass it to sed with option --file=. Command line parameters are not meant to pass along large chunks of data.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.