Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a large number of log files, on a linux box, I need to cleanse sensitive data from before sending to a third party. I have used the below script on previous occasions to perform this task, and it has worked brilliantly (script was built with some help from here :-) ):


help_text () {
cat <<EOF
Usage: $0 [log_directory] [client_name(s)]
exit 0

CMDLINE=""$0" "$@""
if [ -z "$1" ]; then


        counter=`find "$1" -name *.gz |sort |wc -l`

        IFS=$(echo -en "\n\b")
        exec 3<&0
        exec 0<"$2"
        while read -r line
        exec 0<&3

        while [ $n -lt $counter ]
                for i in `find "$1" -name *.gz |sort`
                        gunzip "$i"
                        i_unzip=$(echo "$i" |sed 's/\.[^\.]*$//')
                        sed -ri "s/$pattern/CLIENT/g" "$i_unzip"
                        gzip "$i_unzip"
exit 0

However, now one of our departments has sent me a CLIENT_FILE.txt with 425000+ variables! I think I may have hit some internal limit! If anyone has an idea on how to deal with this many variables I'd really appreciate it.

I have tried splitting the client file into 4 with around 100000 variables in each, this still doesn't work. I'm loathe to keep splitting though as I have 20 directories with up to 190 files in each directory to run through. The more client files I make, the more passes I have to do.

share|improve this question
In your log file, can the client name appear anywhere, or do you see it in a particular place/field/column of the log entry? – glenn jackman Jul 28 '11 at 12:09
The client name can appear anwhere (or not at all). The log files are a complete mess actually, and I have no idea what the third party is going to get out of them. However, mine is not to question... – Steve Jul 28 '11 at 13:39
up vote 1 down vote accepted

I'd try something like this:


while read file; do
    gunzip "$file"  &&  files+=( "${file%.gz}" )
done < <(find "$1" -name '*.gz')

awk '
    FILENAME == ARGV[1] {
    FNR == 1 {
        output = FILENAME ".new"
        for (i=1; i<=NF; i++) {
            if ($i in client_name)
                $i = "CLIENT"
        print > output
' "$2" "${files[@]}"

for file in "${files[@]}"; do
    mv "$file" "$file.old"  &&
    mv "$" "$file"  &&
    gzip "$file"

If your log files have anything more than simple space-delimited lines, the awk script may disrupt the formatting.

share|improve this answer
Looks good, but it's not liking line #5. I've tried << and a single <, but it complains about an unexpected token near < (or near ( in the case of a single <) – Steve Jul 28 '11 at 13:32
What version of bash do you have? The syntax there is correct: redirect stdin (the first "<") from a process substitution (the "<(find ...)") – glenn jackman Jul 28 '11 at 14:20
even though both sh and bash both go to bash 3.2.51, running the script with bash explicitely made it work! Thanks for your help. – Steve Jul 28 '11 at 14:57
The script also worked extremely fast. Much faster than my previous sed script, so Thanks for that too. – Steve Jul 28 '11 at 14:58
Yes, when bash is invoked as "sh", it tries to run in "POSIX mode" so lots of useful bash features will be disabled. Check the man page. – glenn jackman Jul 28 '11 at 15:41

You should try to write the sed pattern into a file and pass it to sed with option --file=. Command line parameters are not meant to pass along large chunks of data.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .