Optimize shell and awk script

Question

I am using a combination of a shell script, awk script and a find command to perform multiple text replacements in hundreds of files. The files sizes vary between a few hundred bytes and 20 kbytes.

I am looking for a way to speed up this script.

I am using cygwin.

The shell script -

#!/bin/bash

if [ $# = 0 ]; then
 echo "Argument expected"
 exit 1
fi



while [ $# -ge 1 ]
do
   if [ ! -f $1 ]; then
     echo "No such file as $1"
     exit 1
   fi


  awk -f ~/scripts/parse.awk $1  > ${1}.$$

   if [ $? != 0 ]; then
      echo "Something went wrong with the script"
     rm ${1}.$$
      exit 1
   fi
mv ${1}.$$ $1
shift
done

The awk script (simplified) -

#! /usr/bin/awk -f

/HHH.Web/{
    if ( index($0,"Email") == 0)  {
        sub(/HHH.Web/,"HHH.Web.Email");
    }
    printf("%s\r\n",$0); 
    next;
}

The command line

find .  -type f  | xargs ~/scripts/run_parser.sh

Answer 1

1) there is a bug in your error handling. If you a single xargs pass passes a group of files, and one file blows up, none of the other latter files will get processed. e.g. if

~/scripts/run_parser.sh file1 file2 file3 file4

gets run, and file2 blows up on awk, file1 gets run, but neither file2, file3, nor file4 will get run. I suggest using continue instead of exit 1 there.

2) You're using Cygwin, you're going to be somewhat slow because of the emulation, can't be helped. It will be faster on Linux using the same tools.

3) If you can hack some perl, I'd suggest seeing what perl -p -i can do. Perl syntax is not much hairier than awk, and you'd have one instantiation of perl vs multiple awk ones.

Unless this is very very slow, I'd maybe chalk it up to emulation issues. Other than the perl -p hack to get over some extra fork/execs of awk, I don't think there's a silver bullet someplace.

Answer 2

This will recurse through all my hundreds of files in less than 10 seconds. Previously, it was taking 15 minutes.

find .  -type f | xargs awk -f ~/scripts/awkv2/parse.awk 

The awk script (simplified) -

/HHH.Web/{
    if ( index($0,"Email") == 0)  {
        sub(/HHH.Web/,"HHH.Web.Email");
    }
    printf("%s\r\n",$0);  > FILENAME
    next;
}

BUT, if the input file is greater than 64kb, the output file is truncated to approx 64kb.

Any ideas why?

Answer 3

It is the buffer size reading a file, and more to to point, you're overwriting your original FILENAME. One solution is to use

outFile= FILENAME ".fix"
printf("%s\r\n",$0);  > outFile

and have a separate pass to

mv ${fileName}.fix ${fileName} 

in bash

I also agree with Rich that Cygwin's emulation causes it to be slower. Besides Linux, depending our your organization's tolerance for non-Gnu opensource software, you could try UWIN (Unix for Windows) from David Korn at http://www2.research.att.com/sw/tools/uwin/ also see http://en.wikipedia.org/wiki/UWIN.

Good luck