Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am using a combination of a shell script, awk script and a find command to perform multiple text replacements in hundreds of files. The files sizes vary between a few hundred bytes and 20 kbytes.

I am looking for a way to speed up this script.

I am using cygwin.

The shell script -

#!/bin/bash

if [ $# = 0 ]; then
 echo "Argument expected"
 exit 1
fi



while [ $# -ge 1 ]
do
   if [ ! -f $1 ]; then
     echo "No such file as $1"
     exit 1
   fi


  awk -f ~/scripts/parse.awk $1  > ${1}.$$

   if [ $? != 0 ]; then
      echo "Something went wrong with the script"
     rm ${1}.$$
      exit 1
   fi
mv ${1}.$$ $1
shift
done

The awk script (simplified) -

#! /usr/bin/awk -f

/HHH.Web/{
    if ( index($0,"Email") == 0)  {
        sub(/HHH.Web/,"HHH.Web.Email");
    }
    printf("%s\r\n",$0); 
    next;
}

The command line

find .  -type f  | xargs ~/scripts/run_parser.sh
share|improve this question
1  
Please don't cross-post. –  Dennis Williamson Feb 17 '11 at 1:40

3 Answers 3

1) there is a bug in your error handling. If you a single xargs pass passes a group of files, and one file blows up, none of the other latter files will get processed. e.g. if

~/scripts/run_parser.sh file1 file2 file3 file4

gets run, and file2 blows up on awk, file1 gets run, but neither file2, file3, nor file4 will get run. I suggest using continue instead of exit 1 there.

2) You're using Cygwin, you're going to be somewhat slow because of the emulation, can't be helped. It will be faster on Linux using the same tools.

3) If you can hack some perl, I'd suggest seeing what perl -p -i can do. Perl syntax is not much hairier than awk, and you'd have one instantiation of perl vs multiple awk ones.

Unless this is very very slow, I'd maybe chalk it up to emulation issues. Other than the perl -p hack to get over some extra fork/execs of awk, I don't think there's a silver bullet someplace.

share|improve this answer
    
Thanks for the suggestions. BTW, I'm want the script to exit if a file fails. –  bryan Feb 16 '11 at 23:08
up vote 0 down vote accepted

This will recurse through all my hundreds of files in less than 10 seconds. Previously, it was taking 15 minutes.

find .  -type f | xargs awk -f ~/scripts/awkv2/parse.awk 

The awk script (simplified) -

/HHH.Web/{
    if ( index($0,"Email") == 0)  {
        sub(/HHH.Web/,"HHH.Web.Email");
    }
    printf("%s\r\n",$0);  > FILENAME
    next;
}

BUT, if the input file is greater than 64kb, the output file is truncated to approx 64kb.

Any ideas why?

share|improve this answer

It is the buffer size reading a file, and more to to point, you're overwriting your original FILENAME. One solution is to use

outFile= FILENAME ".fix"
printf("%s\r\n",$0);  > outFile

and have a separate pass to

mv ${fileName}.fix ${fileName} 

in bash

I also agree with Rich that Cygwin's emulation causes it to be slower. Besides Linux, depending our your organization's tolerance for non-Gnu opensource software, you could try UWIN (Unix for Windows) from David Korn at http://www2.research.att.com/sw/tools/uwin/ also see http://en.wikipedia.org/wiki/UWIN.

Good luck

share|improve this answer
    
Thanks for the help, I have it parsing 400 files in 8 seconds now, see my answer in this thread link –  bryan Feb 18 '11 at 19:58

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.