Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am using a combination of a shell script, awk script and a find command to perform multiple text replacements in hundreds of files. The files sizes vary between a few hundred bytes and 20 kbytes.

I am looking for a way to speed up this script.

I am using cygwin.

The shell script -


if [ $# = 0 ]; then
 echo "Argument expected"
 exit 1

while [ $# -ge 1 ]
   if [ ! -f $1 ]; then
     echo "No such file as $1"
     exit 1

  awk -f ~/scripts/parse.awk $1  > ${1}.$$

   if [ $? != 0 ]; then
      echo "Something went wrong with the script"
     rm ${1}.$$
      exit 1
mv ${1}.$$ $1

The awk script (simplified) -

#! /usr/bin/awk -f

    if ( index($0,"Email") == 0)  {

The command line

find .  -type f  | xargs ~/scripts/
share|improve this question
Please don't cross-post. – Dennis Williamson Feb 17 '11 at 1:40

1) there is a bug in your error handling. If you a single xargs pass passes a group of files, and one file blows up, none of the other latter files will get processed. e.g. if

~/scripts/ file1 file2 file3 file4

gets run, and file2 blows up on awk, file1 gets run, but neither file2, file3, nor file4 will get run. I suggest using continue instead of exit 1 there.

2) You're using Cygwin, you're going to be somewhat slow because of the emulation, can't be helped. It will be faster on Linux using the same tools.

3) If you can hack some perl, I'd suggest seeing what perl -p -i can do. Perl syntax is not much hairier than awk, and you'd have one instantiation of perl vs multiple awk ones.

Unless this is very very slow, I'd maybe chalk it up to emulation issues. Other than the perl -p hack to get over some extra fork/execs of awk, I don't think there's a silver bullet someplace.

share|improve this answer
Thanks for the suggestions. BTW, I'm want the script to exit if a file fails. – bryan Feb 16 '11 at 23:08
up vote 0 down vote accepted

This will recurse through all my hundreds of files in less than 10 seconds. Previously, it was taking 15 minutes.

find .  -type f | xargs awk -f ~/scripts/awkv2/parse.awk 

The awk script (simplified) -

    if ( index($0,"Email") == 0)  {
    printf("%s\r\n",$0);  > FILENAME

BUT, if the input file is greater than 64kb, the output file is truncated to approx 64kb.

Any ideas why?

share|improve this answer

It is the buffer size reading a file, and more to to point, you're overwriting your original FILENAME. One solution is to use

outFile= FILENAME ".fix"
printf("%s\r\n",$0);  > outFile

and have a separate pass to

mv ${fileName}.fix ${fileName} 

in bash

I also agree with Rich that Cygwin's emulation causes it to be slower. Besides Linux, depending our your organization's tolerance for non-Gnu opensource software, you could try UWIN (Unix for Windows) from David Korn at also see

Good luck

share|improve this answer
Thanks for the help, I have it parsing 400 files in 8 seconds now, see my answer in this thread link – bryan Feb 18 '11 at 19:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .