Adding trec format tags to thousands of file

Question

I need to add tags like text of file in thousand of files in a directory and I tried it using cat and outputing it to a stream of file using

for file in *
do
    cat ../gau > temp;  //gau contain format i need to append in each file
    echo $file >>temp;
    cat ../gau_ >>temp ;//contains </DOCID>
    cat $file >>temp;  
    cat ../gau1  >> temp;  //this contain last sentence </DOC>
    cat temp > $file
done

but doing this is very slow. Can someone please tell me a better and more efficient way to do this? Is it possible to do using C? How can we open files in batches and then process them and put back as it can speed up this process since opening and writing file is a bottle neck I suppose.

Is there and premade program (which is efficient and fast) to do this job as we are short on time.

Answer 1

You may see a lot of improvement by only calling cat once per file:

for file in *
do
    echo "$file" | cat ../gau - ../gau_ "$file" ../gau1 > temp && mv "$file"
done

The dash in the arguments of cat picks up the filename from the pipeline. All the arguments are conCATenated together into the output file.

Answer 2

Some optimizations to your existing method:

Some of the cat operations can be replaced with echo:

echo "</DOCID>" >> temp

This will append </DOCID> to the temp file without having to read it from a file first. The same can be done for any of the cats from the gau* files if the content is not too long.

Rename the temp file instead of catting the data to the original file:

mv temp $file

This is an almost instant operation and doesn't involve reading all the data and then writing it all over again.