How to write an efficient bash script to find and sort files and directories, extracting dates?

Question

Background: It is a temporary measure to load some test data into a test environment (a Sybase database).

I have a bunch of files I have to import into database daily. These files are organized as such

30Aug2011/IMPORT_ME.txt
31Aug2011/IMPORT_ME.txt
...
8Sep2011/IMPORT_ME.txt
9Sep2011/IMPORT_ME.txt

After each import I need to run some special sql statement to fix up the date. This is what I did in the main script

import_file.sh 20Aug2011/IMPORT_ME.txt
cat rerun_import_file.tmpl | sed -e "s/XXX/8 Aug 2011/g" > rerun_import_files.sql
$ISQL -i rerun_import_files.sql

...
import_file.sh 9Sep2011/IMPORT_ME.txt
cat rerun_import_file.tmpl | sed -e "s/XXX/9 Sep 2011/g" > rerun_import_files.sql
$ISQL -i rerun_import_files.sql

So I suppose I can make it better by:

1. find all directories that fit the date pattern

2. sort the dates correctly (very important to the setup)

3. parse the date and set the date according in the sql file

However my bash script-fu is not good enough to do it.

Answer 1

Of course, I missed the sorting part of the question. To answer that, I propose a different model that's simpler in some ways:

ls */IMPORT_ME.txt | cut -d/ -f 1 | date --file=- +%s | sort -n | sed -e 's/^/@/' \
| while read DATE
    do
        FILENAME=`date -d ${DATE} +%-d%b%Y/IMPORT_ME.txt`
        REPLACEMENT=`date -d ${DATE} +%-d %b %Y`
        import_file.sh ${FILENAME}
        cat rerun_import_file.tmpl | sed -e "s/XXX/${REPLACEMENT}/g" > rerun_import_files.sql
        $ISQL -i rerun_import_files.sql
    done

The key here is the initial conversion of all of the (filtered for validity) directory names to a sortable time format (in this case Unix 'epoch' time), sorting them, and then converting them back (to two different forms).

If you're working with dates in any more than a purely string manipulation sense, I strongly recommend reading the man page for 'date'. In fact, read it twice, think hard about it, then read it again. 'date' is an extremely powerful command in its domain.

Oh, and I hope the lesson you have learned from this experience is that you should ALWAYS use ISO format dates (date -Iseconds or date -I for examples). ISO format dates sort well as strings using e.g. sort.