Byzantine behaviour in shell script

Question

I wrote the following script to interactively and recursively remove orphan backup files, i.e. remove each file.txt~ that does not have a corresponding file.txt.

#!/bin/sh -x

set -o errexit
unalias -a

backups=$(find . -name "*~")

orphans=""
while read -r file
do
    [ ! -e "${file%~}" ] && orphans=$(echo "$file\n$orphans");
done << EOF
$backups
EOF

if [ -z "$orphans" ]; then
    echo "No orphans."
else
    echo "orphans:\n$orphans"
    echo "$orphans" | xargs --interactive -d '\n' rm
fi

This script does very weird things in a random fashion. Sometimes it behaves correctly, sometimes it ignores the -x options passed to sh, sometimes it executes commented code, sometimes the tests give just wrong results.

The problem seems to be related to the here script, since by redirecting the output of find to a temporary file all problems seem to disappear. But why, where is the error?

---

Solution (thanks to Dennis Williamson): Escape the ~ character. The unescaped ~ in the parameter expansion ${file%~} was creating somehow all the unpredictable behavior.

---

A more readable and deterministic solution, with some fat cut out, could be (thanks to the suggestions of Mikel):

#!/bin/sh
IFS='
'
for backup in $(find . -type f -name "*~"); do
    if [ ! -e "${backup%\~}" ]; then
        rm -i "$backup"
    fi
done

If you are a while read loop fan, things get less elegant because the interactive command rm -i can not be used (it will conflict with the read command). Anyway, a solution could be:

#!/bin/sh
orphans=""
while read -r backup; do
    if [ ! -e "${backup%\~}" ]; then
        orphans=$(echo "$backup\n$orphans");
fi
done << EOF
$(find . -type f -name "*~")
EOF

if [ ! -z "$orphans" ]; then
    echo "$orphans" | xargs --interactive -d '\n' rm
fi

Or, a more complex way is also suggested by Dennis Williamson.

Answer 1

You probably need to escape the tilde in the brace expansion, otherwise it will get expanded to your home directory.

Why don't you pipe the find into your while loop instead of creating variables to hold them? Inside your loop, just do rm -i "$file".

#!/bin/sh -x

set -o errexit
unalias -a

exec 3<&0    # open a duplicate of stdin
flag=false
find . -name "*~" | while IFS=$'\n' read -r file
do
    if [ ! -e "${file%\~}" ]
    then
        orphans="$file"$'\n'"$orphans"

        # use an alternate file descriptor so read and rm -i get along
        rm -i "$file" <&3
        flag=true
    fi
done
exec 3<&-    # close the file descriptor

if ! $flag
then
    echo "No orphans."
else
    echo "orphans:\n$orphans"
fi

If you want to use Bash, you need to move the find to the end of the loop so a subshell is not created.

#!/bin/bash

...

# use an alternate file descriptor so read and rm -i get along
while read -u 3 -r ...

    rm -i ...
    ...

done 3< <(find ...)

...

Answer 2

What do you mean "sometimes"? Sometimes on the same box? Or different behavior on different systems?

What do you mean "executes commented code"? Your example doesn't have any comments.

Some thoughts:

• try set -x in place of /bin/sh -x
• better to use set -e than set -o errexit
• if you're using bash, then call /bin/bash, not /bin/sh, which could be something else
• echo is unnecessary, just use = with a literal newline
• if you do have to use echo, you should use echo -e or ensure xpg_echo is set
• your orphans line is putting them in reverse order, is that deliberate?
• your read loop will fail if file names contain spaces, you should set IFS first

A simpler version:

#!/bin/bash

set -x
set -e

IFS=$'\n'
orphans=false
for backup in $(find . -type f -name "*~"); do
    original=${backup%\~}
    if [ ! -e "$original" ]; then
        orphans=true
        rm -i "$backup"
    fi  
done

if ! $orphans; then
    echo "No orphans."
fi

Or if you want it to work using /bin/sh:

#!/bin/sh

set -x
set -e

IFS='
'
orphans=false
for backup in $(find . -type f -name "*~"); do
    original=${backup%\~}
    if [ ! -e "$original" ]; then
        orphans=true
        rm -i "$backup"
    fi  
done

if ! $orphans; then
    echo "No orphans."
fi