0

I want to write a /bin/sh shell script which will handle any files matching a wildcard. It's easy to handle 1 or more matching files. However, I'm finding it awkward to handle the case of 0 matching files.

The obvious construct is:

#!/bin/sh
for f in *.ext; do
  handle "$f"
done

where *.ext could be one or more expressions which the shell compares to file paths, and handle is a shell command which runs correctly if given a path to an existing file, but fails if giving a path which does not map to a file. (In the case that provokes this question, they are *.flac and ffmpeg, but I think that doesn't matter.)

If there are matching files foo.ext, bar.ext, then this script performs

handle "foo.ext"
handle "bar.ext"

as expected. However, if there not any matching files, this script gives an error message like,

handle: *.ext: No such file or directory

I think I understand why this happens: the bash man page (which I assume is valid for /bin/sh also) says the "list of words following in is expanded, generating a list of items.… If the expansion of the items following in results in an empty list, no commands are executed, and the return status is 0." Apparently when *.ext "is expanded", the list of results includes *.ext, instead of an empty list. But that doesn't explain how to stop it happening.

(Update: There is a sh (1) man page from the Heirloom Project which describes the Bourne Shell sh, not the later Bourne Again Shell bash. Under File Name Generation, it clearly says, "If no file name is found that matches the pattern then the word is left unchanged." It explains why sh leaves a *.ext pattern in the list.)

What is a compact, idiomatic way to write this loop so that it will run zero times with no errors if there are no matching files, but will run once for each matching file? Better yet, what will work with multiple patterns like:

for f in *.ext1 special.ext1 *.ext2; do ...

I prefer to use syntax compatible with /bin/sh, for portability. I happen to be using Mac OS X 10.11.6, but I'm hoping for syntax which works on any Unix-like OS.

I came up with a couple of clumsy, non-idiomatic ways to write such a loop. If I don't immediately get good answers, I'll contribute those as answers, for the record.

  • This is trivial to handle if you're willing to move up to bash. – Ignacio Vazquez-Abrams Apr 2 '18 at 23:49
  • @IgnacioVazquez-Abrams I prefer to stick to /bin/sh, but it's not absolute. I think an answer of the form "switch to bash, then do this" is helpful. It won't be as good as one which says, "here's how to do it in sh", but maybe such an answer doesn't exist. – Jim DeLaHunt Apr 3 '18 at 0:15
  • bash has nullglob, which will prevent the loop from even running. – Ignacio Vazquez-Abrams Apr 3 '18 at 0:29
  • Great! Turn that into a good answer, and you'll get points. – Jim DeLaHunt Apr 3 '18 at 0:34
2

The simplest portable way to do this is to skip the loop for if the expansion doesn't produce something that actually exists:

for f in *.ext1 *.ext2; do
  [ -e "$f" ] || continue
  handle "$f"
done

Explanation: suppose there are several .ext2 files, but no .ext1 files. In that case the wildcards will expand to *.ext1 filea.ext2 fileb.ext2 filec.ext2. So [ -e "*.ext1" ] fails and it runs continue, which skips the rest of that iteration of the loop. Then [ -e "filea.ext2" ] etc succeed, and those iterations of the loop run normally.

BTW, you can modify this to e.g. [ -f "$f" ] || continue to skip anything that isn't a plain file (if you want to skip directories etc).

Of course, if you're running under bash, you can just run shopt -s nullglob first so it doesn't leave unmatched patterns in the list. And then shopt -u nullglob afterward to prevent unexpected side effects (e.g. grep pattern *.nonexistent will try to read from stdin if nullglob is set).

  • Clever use of continue, and excellent explanation. I think the use of nullglob turns into a second answer, if I understand it correctly. @Ignacio Vazquez-Abrahams mentioned it in a comment under the question. – Jim DeLaHunt Apr 3 '18 at 8:24
1

Switch to the Bourne Again Shell (/bin/bash) from the Bourne Shell (/bin/sh), and a simple solution becomes possible.

The bash(1) man page mentions the nullglob option:

If set, bash allows patterns which match no files (see Pathname Expansion above) to expand to a null string, rather than themselves.

The Pathname Expansion section says:

After word splitting, …bash scans each word for the characters *, ?, and [. If one of these characters appears, then the word is regarded as a pattern, and replaced with an alphabetically sorted list of file names matching the pattern. If no matching file names are found, and the shell option nullglob is not enabled, the word is left unchanged. If the nullglob option is set, and no matches are found, the word is removed.…

So, setting the nullglob option gives the desired behaviour for patterns. If no file matches the pattern, the loop does not execute.

#!/bin/bash
shopt -s nullglob
for f in *.ext; do
  handle "$f"
done

But the nullglob option may well interfere with desired behaviour for other commands. So, you will probably find it wise to save the existing setting of nullglob, and restore it afterwards. Fortunately, the shopt -p builtin command emits output in a form that may be reused as input. But it emits output for all options, so use grep to pick out the nullobj setting. See the log below (where $ indicates the bash command prompt):

$ shopt -p | grep nullglob
shopt -u nullglob
$ shopt -s nullglob
$ shopt -p | grep nullglob
shopt -s nullglob

So, the final bash syntax to handle zero or more file matching a wildcard pattern looks like this:

#!/bin/bash
SAVED_NULLGLOB=$(shopt -p | grep nullglob)
shopt -s nullglob
for f in *.ext; do
  handle "$f"
done
eval "$SAVED_NULLGLOB"

Also, the question mentioned multiple patterns, like:

for f in *.ext1 special.ext1 *.ext2; do ...

The nullglob option doesn't affect a word in the list which is not a "pattern", so special.ext1 will still get passed to the loop. The only solution for this is @Gordon Davisson's continue expression, [ -e "$f" ] || continue .

Acknowledgement: both @Ignacio Vazquez-Abrams and @Gordon Davisson alluded to bash(1) and its nullglob option. Thank you. Since they didn't immediately contribute a answer with a fully-worked example, I am doing so myself.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.