How do I parse the output of the find command when filenames have spaces in them?

Question

Using a loop such as

for i in `find . -name \*.txt` 

will break if some filenames have spaces in them.

What technique can I use to avoid this problem?

Answer 1

Ideally you don't do it that way at all, because parsing filenames properly in a shell script is always difficult (fix it for spaces, you'll still have problems with other embedded characters, in particular newline). This is even listed as the first entry in the BashPitfalls page.

That said, there is a way to almost do what you want:

oIFS=$IFS
IFS=$'\n'

find . -name '*.txt' | while read -r i; do
  # use "$i" with whatever you're doing
done

IFS=$oIFS

Remember to also quote $i when using it, to avoid other things interpreting the spaces later. Also remember to set $IFS back after using it, because not doing so will cause bewildering errors later.

This does have one other caveat attached: what happens inside the while loop may take place in a subshell, depending on the exact shell you're using, so variable settings may not persist. The for loop version avoids that but at the price that, even if you apply the $IFS solution to avoid issues with spaces, you will then get in trouble if the find returns too many files.

At some point the correct fix for all of this becomes doing it in a language such as Perl or Python instead of shell.

Answer 2

Use find -print0 and pipe it to xargs -0, or write your own little C program and pipe it to your little C program. This is what -print0 and -0 were invented for.

Shell scripts aren't the best way to handle filenames with spaces in them: you can do it, but it gets clunky.

Answer 3

You can set the "internal field separator" (IFS) to something else than space for the loop argument splitting, e.g.

ORIGIFS=${IFS}
NL='
'
IFS=${NL}
for i in $(find . -name '*.txt'); do
    IFS=${ORIGIFS}
    #do stuff
done
IFS=${ORIGIFS}

I reset IFS after its use in find, mostly because it looks nice, I think. I haven't seen any problems in having it set to newline, but I think this is "cleaner".

Another method, depending on what you want to do with the output from find, is to either directly use -exec with the find command, or use -print0 and pipe it into xargs -0. In the first case find takes care of the file name escaping. In the -print0 case, find prints its output with a null separator, and then xargs splits on this. Since no file name can contain that character (what I know of), this is always safe as well. This it mostly useful in simple cases; and usually is not a great substitute for a full for loop.

Answer 4

Using find -print0 with xargs -0

Using find -print0 combined with xargs -0 is completely robust against legal file names, and is one of the most extensible methods available. For example, say you wanted a listing of every PDF file within the current directory. You could write

$ find . -iname '*.pdf' -print0 | xargs -0 -n 1 echo

This will find every PDF (via -iname '*.pdf') in the current directory (.) and any sub-directory, and pass each of them as an argument to the echo command. Because we specified the -n 1 option, xargs will only pass one argument at a time to echo. Had we omitted that option, xargs would have passed as many as possible to echo. (You can echo short input | xargs --show-limits to see how many bytes are allowed in a command line.)

What does xargs do, exactly?

We can clearly see the effect xargs has on its input — and the effect of -n in particular — by using a script which echoes its arguments in a more precise manner than echo.

$ cat > echoArgs.sh <<'EOF'
#!/bin/bash
echo "Number of arguments: $#"

[[ $# -eq 0 ]] && exit

for i in $(seq 1 $#); do
    echo "Arg $i: <$1>"
    shift
done
EOF

$ find . -iname '*.pdf' -print0 | xargs -0 ./echoArgs.sh
$ find . -iname '*.pdf' -print0 | xargs -0 -n 1 ./echoArgs.sh

Note that it handles spaces and newlines perfectly well,

$ touch 'A space-age
new line of vending machines.pdf'
$ find . -iname '*space*' -print0 | xargs -0 -n 1 ./echoArgs.sh

which would be especially troublesome with the following common solution:

chmod +x ./echoArgs.sh
for file in $(ls *spacey*); do
  ./echoArgs.sh "$file"
done

Notes

• How to print “$” in here-document

Answer 5

I disagree with the bash bashers, because bash, along with the *nix tool set, is quite adept at handling files (including ones whose names have embedded whitespace).

Actually, find gives you fine grain control over choosing which files to process... On the bash side, you really only need to realize that you must make you strings into bash words; typically by using "double quotes", or some other mechanism like using IFS, or find's {}

Note that in most/many situations you do not need to set and reset IFS; just use IFS locally as shown in the examples below. All three handle whitespace fine. Also you do not need a "standard" loop structure, because find's \; is effectively a loop; just put your loop logic into a bash function (if you aren't calling a standard tool).

IFS=$'\n' find ~/ -name '*.txt' -exec  function-or-util {} \;  

And, two more examples

IFS=$'\n' find ~/ -name '*.txt' -exec  printf 'Hello %s\n' {} \;  
IFS=$'\n' find ~/ -name '*.txt' -exec  echo {} \+ |sed 's/home//'  

'findalso allows you to pass multiple filenames as args to you script ..(if it suits your need: use+instead\;`)