I just want to get all the files in a certain directory into a bash array (assuming that none of the files have a newline in the name):

So:

myarr=()
find . -maxdepth 1  -name "mysqldump*" | mapfile -t myarr; echo "${myarr[@]}"

Empty result!

If I do the roundabout way of using a file, temporary or otherwise:

myarr=()
find . -maxdepth 1  -name "mysqldump*" > X
mapfile -t myarray < X
echo "${myarray[@]}"

Result!

But why doesn't mapfile read properly from a pipe?

  • 5
  • Excellent answers all around, thank you everyone. Interesting how the execution strategy of the pipeline (each part running in a spearate process) leaks "upwards" and modifies the apparent meaning of the code, basically silently putting "local" in front of every variable appearing in the pipe. In a language that is something other than crazy glue for other programs, that would be bug, hopefully. – David Tonhofer Aug 15 at 7:30
  • 2
    If you give the code to shellcheck, you get warnings: SC2030: "Modification of var is local (to subshell caused by pipeline)" and SC2031: "var was modified in a subshell. That change might be lost.". Excellent. – David Tonhofer Aug 15 at 7:31
  • Why using find and mapfile here at all and not just simply myarr=(mysqldump*)? This will even work with filennames with spaces and newlines. – BlackJack Aug 15 at 14:43
  • 1
    Just noticed that one has to turn nullglob option on (shopt -s nullglob) on for myarr=(mysqldump*) to not end up with the array ('mysqldump*') in case no files match. – David Tonhofer Aug 16 at 8:52
up vote 21 down vote accepted

From man 1 bash:

Each command in a pipeline is executed as a separate process (i.e., in a subshell).

Such subshells inherit variables from the main shell but they are independent. This means mapfile in your original command operates on its own myarr. Then echo (being outside the pipe) prints empty myarr (which is the main shell's myarr).

This command works differently:

find . -maxdepth 1 -name "mysqldump*" | { mapfile -t myarr; echo "${myarr[@]}"; }

In this case mapfile and echo operate on the same myarr (which is not the main shell's myarr).

To change the main shell's myarr you have to run mapfile in the main shell exactly. Example:

myarr=()
mapfile -t myarr < <(find . -maxdepth 1 -name "mysqldump*")
echo "${myarr[@]}"
  • Added the link to "process substituion" as given in Attie's response, in case a visitor has a TL;DR moment. – David Tonhofer Aug 16 at 9:05

Bash runs the commands of a pipeline in a subshell environment, so any variable assignments etc. that take place within it aren't visible to the rest of the shell.

Dash (Debian's /bin/sh) as well as busybox's sh are similar, while zsh and ksh run the last part in the main shell. In Bash, you can use shopt -s lastpipe to do the same, but it only works when job control is disabled, so not in interactive shells by default.

So:

$ bash -c 'x=a; echo b | read x; echo $x'
a
$ bash -c 'shopt -s lastpipe; x=a; echo b | read x; echo $x'
b

(read and mapfile have the same issue.)

Alternatively (and as mentioned by Attie), use process substitution, which works like a generalized pipe, and is supported in Bash, ksh and zsh.

$ bash -c 'x=a; read x < <(echo b); echo $x'
b

POSIX leaves it unspecified if the parts of a pipeline run in subshells or not, so it can't really be said that any of the shells would be "wrong" in this.

  • 1
    If you disable bash's job control you can use lastpipe in an interactive shell, too : set +m; shopt -s lastpipe; x=a; echo b | read x; echo $x; set -m – Cyrus Aug 15 at 5:36
  • @Cyrus, ah right, I'd forgotten the details, thanks – ilkkachu Aug 15 at 8:17

As Kamil has pointed out, each element in the pipeline is a separate process.

You can use the following process substitution to get find to run in a different process, with the mapfile invocation remaining in your current interpreter, allowing access to myarr afterwards:

myarr=()
mapfile -t myarr < <( find . -maxdepth 1  -name "mysqldump*" )
echo "${myarr[@]}"

b < <( a ) will act similarly to a | b in terms of how the pipeline is wired - the difference is that b is executed "here".

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.