23

I just want to get all the files in a certain directory into a bash array (assuming that none of the files have a newline in the name):

So:

myarr=()
find . -maxdepth 1  -name "mysqldump*" | mapfile -t myarr; echo "${myarr[@]}"

Empty result!

If I do the roundabout way of using a file, temporary or otherwise:

myarr=()
find . -maxdepth 1  -name "mysqldump*" > X
mapfile -t myarray < X
echo "${myarray[@]}"

Result!

But why doesn't mapfile read properly from a pipe?

7
  • 6
  • 1
    Excellent answers all around, thank you everyone. Interesting how the execution strategy of the pipeline (each part running in a spearate process) leaks "upwards" and modifies the apparent meaning of the code, basically silently putting "local" in front of every variable appearing in the pipe. In a language that is something other than crazy glue for other programs, that would be bug, hopefully. Aug 15, 2018 at 7:30
  • 2
    If you give the code to shellcheck, you get warnings: SC2030: "Modification of var is local (to subshell caused by pipeline)" and SC2031: "var was modified in a subshell. That change might be lost.". Excellent. Aug 15, 2018 at 7:31
  • Why using find and mapfile here at all and not just simply myarr=(mysqldump*)? This will even work with filennames with spaces and newlines.
    – BlackJack
    Aug 15, 2018 at 14:43
  • 1
    Just noticed that one has to turn nullglob option on (shopt -s nullglob) on for myarr=(mysqldump*) to not end up with the array ('mysqldump*') in case no files match. Aug 16, 2018 at 8:52

6 Answers 6

35

From man 1 bash:

Each command in a pipeline is executed as a separate process (i.e., in a subshell).

Such subshells inherit variables from the main shell but they are independent. This means mapfile in your original command operates on its own myarr. Then echo (being outside the pipe) prints empty myarr (which is the main shell's myarr).

This command works differently:

find . -maxdepth 1 -name "mysqldump*" | { mapfile -t myarr; echo "${myarr[@]}"; }

In this case mapfile and echo operate on the same myarr (which is not the main shell's myarr).

To change the main shell's myarr you have to run mapfile in the main shell exactly. Example:

myarr=()
mapfile -t myarr < <(find . -maxdepth 1 -name "mysqldump*")
echo "${myarr[@]}"
2
11

Bash runs the commands of a pipeline in a subshell environment, so any variable assignments etc. that take place within it aren't visible to the rest of the shell.

Dash (Debian's /bin/sh) as well as busybox's sh are similar, while zsh and ksh run the last part in the main shell. In Bash, you can use shopt -s lastpipe to do the same, but it only works when job control is disabled, so not in interactive shells by default.

So:

$ bash -c 'x=a; echo b | read x; echo $x'
a
$ bash -c 'shopt -s lastpipe; x=a; echo b | read x; echo $x'
b

(read and mapfile have the same issue.)

Alternatively (and as mentioned by Attie), use process substitution, which works like a generalized pipe, and is supported in Bash, ksh and zsh.

$ bash -c 'x=a; read x < <(echo b); echo $x'
b

POSIX leaves it unspecified if the parts of a pipeline run in subshells or not, so it can't really be said that any of the shells would be "wrong" in this.

2
  • 4
    If you disable bash's job control you can use lastpipe in an interactive shell, too : set +m; shopt -s lastpipe; x=a; echo b | read x; echo $x; set -m
    – Cyrus
    Aug 15, 2018 at 5:36
  • @Cyrus, ah right, I'd forgotten the details, thanks
    – ilkkachu
    Aug 15, 2018 at 8:17
9

As Kamil has pointed out, each element in the pipeline is a separate process.

You can use the following process substitution to get find to run in a different process, with the mapfile invocation remaining in your current interpreter, allowing access to myarr afterwards:

myarr=()
mapfile -t myarr < <( find . -maxdepth 1  -name "mysqldump*" )
echo "${myarr[@]}"

b < <( a ) will act similarly to a | b in terms of how the pipeline is wired - the difference is that b is executed "here".

1

If we use:

myarr=()
mapfile -t myarr < <( find . -maxdepth 1  -name "mysqldump*" )
echo "${myarr[@]}"

${myarr[@]} is not lost but $? of <(find..) is lost

If we use:

find . -maxdepth 1 -name "mysqldump*" | mapfile -t myarr

${myarr[@]} is lost but $? of find is not lost

if we use

shopt -s lastpipe
find . -maxdepth 1 -name "mysqldump*" | mapfile -t myarr

${myarr[@]} is not lost, $? of find is not lost but you need to disable bash job control (set +m)

if we use something like this

line=$(find . -maxdepth 1 -name "mysqldump*"); rc=$?; mapfile -d '\n' myarr <<< "$line"

${myarr[@]} is not lost, $? of find is not lost, bash job control (set -m) does not need to be disabled ... but you you can't use \0 null byte as delimiter (find ... -print0 | mapfile -d '' ...)

The only way I have found come from https://stackoverflow.com/questions/51072044/process-substitution-and-exit-codes and need bash 4.4.x

exec {psfd}< <(find . -maxdepth 1 -name "mysqldump*" -print0); procsub_pid=$!
mapfile -d '\0' myarr <&$psfd
exec {psfd}<&-
wait "$procsub_pid"
echo $?
echo "${myarr[@]}"

${myarr[@]} is not lost, $? of find is not lost, bash job control (set -m) does not need to be disabled and \0 null byte can be used as delimiter.

0

I don't know why, per se, but looking at shellcheck SC2206: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a., I can see how to make your example work: use a herestring (<<<), like this:

# For bash
mapfile -t array <<< "$var"

For your particular case, therefore, do this:

# Capture all `find` output into a regular "indexed" bash array by using
# `mapfile` and a herestring (`<<<`)
mapfile -t array <<< "$(find . -maxdepth 1  -name "mysqldump*")"

# Print all elements at once in a big blob (this also demonstrates how
# to pass all elements in the array as arguments to another function)
echo "${array[@]}"

# OR, print all elements nicely, one element per line
for element in "${array[@]}"; do
    echo "  $element"
done

# OR, the same thing as just above but as a one-liner:
for element in "${array[@]}"; do echo "  $element"; done
-1

Here I am presenting my solution.

I refer to items in the very usefull arcticle

cant-pipe-in-bashs-mapfile-but-why

in

superuser: /questions/1348948/

There the main goal is not only to store the filenames in the array, but to store each one line as an alement in the array.

This I have solved for me.

Here I present my current solution. It is a full-featured script.

  • the array filled by bash command mapfile, resp. readarray is not lost;

  • $? of find and of mapfile are printed (of find to STDERR);

  • bash job control (set -m) does not need to be disabled; thus the script can - without any preparatory work - be run out of "fc";

  • the problems with the delimiter ascii null are avoided;

Remarks:

The script is in a debugging state:

lines are numbered with the index of the array;

before lines of a new file will begin, filename with complete directory entry is put into an element of the array and is printed;

framed with comment lines;

both, the directory entry and the framing lines have "## " at beginning of line;

thus can easily be parsed and removed;

# +++++

mapfile -t  WEarr  <<< "$(find /tmp -maxdepth 1 -name "WEx_08*"  -exec /usr/bin/printf "#%.0s" {1..70} \; -exec echo "" \; -exec echo -n "## " \; -exec ls -lad --time-style=long-iso \{\} \;  -exec /usr/bin/printf "#%.0s" {1..70} \; -exec echo "" \;   -exec cat \{\} \; ; WE_retcod_find=$?; echo "WE_retcod_find: ~$WE_retcod_find~" >&2)"

WE_retcod_mapfile=$?
echo "WE_retcod_mapfile: $WE_retcod_mapfile~"

WE_noe_WEarr=${#WEarr[@]}
echo "WE_noe_WEarr: ~$WE_noe_WEarr~"

# for WE_element in "${WEarr[@]}"; do 
#   echo "WE_element: ~$WE_element~"
# done

for ((i=0; $i<$WE_noe_WEarr; i++)); do
  echo "i: $i ;; ~${WEarr[$i]}~"
done 

# -----

-.-

1
  • 1
    That's not an "article", that's this question and you didn't link it correctly. How does your answer solve the problem AND differ substantially from the answers already given? Aug 28, 2022 at 1:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .