1

So, I was looking to take the output of unzip -Z1 into an array and found this answer; their first option, using mapfile and process substitution WITH input redirection, works a charm.

But then I thought, "wait, process substitution," which creates a file descriptor from the stdout, "then use input redirection on that to put the contents into stdin?"
Is that not equivalent to just piping?

Apparently no. No it isn't. Here I try to put the contents of ls into a variable, but using a pipe: à la ls | mapfile -t test
enter image description here Nada.

But, if I follow that answer to a T: id est mapfile -t test < <(ls)
enter image description here Voilà.

But why? diff can't tell the difference. At least between their content.
enter image description here

I couldn't find anything inherently special about < <(), other than it creates a file descriptor. Using that as a theory, I tried a HEREDOC, which makes a file descriptor too. It worked:

mapfile -t test2 << END && printf '%s ' "${test2[@]}"
this
is
an
array
END

But not if it's laundered through stdin via cat, try:

cat << END | mapfile -t test3 && printf '%s ' "${test3[@]}"
this
is
an
array
END

So, my questions.
That HEREDOC discovery/file descriptor theory shouldn't matter for mapfile as its manuals all say it uses "standard input", right?
If 'standard input' somehow means it has to be a fancier descriptor; why? Why is it unable to use a simple read-once input stream to generate the array?
And finally if both of those are answered/on the right track, why didn't the command fail when it expected an fd? You'll see my terminal has 0 printed a bunch (and we use && for the last examples), that's $? on my prompt, so there was no error reported by the builtin.

I tried this on Fedora 30 and RHEL 6, so I don't think it's a bug.

  • 1
    I think this is going to come down to how each part of a pipeline runs in its own subshell, so in some of the ways you're invoking mapfile, the "test" variable it creates is not accessible to other parts of the pipeline or the parent shell because it only existed in the subshell that mapfile ran in. – Spiff Aug 12 '19 at 5:08
  • shopt -s lastpipe should make ls | mapfile … work (in a script where job control is not active). – Kamil Maciorowski Aug 12 '19 at 7:20
2

It works; it's just that the results disappear immediately afterwards.

The difference is that foo | mapfile is a pipeline consisting of two new processes – each element, be it an external program or a shell builtin, is run in a subshell. So although mapfile still does its job it has no way of transferring the result to the parent shell process.

For comparison, you would get the same result with (mapfile < foo.txt) or ls | while read line; do ... done or even (foo=bar); echo $foo. In all these cases, if you compare $BASHPID and $$, you'll see different values – the former shows the subshell process PID being different from the main shell PID.

When using a redirection, the mapfile command runs in the main shell process and can alter shell variables correctly. It doesn't much care if it's taking redirection from a real file, or from a <<-generated temporary file, or from a <()-generated magic file – in this case, the shell handles that as a separate step and doesn't cause a pipeline to be built. (The inner command of <() or $() is a subshell, of course.)


Note that the variable can actually be used in the subshell – it just cannot be transferred upwards. So for one-shot operations, the 2nd part of pipeline can be quite elaborate:

unzip -l | {
    mapfile -t test
    for file in "${test[@]}"; do
        echo "I got $file!"
    done
}
zipinfo | while read -r file; do
    echo "I got $file!"
done
| improve this answer | |
  • I can't for the life of me remember when, but this has tripped me up on something before. Without explicit parenthesis I keep forgetting simple pipes are subshells too. They just always seem so flat, not embedded in anything, or under a different "scope". It just struck me as a waste to convert to a file, then convert back to stdin, but I guess there was no other way to begin with – Hashbrown Aug 12 '19 at 5:20
  • If it helps, both <<EOF and <<< "text" actually create a temporary file in /tmp behind the scenes... – user1686 Aug 12 '19 at 6:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.