Why does assigning a variable with command substitution, and then echoing that variable, always fail?

Question

Why does the following not work in Bash?

# Ensure TEST is unset
export TEST=''
echo "Hello world!" > test.txt && TEST="$(cat test.txt)" echo "$TEST"

It always returns a blank line for some reason.

Doing it on separate lines gives the expected result:

# Ensure TEST is unset
export TEST=''
echo "Hello world" > test.txt
TEST=$(cat test.txt)
echo "$TEST"
Hello world!

How can I make this work in a one-liner?

I need this to decrypt a GPG file containing a string with special characters which I then pass to Ansible.

ANSIBLE_VAULT_PASSWORD="$( gpg -d ~/gpg_encrypted_vault_password_file 2>/dev/null )" ansible-playbook etc etc etc

For some reason, Ansible developers think storing a vault password in a plain text file is secure and I don't like typing (or copy-pasting) long secure passwords every time I test a play, so using the GPG agent with an encrypted file is the only secure workaround that I can come up with.

Answer 1

Not to conflict with any of the other pre-existing answers, but I thought a step-by-step analysis may help. (u1686_grawity's answer looks accurate, but I thought this might be more clear...)

# ensure TEST is unset export TEST=''

Good news: This likely works as you expected. The # turns the first line into a comment, and then the next line makes the TEST variable have an empty string (zero bytes).

echo "Hello world!" > test.txt && TEST="$(cat test.txt)" echo "$TEST"

The shell notices the &&. It will run the first portion of the code...

echo "Hello world!" > test.txt

Then, it processes the &&. It looks at the return value of the echo command, and sees that it is zero. Therefore, it can proceed to run the second command. The second command is:

TEST="$(cat test.txt)" echo "$TEST"

So, the first thing the shell is going to do here is to replace the variables. The "command substitution" acts like a variable here, so the "cat text.txt" command runs. In the shell's perspective, the command now looks more like:

TEST="Hello world!" echo "$TEST"

Now, that all probably looks good so far, but here's the big issue. The shell doesn't start running the 'TEST="Hello world!"' command yet. Instead, the shell notices that there is still a reference to $TEST, so the shell replaces that.

Now, the command that the shell plans to run looks something like this:

TEST="Hello world!" echo ""

I suppose that if I'm going to go into some full detail, I will point out that the shell does perform the act of breaking this command into pieces. This action removes the quotation marks. So the individual pieces look like this:

1. TEST
2. =
3. Hello world!
4. echo
5. (empty value)

Okay, now the shell finally is done replacing things, so it can go onto the next task, which is to find what executable code to run.

It notices that the command line starts with a form of A=B C (where A represents a variable name, the equal sign specifies that a variable will be assigned a value, and B represents the value to be assigned, and then C is an optional portion which represents a command to run.

When the shell notices the command line fits this general pattern, the executable code that it is going to run is the code that handles the equal sign. Basically, the equal sign is similar to an executable filename, in the sense that the equal sign ends up controlling what code the shell is going to be executing. The shell proceeds to execute the code to assign a variable named TEST to have a value of Hello world!.

Then, after that is done, the shell will start the requested command, which uses the 4th and 5th potions identified above (where the 4th portion is the echo command, and the 5th portion is the empty string).

The reason that i's answer (err... I mean, sqrt-1's answer) works is that the second && causes the shell to run the TEST="$(cat test.txt)" code's return value is, notice that is zero, and only then proceed to process what comes after the second && (so then the echo "$TEST" portion will have the $TEST variable processed, and by now it all works).

Note that sqrt-1's provided answer shows some code that is a little bit wasteful, although understandably so because it does directly answer the asked question. What it basically does is:

First, splits the command into three parts:

echo "Hello world!" > test.txt

TEST="$(cat test.txt)"

echo "$TEST"

The third part only runs if the second part (runs and) succeeds, and the second part only runs if the first command succeeds, because of the two && operators.

Another variation could be:

echo "Hello world!" > test.txt && ( TEST="$(cat test.txt)" ; echo "$TEST" )

With that semi-colon, the results of the 'TEST="$(cat test.txt)"' will not be evaluated to determine whether the 'echo "$TEST"' will be run. When I see a &&, I tend to try to figure out what is going to be evaluated, and why that would matter. So, despite the added complexity of needing to use parenthesis, this variation with a semi-colon actually seems to me like it may be a bit simpler to mentally process.

Answer 2

It doesn't work because $var expansions are performed by the shell before the command line is run, i.e. not by the 'echo' command.

Meanwhile, VAR=value somecommand is not a regular assignment – it is only meant to inject an environment variable into the new process, but has no effect on shell variables before that process is created, i.e. it is completely different from VAR=value; somecommand.

So in other words, it happens in the opposite order than what your post's title describes. First the (stil-empty) "$TEST" is expanded into "", then echo "" is run with an additional TEST=... environment variable which it does not care about.

Since your test commands are supposed to imitate the behavior of Ansible, they should be actually looking at their environment variables and not relying on pre-expansion by the shell; something like VAR=value env or VAR=value printenv VAR would be a much better fit.

Answer 3

Why does the following not work in bash?

echo "Hello world!" > test.txt && TEST="$(cat test.txt)" echo "$TEST"

If you run it though ShellCheck – shell script analysis tool it will tell you why:

$ shellcheck myscript
 
Line 2:
echo "Hello world!" > test.txt && TEST="$(cat test.txt)" echo "$TEST"
                                  ^-- SC2097 (warning): This assignment is only seen by the forked process.
>>                                                             ^-- SC2098 (warning): This expansion will not see the mentioned assignment.

See ShellCheck: SC2097 – This assignment is only seen by the forked process.

Answer 4

Based on the link provided by @DavidPostill, I would use

echo "Hello world!" > test.txt && TEST="$(cat test.txt)" && echo "$TEST"

DUE EDIT
As many users rightly pointed out, mine is far to be a useful answer: It allows no one to learn anything about the topic.
Please, refer instead to @TOOGAM's post below, which is the best answer (my personal opinion) and fully exposes why @Peter Mortensen's code doesn't work, how bash variabile expansion works and even why my use of a second && Is not the optimal choice.