How can I make the bash backtick operator keep newlines in output?

Question

Is there a way to make bash not eat newlines in the backtick substitution?

For example:

var=`echo line one && echo line two`
echo $var

line one line two

What I want is

var=`echo line one && echo line two` # plus some magic
echo $var

line one
line two

Answer 1

It's not an issue with backticks substitution, but with echo; you have to quote the variable to get the control characters working:

$ var=`echo line one && echo line two`
$ echo "$var"
line one
line two

Answer 2

In brief, what is needed is to add a suffix to end of the output when capturing it, and remove it when using the variable, and always quote variables:

# add a suffix to the end of the output when capturing it
var=$( cat /no/file/here 2>&1; echo extra)
# when using the variable, remove the suffix, and quote the variable
echo -n "${var%extra}"

Compare it to the following, in which the output from the second command will not end with a newline.

var=$( cat /no/file/here 2>&1)
echo -n "${var}"

Explanation of what is happening

Newlines in string

First determine the byte sequence that is expected:

$ { echo a; echo b; } | xxd
0000000: 610a 620a                                a.b.

Now, use Command Substitution (Section 3.5.4) to attempt to capture this byte sequence:

$ x=$( echo a; echo b; )

Then, do a simple echo to verify the byte sequence:

$ echo $x | xxd
0000000: 6120 620a                                a b.

So it seems the first newline was replaced with a space, and the second newline was left intact. But why ?

Lets look at what is actually happening here:

First, bash will do Shell Parameter Expansion (Section 3.5.3)

The ‘$’ character introduces parameter expansion, command substitution, or arithmetic expansion. The parameter name or symbol to be expanded may be enclosed in braces, which are optional but serve to protect the variable to be expanded from characters immediately following it which could be interpreted as part of the name.

Then bash will do Word Splitting (Section 3.5.7)

The shell scans the results of parameter expansion, command substitution, and arithmetic expansion that did not occur within double quotes for word splitting.

The shell treats each character of $IFS as a delimiter, and splits the results of the other expansions into words on these characters. If IFS is unset, or its value is exactly , ...

Next, bash will treat it as a Simple Command (Section 3.2.1)

A simple command is the kind of command encountered most often. It’s just a sequence of words separated by blanks, terminated by one of the shell’s control operators (see Definitions). The first word generally specifies a command to be executed, with the rest of the words being that command’s arguments.

The definition of blanks (Section 2 - Definitions)

blank A space or tab character.

Finally, bash invokes the echo (Section 4.2 - Bash Builtin Commands) internal command

... Output the args, separated by spaces, terminated with a newline. ...

So to summarise, the newlines are removed by Word Splitting, and then echo gets 2 args, "a", and "b", and then outputs them seperated by spaces and ending with a newline.

Doing what @cyrus suggested (and suppress the newline from echo with -n) the result is better:

$ echo -n "$x" | xxd
0000000: 610a 62                                  a.b

Newlines at end of string

Its still not perfect though, the trailing newline is gone. Looking closer at Command Substitution (Section 3.5.4):

Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command, with any trailing newlines deleted.

Now that its clear why the newline goes away, bash can be fooled into keeping it. To do this, add an additional string to the end, and remove it when using the variable:

$ x=$( echo a; echo b; echo -n extra )
$ echo -n "${x%extra}" | xxd
0000000: 610a 620a                                a.b.

Another example

$ cat /no/file/here 2>&1 | xxd
0000000: 6361 743a 202f 6e6f 2f66 696c 652f 6865  cat: /no/file/he
0000010: 7265 3a20 4e6f 2073 7563 6820 6669 6c65  re: No such file
0000020: 206f 7220 6469 7265 6374 6f72 790a        or directory.
$ cat /no/file/here 2>&1 | cksum
3561909523 46
$ 
$ var=$( cat /no/file/here 2>&1; rc=${?}; echo extra; exit ${rc})
$ echo $?
1
$ 
$ echo -n "${var%extra}" | xxd
0000000: 6361 743a 202f 6e6f 2f66 696c 652f 6865  cat: /no/file/he
0000010: 7265 3a20 4e6f 2073 7563 6820 6669 6c65  re: No such file
0000020: 206f 7220 6469 7265 6374 6f72 790a        or directory.
$ echo -n "${var%extra}" | cksum
3561909523 46

Answer 3

I would have put my small addition in a comment on the answer of cYrus, but I cannot yet comment on answers I think. So I'll duplicate part of cYrus his answer for completeness.

What happens in the first line of your code is that indeed the backticks substitution eats the trailing newlines of the output of the substituted command, as documented. But it does not eat the newline between your two lines.

What happens in the second line of code is that echo is executed with two arguments, the first and the second line. Quoting the expanded variable will fix that: echo "$var" This will preserve the newline between your line one and line two. And echo will add a trailing newline by default so that everything is okay.

Now if you wanted to preserve all trailing newlines in the output of some command substitution, a workaround is to add another line the output, which you can remove after the command substitution, see here .