209

How does one properly iterate over lines in bash either in a variable, or from the output of a command? Simply setting the IFS variable to a new line works for the output of a command but not when processing a variable that contains new lines.

For example

#!/bin/bash

list="One\ntwo\nthree\nfour"

#Print the list with echo
echo -e "echo: \n$list"

#Set the field separator to new line
IFS=$'\n'

#Try to iterate over each line
echo "For loop:"
for item in $list
do
        echo "Item: $item"
done

#Output the variable to a file
echo -e $list > list.txt

#Try to iterate over each line from the cat command
echo "For loop over command output:"
for item in `cat list.txt`
do
        echo "Item: $item"
done

This gives the output:

echo: 
One
two
three
four
For loop:
Item: One\ntwo\nthree\nfour
For loop over command output:
Item: One
Item: two
Item: three
Item: four

As you can see, echoing the variable or iterating over the cat command prints each of the lines one by one correctly. However, the first for loop prints all the items on a single line. Any ideas?

267

With bash, if you want to embed newlines in a string, enclose the string with $'':

$ list="One\ntwo\nthree\nfour"
$ echo "$list"
One\ntwo\nthree\nfour
$ list=$'One\ntwo\nthree\nfour'
$ echo "$list"
One
two
three
four

And if you have such a string already in a variable, you can read it line-by-line with:

while read -r line; do
    echo "... $line ..."
done <<< "$list"
  • 28
    Just a note that the done <<< "$list" is crucial – Jason Axelson Sep 26 '12 at 7:53
  • 20
    The reason done <<< "$list" is crucial is because that will pass "$list" as the input to read – wisbucky Mar 11 '15 at 21:00
  • 21
    I need to stress this: DOUBLE-QUOTING $list is very crucial. – André Chalella Jun 10 '15 at 7:44
  • 8
    echo "$list" | while ... could appear more clear than ... done <<< "$line" – kyb May 14 '18 at 10:07
  • 4
    There are downsides to that approach as the while loop will run in a subshell: any variables assigned in the loop will not persist. – glenn jackman May 14 '18 at 14:13
60

You can use while + read:

some_command | while read line ; do
   echo === $line ===
done

Btw. the -e option to echo is non-standard. Use printf instead, if you want portability.

  • 10
    Note that if you use this syntax, variables assigned inside the loop won't stick after the loop. Oddly enough, the <<< version suggested by glenn jackman does work with variable assignment. – Sparhawk Oct 3 '13 at 4:51
  • 7
    @Sparhawk Yes, that's because the pipe starts a subshell executing the while part. The <<< version does not (in new bash versions, at least). – maxelost Oct 21 '13 at 13:53
25
#!/bin/sh

items="
one two three four
hello world
this should work just fine
"

IFS='
'
count=0
for item in $items
do
  count=$((count+1))
  echo $count $item
done
  • 2
    This outputs all items, not lines. – Tomasz Posłuszny Aug 13 '15 at 15:05
  • 4
    @TomaszPosłuszny No, this outputs 3 (count is 3) lines on my machine. Note the setting of the IFS. You could also use IFS=$'\n' for the same effect. – jmiserez May 10 '16 at 12:33
10

Here's a funny way of doing your for loop:

for item in ${list//\\n/
}
do
   echo "Item: $item"
done

A little more sensible/readable would be:

cr='
'
for item in ${list//\\n/$cr}
do
   echo "Item: $item"
done

But that's all too complex, you only need a space in there:

for item in ${list//\\n/ }
do
   echo "Item: $item"
done

You $line variable doesn't contain newlines. It contains instances of \ followed by n. You can see that clearly with:

$ cat t.sh
#! /bin/bash
list="One\ntwo\nthree\nfour"
echo $list | hexdump -C

$ ./t.sh
00000000  4f 6e 65 5c 6e 74 77 6f  5c 6e 74 68 72 65 65 5c  |One\ntwo\nthree\|
00000010  6e 66 6f 75 72 0a                                 |nfour.|
00000016

The substitution is replacing those with spaces, which is enough for it to work in for loops:

$ cat t.sh
#! /bin/bash
list="One\ntwo\nthree\nfour"
echo ${list//\\n/ } | hexdump -C

$ ./t.sh 
00000000  4f 6e 65 20 74 77 6f 20  74 68 72 65 65 20 66 6f  |One two three fo|
00000010  75 72 0a                                          |ur.|
00000013

Demo:

$ cat t.sh
#! /bin/bash
list="One\ntwo\nthree\nfour"
echo ${list//\\n/ } | hexdump -C
for item in ${list//\\n/ } ; do
    echo $item
done

$ ./t.sh 
00000000  4f 6e 65 20 74 77 6f 20  74 68 72 65 65 20 66 6f  |One two three fo|
00000010  75 72 0a                                          |ur.|
00000013
One
two
three
four
  • Interesting, can you explain what is going on here? It looks like you are replacing \n with a new line... What is the difference between the original string and the new one? – Alex Spurling May 16 '11 at 13:15
  • @Alex: updated my answer - with a simpler version too :-) – Mat May 16 '11 at 13:24
  • I tried this out and it doesn't appear to work if you have spaces in your input. In the example above we have "one\ntwo\nthree" and this works but it fails if we have "entry one\nentry two\nentry three" as it also adds a new line for the space too. – John Rocha Nov 8 '12 at 18:15
  • 2
    Just a note that instead of defining cr you can use $'\n'. – devios1 Dec 21 '13 at 19:12
0

You can also first convert the variable into an array, then iterate over this.

lines="abc
def
ghi"

declare -a theArray

while read -r line
do
    theArray+=($line)            
done <<< "$lines"

for line in "${theArray[@]}"
do
    echo "$line"
    #Do something complex here that would break your read loop
done

This is only usefull if you do not want to mess with the IFS and also have issues with the read command, as it can happen, if you call another script inside the loop that that script can empty your read buffer before returning, as it happened to me.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.