Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

How does one properly iterate over lines in bash either in a variable, or from the output of a command? Simply setting the IFS variable to a new line works for the output of a command but not when processing a variable that contains new lines.

For example

#!/bin/bash

list="One\ntwo\nthree\nfour"

#Print the list with echo
echo -e "echo: \n$list"

#Set the field separator to new line
IFS=$'\n'

#Try to iterate over each line
echo "For loop:"
for item in $list
do
        echo "Item: $item"
done

#Output the variable to a file
echo -e $list > list.txt

#Try to iterate over each line from the cat command
echo "For loop over command output:"
for item in `cat list.txt`
do
        echo "Item: $item"
done

This gives the output:

echo: 
One
two
three
four
For loop:
Item: One\ntwo\nthree\nfour
For loop over command output:
Item: One
Item: two
Item: three
Item: four

As you can see, echoing the variable or iterating over the cat command prints each of the lines one by one correctly. However, the first for loop prints all the items on a single line. Any ideas?

share|improve this question
add comment

4 Answers

up vote 35 down vote accepted

With bash, if you want to embed newlines in a string, enclose the string with $'':

$ list="One\ntwo\nthree\nfour"
$ echo "$list"
One\ntwo\nthree\nfour
$ list=$'One\ntwo\nthree\nfour'
$ echo "$list"
One
two
three
four

And if you have such a string already in a variable, you can read it line-by-line with:

while read -r line; do
    echo "... $line ..."
done <<< "$list"
share|improve this answer
    
I think this is the most sensible choice in my case. Thanks! –  Alex Spurling May 16 '11 at 14:22
3  
Just a note that the done <<< "$list" is crucial –  Jason Axelson Sep 26 '12 at 7:53
add comment

You can use while + read:

some_command | while read line ; do
   echo === $line ===
done

Btw. the -e option to echo is non-standard. Use printf instead, if you want portability.

share|improve this answer
1  
Note that if you use this syntax, variables assigned inside the loop won't stick after the loop. Oddly enough, the <<< version suggested by glenn jackman does work with variable assignment. –  Sparhawk Oct 3 '13 at 4:51
2  
@Sparhawk Yes, that's because the pipe starts a subshell executing the while part. The <<< version does not (in new bash versions, at least). –  maxelost Oct 21 '13 at 13:53
add comment

Here's a funny way of doing your for loop:

for item in ${list//\\n/
}
do
   echo "Item: $item"
done

A little more sensible/readable would be:

cr='
'
for item in ${list//\\n/$cr}
do
   echo "Item: $item"
done

But that's all too complex, you only need a space in there:

for item in ${list//\\n/ }
do
   echo "Item: $item"
done

You $line variable doesn't contain newlines. It contains instances of \ followed by n. You can see that clearly with:

$ cat t.sh
#! /bin/bash
list="One\ntwo\nthree\nfour"
echo $list | hexdump -C

$ ./t.sh
00000000  4f 6e 65 5c 6e 74 77 6f  5c 6e 74 68 72 65 65 5c  |One\ntwo\nthree\|
00000010  6e 66 6f 75 72 0a                                 |nfour.|
00000016

The substitution is replacing those with spaces, which is enough for it to work in for loops:

$ cat t.sh
#! /bin/bash
list="One\ntwo\nthree\nfour"
echo ${list//\\n/ } | hexdump -C

$ ./t.sh 
00000000  4f 6e 65 20 74 77 6f 20  74 68 72 65 65 20 66 6f  |One two three fo|
00000010  75 72 0a                                          |ur.|
00000013

Demo:

$ cat t.sh
#! /bin/bash
list="One\ntwo\nthree\nfour"
echo ${list//\\n/ } | hexdump -C
for item in ${list//\\n/ } ; do
    echo $item
done

$ ./t.sh 
00000000  4f 6e 65 20 74 77 6f 20  74 68 72 65 65 20 66 6f  |One two three fo|
00000010  75 72 0a                                          |ur.|
00000013
One
two
three
four
share|improve this answer
    
Interesting, can you explain what is going on here? It looks like you are replacing \n with a new line... What is the difference between the original string and the new one? –  Alex Spurling May 16 '11 at 13:15
    
@Alex: updated my answer - with a simpler version too :-) –  Mat May 16 '11 at 13:24
    
I tried this out and it doesn't appear to work if you have spaces in your input. In the example above we have "one\ntwo\nthree" and this works but it fails if we have "entry one\nentry two\nentry three" as it also adds a new line for the space too. –  John Rocha Nov 8 '12 at 18:15
    
Just a note that instead of defining cr you can use $'\n'. –  chaiguy Dec 21 '13 at 19:12
add comment
#!/bin/sh

items="
one two three four
hello world
this should work just fine
"

IFS='
'
count=0
for item in $items
do
  count=$((count+1))
  echo $count $item
done
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.