looping through `ls` results in bash shell script

Question

Does any one have a template shell script for doing something with ls for a list of directory names and looping through each one and doing something?

I'm planning to do ls -1d */ to get the list of directory names.

Answer 1

Just use a for..do..done loop:

for f in `ls`; do
  echo "File -> $f"
done

You can replace the ls with ls *.txt or any other command that returns a list (of files, directories, or anything for that matter), or actually replace it with a list (not a command that generates a list) by removing the quotes.

Within the do loop, you just refer to the loop variable with the dollar sign prefix (so $f in the above example). You can echo it or do anything else to it you want.

For example, to rename all the .xml files in the current directory to .txt:

for x in `ls *.xml`; do 
  t=`echo $x | sed 's/\.xml$/.txt/'`; 
  mv $x $t && echo "moved $x -> $t"
done

Answer 2

Using the output of ls to get filenames is a bad idea. It can lead to malfunctioning and even dangerous scripts. This is because a filename can contain any character except / and the nullcharacter, and ls does not use either of those characters as delimiters, so if a filename has a space or a newline, you will get unexpected results.

There are two very good ways of iterating over files. Here, I've used simply echo to demonstrate doing something with the filename; you can use anything, though.

The first is to use the shell's native globbing features.

for dir in */; do
  echo "$dir"
done

The shell expands */ into separate arguments that the for loop reads; even if there is a space, newline, or any other strange character in the filename, for will see each complete name as an atomic unit; it's not parsing the list in any way.

If you want to go recursively into subdirectories, then this won't do unless your shell has some extended globbing features (such as bash's globstar. If your shell doesn't have these features, or if you want to ensure that your script will work on a variety of systems, then the next option is to use find.

find . -type d -exec echo '{}' \;

Here, the find command will call echo and pass it an argument of the filename. It does this once for each file it finds. As with the previouos example, there is no parsing of a list of filenames; instead, a fileneame is sent completely as an argument.

The syntax of the -exec argument looks a little funny. find takes the first argument after -exec and treats that as the program to run, and every subsequent argument, it takes as an argument to pass to that program. There are two special arguments that -exec needs to see. The first one is {}; this argument gets replaced with a filename that the previous parts of find generates. The second one is ;, which lets find know this is the end of the list of arguments to pass to the program; find needs this because you can continue with more arguments to find that are not intended for the exec'd program. The reason for the \ is that the shell also treats ; specially - it represents the end of a command, so we need to escape it so that the shell gives it as an argument to find rather than consuming it for itself; another way of getting the shell to not treat it specially is to put it in in quotes: ';' works just as well as \; for this purpose.

Answer 3

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:

ls | parallel echo {} is in this dir

To rename all .txt to .xml:

ls *.txt | parallel mv {} {.}.xml

Watch the intro video for GNU Parallel to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

Answer 4

For files with spaces in you will have to make sure to quote the variable like:

 for i in $(ls); do echo "$i"; done;

or, you can change the input field separator (IFS) environment variable:

 IFS=$'\n';for file in $(ls); do echo $i; done

Finally, depending on what you're doing, you may not even need the ls:

 for i in *; do echo "$i"; done;

Answer 5

Why not set IFS to a newline, then capture the output of ls in an array? Setting IFS to newline should resolve issues with funny characters in file names; using ls can be nice because it has a built-in sort facility.

(In testing I was having trouble setting IFS to \n but setting it to newline backspace works, as suggested elsewhere here):

E.g. (assuming desired ls search pattern passed in $1):

SAVEIFS=$IFS
IFS=$(echo -en "\n\b")

FILES=($(/bin/ls "$1"))

for AFILE in ${FILES[@]}
do
    ... do something with a file ...
done

IFS=$SAVEIFS

This is especially handy on OS X, e.g., to capture a list of files sorted by creation date (oldest to newest), the ls command is ls -t -U -r.

Answer 6

This is how I do it, but there are probably more efficient ways.

ls > filelist.txt

while read filename; do echo filename: "$filename"; done < filelist.txt