3

How to distinguish a file from a directory in a ls output? I want to work with files and go into directories, however, I just get a list of names of them all:

for i in ls B 
do
  echo $i
done
  • 2
    at a minimum you'd have to use ls -l – Mokubai Feb 23 '17 at 19:07
  • In general, attempting to parse ls is a bad idea. It tends to break on filenames with whitespace, or that look like globs. – 8bittree Feb 23 '17 at 20:52
2

From the ls man page you can see which entries are directories using

  -F, --classify
          append indicator (one of */=>@|) to entries

So if you use

for i in $(ls -F B) ; do
    echo $i
done

You should see that directories have a / appended, and other files do not.


However, if you want to descend into directories, it may be better to use test

for f in $(ls B) ; do
    if [ -d $f ] ; then
        recurse_into_directory
    elif [ -f $f ]
        process_file
    else
        echo "$f: neither regular file nor directory"
    fi
done
| improve this answer | |
  • 1
    You can avoid the problems with parsing ls altogether by using for f in B/* ; do instead at the beginning of your test example. – 8bittree Feb 23 '17 at 20:29
  • Great answer! Could you please tell me how to check if the file name follows some pattern? I tried elif [ -f $f ] -and -regex ".*$Patern.*"; then but it is never true... – Kosarar Feb 26 '17 at 19:08
  • superuser.com/questions/1183035/… – Kosarar Feb 26 '17 at 19:20
0

You write that you "want to work with files and go into directories", so jumping straight to ls as your solution may be premature. It would be helpful to know exactly what you mean by "work with files and go into directories" to give the best solution.

Here are a couple for common use cases, though:

Manipulating individual files recursively

Suppose you want to do something to every file matching some criterion starting in the current directory and continuing into every sub directory.

For example: find the line count of every file with a .txt extension. The command for getting a line count of a single file is wc -l $filename. (If you give it multiple file names, it will output the line count of each individually followed by the total.)

So that's how to solve the problem with one file—which is always the first question you have to answer before you can move on—but, how to do it on all files recursively? This part of the problem is solved with the find command, the Unix command for directory traversal.

find can be difficult command to learn in detail, but for simple cases such as this, it's quite easy. The first thing to know is that every find command is in the following format:

find DIR [PREDICATE, ..]

DIR is the starting directory (for this example, ., which is always the current working directory). A PREDICATE is an expression the find uses to either decide what to do next when considering a file or directory, or to do something with that file or directory.

The basic algorithm find follows is: try the first (leftmost in the command line) predicate on the current item (file or directory) being examined. If the predicate is true, then try the next predicate in the command line. Continue until all predicates given have been tried. If a predicate is false, then stop working with this item and start again with the next item (starting again with the first predicate).

If the item being examined is a directory, then once the last predicate has been reached or a predicate is false, find continues with the items inside the directory. There are two main exceptions to this:

  1. The -prune predicate can be used to selectively disable this; if the -prune predicate is reached and the current item is a directory, or

  2. The -maxdepth=N option (not a predicate, it appears before DIR in the command line) can be used to limit how deep find will search; if the current directory is N or more levels deeper than the starting directory,

    then in either case, the directory's contents (and sub-contents, recursively) are not examined, and the next item will be the same as if the current item were a file rather than a directory.

Speaking of: if the item being examined is a file, the "next item" is the next entry in the same directory, or, if there are no items remaining in the directory, the current directory is "popped" out of and processing continues with the next item being whatever the next item would have been when the directory was entered.

What does "processing an item" mean? It means that each predicate is tried, left to right in the command line, until one is false, or all have been tried.

(At this point there's a divergence between some different versions of find. In many newer ones, such as the version found on Linux, if the last predicate is true and was not an "action" predicate, then find assumes you meant to do something, so it acts as if the -print predicate were given to cause the pathname to be printed out. In older versions of find, this was not the case, and the result of such a item's processing would be nil.

To illustrate: the simplest command find . without any predicates. On the newer variants of find, this will result in a list of all pathnames starting in the current directory and progressing recursively until all have been printed. On the older variants of find, the same command will take just as long to run (it has to check all the files recursively against the—in this case, nonexistent—predicates), but will output absolutely nothing.)

Before leaving the subject of processing predicates, I'll note that my explanation thus far has made it sound like the only possibility for predicates is AND-ing them logically. This isn't true because

  • there is also a -o predicate that ORs two predicates (actually, there's an -a AND predicate as well, but it's rarely needed because as I wrote above, that's the default behavior);
  • find allows the use of parentheses (which, because of shell-escape rules, are ordinarily written \( and \)) to group several predicates into one expression; and
  • there is a negation operator which is usually written \!.

With all that out of the way, we can now return to the question of how to get the line count of every file with a .txt suffix:

  1. As mentioned, the command for getting the line count of a file is wc -l.
  2. There is a predicate available for running a command on the file currently being examined by find. It is -exec CMD ;, including the semicolon (which must be escaped if necessary), and in the text of CMD will replace any occurrence of the token {} with the pathname currently being examined.
  3. Another predicate lets us check for the suffix of a file: -name PATTERN. So in this case, where we want files with a .txt extension, we use *.txt as our pattern.

So, knowing all this, the command we can write is:

find . -name '*.txt' -exec wc -l {} \;

(We use quotes around *.txt and a backslash before the semicolon both in order to prevent the shell from interpreting those characters as special so that find can see them.) This will check the line count of every file so named recursively.

There's a small wrinkle here that depending on the context you may be free to ignore: what if you had a directory named something ending in .txt? You'll get something like the following:

$ find . -name '*.txt' -exec wc -l {} \; 
42 ./myfile.txt
wc: ./foo.txt: Is a directory
0 ./foo.txt
1 ./foo.txt/bar.txt

To fix this, you must add another predicate, -type f, to tell find to only do the -exec predicate on files that are normal text files:

$ find . -type f -name '*.txt' -exec wc -l {} \;
42 ./myfile.txt
1 ./foo.txt/bar.txt

(You might wonder if it matters whether the -type f appears before or after the -name '*.txt' predicate. It does not, because directories are always descended into, unless the -prune or -maxdepth is present, as mentioned previously.)

Note that the above is possible using ls in combination with advanced features of the Bash or Zsh shells. But those solutions are much harder to explain and to get right, so I'm going to assume that your mentioning ls was premature implementation. (See the XY problem.)

Gathering a list of files, then manipulating them together

I mentioned that, if given more than one file name, wc -l gives a file-by-file count, followed by a grand total. But the above solution didn't get a grand total, because wc was being run once for each file named *.txt. But what if you wanted that grand total?

In this case, you could use ls, but you'd face a problem: if any of your filenames potentially contained spaces or other characters that are special to the shell, you could get an error or even inadvertently run a command you didn't mean to.

So once again, it's better to turn to find. Newer versions of find (mostly, the same ones that I previously mentioned would insert -print for you if you left it out) have a feature for this: use the -exec predicate as before, but instead of ending with a semicolon, end with a plus (+). So:

$ find . -type f -name '*.txt' -exec wc -l {} \+
  42 ./myfile.txt
   1 ./foo.txt/bar.txt
  43 total

For those version of find lacking this feature, you'd use find in conjunction with another program, xargs. xargs takes its input and runs a command with the input given as the command's arguments. So here's how we'd use it to replicate our first command:

$ find . -type f -name '*.txt' -print | xargs wc -l
  42 ./myfile.txt
   1 ./foo.txt/bar.txt
  43 total

This command still has a problem, though, if one of the filenames contains a space:

$ ls
My Spacey File.txt  foo.txt  myfile.txt  rakudo-info.md
$ find . -type f -name '*.txt' -print | xargs wc -l
  42 ./myfile.txt
wc: ./My: No such file or directory
wc: Spacey: No such file or directory
wc: File.txt: No such file or directory
   1 ./foo.txt/bar.txt
  43 total

In this case, wc saw each word of the file name My Spacey File.txt as a separate argument. To fix this, we use a feature of find and a corresponding feature of xargs that uses the null character (\0, which is illegal in filenames) as the delimiter instead of newlines:

$ find . -type f -name '*.txt' -print0 | xargs -0 wc -l
  42 ./myfile.txt
   1 ./My Spacey File.txt
   1 ./foo.txt/bar.txt
  44 total

The -print0 predicate tells find to send its output delimited by nulls; the -0 option of xargs does the same for its input.

A final caveat

If you have a very large number of files, or the total number of characters of all the filenames in aggregate is very large, you could run into limits of the number or size of arguments allowed by the system. In this case, both the -exec ... \+ predicate of find and xargs will split the list and run the command multiple times so that each file name is used once.

On modern systems, this limit is sufficiently large that you needn't worry about it until you get into the thousands of file names at least.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.