Is there anything faster than `find . | wc -l` to count files in a directory?

Question

Not uncommonly I have to count the number of files in a directory, sometimes this runs into the millions.

Is there a better way than just enumerating and counting them with find . | wc -l ? Is there some kind of filesystem call you can make on ext3/4 that is less I/O intensive?

Answer 1

Not a fundamental speed-up but at least something :)

find . -printf \\n | wc -l

You really do not need to pass the list of file names, just the newlines suffice. This variant is about 15 % faster on my Ubuntu 12.04.3 when the directories are cached in RAM. In addition this variant will work correctly with file names containing newlines.

Interestingly this variant seems to be a little bit slower than the one above:

find . -printf x | wc -c

Special case - but really fast

If the directory is on its own file system you can simply count the inodes:

df -i .

If the number of directories and files in other directories than the counted one do not change much you can simply subtract this known number from the current df -i result. This way you will be able to count the files and directories very quickly.

Answer 2

I have written ffcnt for exactly that purpose. It retrieves the physical offset of directories themselves with the fiemap ioctl and then scheduling the directory traversal in multiple sequential passes to reduce random access. Whether you actually get a speedup compared to find | wc depends on several factors:

• filesystem type: filesystems such as ext4 which support the fiemap ioctl will benefit most
• random access speed: HDDs benefit far more than SSDs
• directory layout: the higher the number of nested directories, the more optimization potential

(re)mounting with relatime or even nodiratime may also improve speed (for all methods) when the accesses would otherwise cause metadata updates.

Answer 3

Actually, on my system (Arch Linux) this command

   ls -A | wc -l

is faster than all of the above:

   $ time find . | wc -l
  1893

   real    0m0.027s
   user    0m0.004s
   sys     0m0.004s
   $ time find . -printf \\n  | wc -l
   1893

   real    0m0.009s
   user    0m0.000s
   sys     0m0.008s
   $ time find . -printf x  | wc -c
   1893

   real    0m0.009s
   user    0m0.000s
   sys     0m0.008s
   $ time ls -A | wc -l
   1892

   real    0m0.007s
   user    0m0.000s
   sys     0m0.004s

Answer 4

Use fd instead. It's a fast alternative to find that traverses folders in parallel

$ time find ~ -type f 2>/dev/null | wc -l
  445705
find ~ -type f 2> /dev/null  0.84s user 13.57s system 51% cpu 28.075 total
wc -l  0.03s user 0.02s system 0% cpu 28.074 total

$ time fd -t f -sHI --show-errors . ~ 2>/dev/null | wc -l
  445705
fd -t f -sHI --show-errors . ~ 2> /dev/null  2.66s user 14.81s system 628% cpu 2.780 total
wc -l  0.05s user 0.05s system 3% cpu 2.779 total

As you can see, to match the options of find in fd you'll need -sHI --show-errors. By default fd skips hidden files/folders and .gitignore and also doesn't print out permission errors so it's even far faster than this.

It's possible to tune this further by printing only a new line instead of piping the whole path. In find you can achieve that with -printf '\n'. This isn't currently supported on fd but it's a feature being requested

---

Note that in Ubuntu due to name clashing you'll need to use fdfind instead of fd. You can just alias fd=fdfind to overcome the longer name. And this obviously the above command won't work for file names containing \n. You'll need to fix it like this

fd -t f -sHI . ~ | tr '\0\n' '\n\0' | wc -l

Another nice thing about fd is that when running it in an interactive terminal you'll also get nice colorized texts unlike the output from find.