2

I want to count the files in one folder, however there are too many files, so ls -1 | wc -l takes more than a few minutes. Is there any faster method? Preferably one with a lower complexity?

I would prefer a solution that can be run from the command-line, however other solutions would still help.

Edit: The above mentioned solution does not work at all:

$ ls -1 | wc -l
ls: memory exhausted
0
  • Having do many files in a single folder is not a good idea altogether - every operation on this folder will take ages. – Eugen Rieck Apr 15 at 11:43
  • ls -1 suggests you're after directories, symlinks, named fifos etc. as well as regular files; non-recursively. Please confirm. – Kamil Maciorowski Apr 15 at 11:45
  • @KamilMaciorowski correct. However a solution only including files will also work in this case. – Max Matti Apr 15 at 12:49
  • 1
    Try ls -U instead. – CodeIt Apr 15 at 12:54
  • 1
    @CodeIt $ ls -1 -U keys | wc -l did finally show that it's around 18m files. However it's still not that fast. I will try the c program later. – Max Matti Apr 15 at 20:07
2

One way to use find and count:

find . -type f | wc -l
  • 1
    I suspect if ls runs OOM then find will do not much better. – Eugen Rieck Apr 15 at 13:09
1

Maybe ask Python:

python -c "import glob; print len(glob.glob('*'))"

Notes:

  • Double quotes outside and single quotes around pattern or vice-versa.
  • Pattern can include a path: '/tmp/*'

I'm curious to know which is faster compared to find, I cannot create a big enough directory to make a sensible measurement.

Of course this is also subject to possible OOM...

Edit: tested with almost 1M files in a tmpfs:

▶▶time python -c "import glob; print len(glob.glob('*'))"
999990

real    0m0.774s
user    0m0.471s
sys     0m0.303s
▶▶time find . | wc -l
999992

real    0m0.522s
user    0m0.231s
sys     0m0.324s

So find is faster. Btw, when it works, ls is much slower than either.

0

I did try some of the suggestions using the zsh time function:

$ time ./getdents keys | wc -l
15228414
./getdents keys  0,29s user 4,34s system 91% cpu 5,061 total
wc -l  0,11s user 0,13s system 4% cpu 5,061 total


$ time find keys -type f | wc -l
19144815
find keys -type f  10,38s user 7,82s system 3% cpu 7:42,05 total
wc -l  0,21s user 0,25s system 0% cpu 7:42,08 total


$ time ls -1 -U keys | wc -l
16986019
ls --color=tty -1 -U keys  3,88s user 6,69s system 2% cpu 7:53,60 total
wc -l  0,21s user 0,22s system 0% cpu 7:53,60 total


$ time python -c "import glob; print len(glob.glob('*'))"
17750179
python -c "import glob; print len(glob.glob('*'))"  8,20s user 12,37s system 1% cpu 32:00,29 total

The numbers they report differ because I was copying files out of the large directory at the same time (and because the find command also lists files in subdirectories which can be fixed by using -maxdepth=1) and did not start all programs at the same time. However a general trend is clearly visible. The first one is the C variant found in this thread linked by CodeIt.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.