31

I've got a filesystem which has a couple million files and I'd like to see a distribution of file sizes recursively in a particular directory. I feel like this is totally doable with some bash/awk fu, but could use a hand. Basically I'd like something like the following:

1KB: 4123
2KB: 1920
4KB: 112
...
4MB: 238
8MB: 328
16MB: 29138
Count: 320403345

I feel like this shouldn't be too bad given a loop and some conditional log2 filesize foo, but I can't quite seem to get there.

Related Question: How can I find files that are bigger/smaller than x bytes?.

6 Answers 6

36

This seems to work pretty well:

find . -type f -print0 | xargs -0 ls -l | awk '{size[int(log($5)/log(2))]++}END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | sort -n

Its output looks like this:

         0   1
         8   3
        16   2
        32   2
        64   6
       128   9
       256   9
       512   6
      1024   8
      2048   7
      4096  38
      8192  16
     16384  12
     32768   7
     65536   3
    131072   3
    262144   3
    524288   6
   2097152   2
   4194304   1
  33554432   1
 134217728   4
where the number on the left is the lower limit of a range from that value to twice that value and the number on the right is the number of files in that range.

5
  • I edited your answer to use find instead of ls so that it was recursive and didn't do any directory counting. Anyone want to take a crack at prettying up the left hand column output?
    – notpeter
    Mar 13, 2013 at 20:49
  • But the original question was about "distribution of file sizes in a particular directory", so it's not OK to change the ls to a find. I'm putting it back the way it was.
    – garyjohn
    Mar 13, 2013 at 21:15
  • @notpeter: Sorry, I didn't recognize you as the author of the question. I changed my answer to make it search recursively. On my system, though, using xargs is significantly faster than -exec, so I used that method.
    – garyjohn
    Mar 13, 2013 at 23:01
  • 1
    No worries. Now we can just delete our comments are pretended it was always the right answer. ;)
    – notpeter
    Mar 14, 2013 at 16:29
  • Instead of parsing ls's output, you may use stat --printf "%s\n" nsemail.eml to get the file size.
    – audeoudh
    Dec 1, 2021 at 10:48
24

Based on garyjohn's answer, here is a one-liner, which also formats the output to human readable:

find . -type f -print0 | xargs -0 ls -l | awk '{ n=int(log($5)/log(2)); if (n<10) { n=10; } size[n]++ } END { for (i in size) printf("%d %d\n", 2^i, size[i]) }' | sort -n | awk 'function human(x) { x[1]/=1024; if (x[1]>=1024) { x[2]++; human(x) } } { a[1]=$1; a[2]=0; human(a); printf("%3d%s: %6d\n", a[1],substr("kMGTEPYZ",a[2]+1,1),$2) }'

Here is the expanded version of it:

find . -type f -print0                                                   \ 
 | xargs -0 ls -l                                                        \
 | awk '{ n=int(log($5)/log(2));                                         \
          if (n<10) n=10;                                                \
          size[n]++ }                                                    \
      END { for (i in size) printf("%d %d\n", 2^i, size[i]) }'           \
 | sort -n                                                               \ 
 | awk 'function human(x) { x[1]/=1024;                                  \
                            if (x[1]>=1024) { x[2]++;                    \
                                              human(x) } }               \
        { a[1]=$1;                                                       \ 
          a[2]=0;                                                        \
          human(a);                                                      \
          printf("%3d%s: %6d\n", a[1],substr("kMGTEPYZ",a[2]+1,1),$2) }' 

In the first awk I defined a minimum file size to collect all the files less than 1kb to one place. In the second awk, function human(x) is defined to create a human readable size. This part is based on one of the answers here: https://unix.stackexchange.com/questions/44040/a-standard-tool-to-convert-a-byte-count-into-human-kib-mib-etc-like-du-ls1

The sample output looks like:

  1k:    335
  2k:     16
 32k:      5
128k:     22
  1M:     54
  2M:     11
  4M:     13
  8M:      3
1
  • I chose this solution in order to find out which files are 128K and under in order to set the recordsize of an OS installation on ZFS. I'm afraid it's experiencing slowdown from having many files smaller than its 128K record size, but it's hard to know if that's why without counting the files. Feb 29, 2020 at 1:45
4

Try this:

find . -type f -exec ls -lh {} \; | 
 gawk '{match($5,/([0-9.]+)([A-Z]+)/,k); if(!k[2]){print "1K"} \
        else{printf "%.0f%s\n",k[1],k[2]}}' | 
sort | uniq -c | sort -hk 2 

OUTPUT :

 38 1K
 14 2K
  1 30K
  2 62K
  12 2M
  2 3M
  1 31M
  1 46M
  1 56M
  1 75M
  1 143M
  1 191M
  1 246M
  1 7G

EXPLANATION :

  • find . -type f -exec ls -lh {} \; : simple enough, find files in the current dir and run ls -lh on them

  • match($5,/([0-9.]+)([A-Z]+)/,k); : this will extract the file size, and save each match into the array k.

  • if(!k[2]){print "1K"} : if k[2] is undefined the file size is <1K. Since I am imagining you don't care about such tiny sizes, the script will print 1K for all files whose size is <=1K.

  • else{printf "%.0f%s\n",k[1],k[2]} : if the file is larger than 1K, round the file size to the closest integer and print along with its modifier (K,M, or G).

  • sort | uniq -c : count the occurrences of each line (file size) printed.

  • sort -hk 2 : sort according to the second field in human readable format. This way, 7G is sorted after 8M.

3
  • I appreciate the explanations, I think it's helpful for people trying to figure it out. That said, your script doesn't work for me for two reasons 1) My GNU LS is old and so gives different human readable size output for 'ls -lh' (bytes not K/M/G/T) and 2) because there's too many buckets. With file sizes between 1K and 1G there are 2000 buckets, half of which are 1KB half of which are 1MB. Worth it though for 'uniq -c' that's new to me.
    – notpeter
    Mar 13, 2013 at 21:00
  • Very slow but works great. Thanks!
    – Dawid
    May 19, 2020 at 20:06
  • 1
    @dawid oh wow, thanks but this is actually quite bad. It will break on any slightly hard filename. If there's a space in the name, for example, let alone if there's a newline. And it won't work for different locales, it is really slow (as you found out). Kind of embarrassing, really. In my defense, this was years ago! I'll try and whip up a better one tomorrow.
    – terdon
    May 19, 2020 at 21:31
1

Building on top of a couple of the earlier answers, a script (longer) with more detailed output.

$ cat ~/bin/fszdist
#!/bin/bash

AWK_SCRIPT='
# prettyfier
function psize(s) {
    kilo = 0;
    mega = 0;
    giga = 0;
    if (s >= 1024) {
      s /= 1024;
      kilo = 1;
    }
    if (s >= 1024) {
      s /= 1024;
      mega = 1;
    }
    if (s >= 1024) {
      s /= 1024;
      giga = 1;
    }
    suffix = (giga == 1) ? "g" : ((mega == 1) ? "m" : ((kilo == 1) ? "k" : ""))
    return int(100*s + 0.5)/100 suffix
}

{
  i = ($5 == 0) ? -1 : int(log($5)/log(2));
  ++sz[i];
  bytes[i] += $5
}
END {
  acc=0;
  bs_acc = 0;
  n=length(sz);
  printf("%11s %10s %10s %10s %10s\n", "[n, 2*n)", "count", "acc_count", "bytes", "acc_bytes");
  for (i=-1; i < n; ++i) {
    acc += sz[i];
    bs_acc += bytes[i]
    p = (i == -1) ? 0 : 2^i;
    printf("%11s %10d %10d %10s %10s\n", psize(p), sz[i], acc, psize(bytes[i]), psize(bs_acc));
  }
}
'

find . -type f -print0 | \
    xargs -0 ls -l | \
    awk "$AWK_SCRIPT"

Running it on a directory cloned from some github repository:

$ fszdist
   [n, 2*n)      count  acc_count      bytes  acc_bytes
          0        329        329          0          0
          1         29        358         29         29
          2         53        411        121        150
          4          4        415         24        174
          8         41        456        489        663
         16         62        518      1.39k      2.04k
         32        829       1347     38.33k     40.37k
         64        716       2063     69.21k    109.58k
        128       2013       4076    394.55k    504.13k
        256       3761       7837      1.37m      1.86m
        512       5758      13595      4.15m      6.02m
         1k      10728      24323     15.93m     21.95m
         2k      15539      39862     42.84m     64.78m
         4k      10321      50183     59.33m    124.11m
         8k       8694      58877     90.85m    214.96m
        16k       3585      62462     75.22m    290.18m
        32k       3741      66203    154.93m    445.11m
        64k        571      66774     47.59m     492.7m
       128k        454      67228     79.71m    572.41m
       256k        235      67463     88.38m    660.79m
       512k        109      67572     77.87m    738.65m
         1m         58      67630     88.43m    827.08m
         2m         68      67698    179.28m   1006.36m
         4m         44      67742    245.79m      1.22g
         8m          9      67751    102.13m      1.32g
        16m         14      67765     287.1m       1.6g
        32m          8      67773    392.46m      1.99g
        64m         11      67784      1.02g         3g
       128m         11      67795      1.83g      4.83g
       256m          2      67797     583.2m       5.4g
       512m          3      67800      1.72g      7.13g
         1g          0      67800          0      7.13g
0

Here's the code, if you'd like to compare line count, instead of raw byte size

find . -type f -print0 | xargs -0 wc -l | awk '{size[int(log($1)/log(2))]++}END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | sort -n
0

I had to do something similar but on some petabyte of data, and since running find would takes ages (a week in my estimation) I tried to run find in parallel on subdirectories, of course this might be useless in some cases, but worked for me, so here is the script I used for the task: file size distribution stats

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.