Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I've got a filesystem which has a couple million files and I'd like to see a distribution of file sizes recursively in a particular directory. I feel like this is totally doable with some bash/awk fu, but could use a hand. Basically I'd like something like the following:

1KB: 4123
2KB: 1920
4KB: 112
...
4MB: 238
8MB: 328
16MB: 29138
Count: 320403345

I feel like this shouldn't be too bad given a loop and some conditional log2 filesize foo, but I can't quite seem to get there.

Related Question: How can I find files that are bigger/smaller than x bytes?.

share|improve this question

2 Answers 2

up vote 3 down vote accepted

This seems to work pretty well:

find . -type f -print0 | xargs -0 ls -l | awk '{size[int(log($5)/log(2))]++}END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | sort -n

Its output looks like this:

         0   1
         8   3
        16   2
        32   2
        64   6
       128   9
       256   9
       512   6
      1024   8
      2048   7
      4096  38
      8192  16
     16384  12
     32768   7
     65536   3
    131072   3
    262144   3
    524288   6
   2097152   2
   4194304   1
  33554432   1
 134217728   4
where the number on the left is the lower limit of a range from that value to twice that value and the number on the right is the number of files in that range.

share|improve this answer
    
I edited your answer to use find instead of ls so that it was recursive and didn't do any directory counting. Anyone want to take a crack at prettying up the left hand column output? –  notpeter Mar 13 '13 at 20:49
    
But the original question was about "distribution of file sizes in a particular directory", so it's not OK to change the ls to a find. I'm putting it back the way it was. –  garyjohn Mar 13 '13 at 21:15
    
@notpeter: Sorry, I didn't recognize you as the author of the question. I changed my answer to make it search recursively. On my system, though, using xargs is significantly faster than -exec, so I used that method. –  garyjohn Mar 13 '13 at 23:01
    
No worries. Now we can just delete our comments are pretended it was always the right answer. ;) –  notpeter Mar 14 '13 at 16:29

Try this:

find . -type f -exec ls -lh {} \; | 
 gawk '{match($5,/([0-9.]+)([A-Z]+)/,k); if(!k[2]){print "1K"} \
        else{printf "%.0f%s\n",k[1],k[2]}}' | 
sort | uniq -c | sort -hk 2 

OUTPUT :

 38 1K
 14 2K
  1 30K
  2 62K
  12 2M
  2 3M
  1 31M
  1 46M
  1 56M
  1 75M
  1 143M
  1 191M
  1 246M
  1 7G

EXPLANATION :

  • find . -type f -exec ls -lh {} \; : simple enough, find files in the current dir and run ls -lh on them

  • match($5,/([0-9.]+)([A-Z]+)/,k); : this will extract the file size, and save each match into the array k.

  • if(!k[2]){print "1K"} : if k[2] is undefined the file size is <1K. Since I am imagining you don't care about such tiny sizes, the script will print 1K for all files whose size is <=1K.

  • else{printf "%.0f%s\n",k[1],k[2]} : if the file is larger than 1K, round the file size to the closest integer and print along with its modifier (K,M, or G).

  • sort | uniq -c : count the occurrences of each line (file size) printed.

  • sort -hk 2 : sort according to the second field in human readable format. This way, 7G is sorted after 8M.

share|improve this answer
    
I appreciate the explanations, I think it's helpful for people trying to figure it out. That said, your script doesn't work for me for two reasons 1) My GNU LS is old and so gives different human readable size output for 'ls -lh' (bytes not K/M/G/T) and 2) because there's too many buckets. With file sizes between 1K and 1G there are 2000 buckets, half of which are 1KB half of which are 1MB. Worth it though for 'uniq -c' that's new to me. –  notpeter Mar 13 '13 at 21:00

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.