How do I find out which file type is taking up the most space?

Question

I have a zillion little files in a complex folder hierarchy; these are taking up too much space (ie. terabytes). I'd like to find out which file type (ie., .pdf etc) is taking up all the room so that I can consider deleting all of the files of that type.

Is there a bash command/script that would solve this for me easily? A python script would also be workable, but I have no gui.

Thanks! Alex

This seems relevant: https://stackoverflow.com/questions/1358920/bash-measure-disk-space-of-certain-file-types-in-aggregate

Answer 1

Quicktut:

Listing all files in a directory:

import os
path="C:\\somedirectory"  # insert the path to the directory of interest
dirList=os.listdir(path)
for fname in dirList:
    print fname

You have to make the above recursive in order to get it working on subfolders also.

Getting filesize of a file:

import os
b= os.path.getsize("/path/isa_005.mp3")

Get file extension:

import os
ext = os.path.splitext(file_name)[1]

Now, all you have to do is create a dictionary mapping extensions to accumulated file size and print out the dictionary (maybe after sorting or something).

Hope it helps.

Answer 2

I just wrote this in about 5 minutes. Instead of os.path.getsize I use os.stat's st_size. I don't think it really matters. I use os.walk to recursively "walk" through all the directories in the current working director '.' This wasn't written for efficiency or performance in mind, just to get something going. The end result is a dictionary filled with file extensions for keys and each value is converted into a string representing a human readable format of the total size for each file type. I took a method written by someone else to do the human formatting. The last part is some fancy smancy to sort the file types by size. If you hit ctrl+c it kills the "sizing-up" and just prints the results it had time to gather. Pretty fun! Thanks for the ride, enjoy.

import os

#using code ripped from:
#http://www.5dollarwhitebox.org/drupal/node/84
#to convert to human readable format
def convert_bytes(bytes):
   bytes = float(bytes)
   if bytes >= 1099511627776:
      terabytes = bytes / 1099511627776
      size = '%.2fT' % terabytes
   elif bytes >= 1073741824:
      gigabytes = bytes / 1073741824
      size = '%.2fG' % gigabytes
   elif bytes >= 1048576:
      megabytes = bytes / 1048576
      size = '%.2fM' % megabytes
   elif bytes >= 1024:
      kilobytes = bytes / 1024
      size = '%.2fK' % kilobytes
   else:
      size = '%.2fb' % bytes
   return size

typesizeH = {}
typesize = {}


try:
   for root, dirs, files in os.walk('.'):
      for file in files:
         prefix, extension = os.path.splitext(file)
         if extension not in typesize:
            typesize[extension] = 0
         typesize[extension] += os.stat(root + os.sep + file).st_size
except KeyboardInterrupt:
   pass


for key in typesize:
   typesizeH[key] = convert_bytes(typesize[key])

print str(typesizeH)

types = typesize.keys()
types.sort(cmp=lambda a,b: cmp(typesize[a], typesize[b]), reverse=True)
print "Filetype\tSize"
for type in types:
   print "%s\t%s" % (type, typesizeH[type])

Result:

Filetype        Size
.7z     99.84M
.hpp    42.88M
.lib    39.40M
.ncb    28.50M
.dll    27.87M
.exe    25.26M
.h      10.33M
.obj    10.18M
.zip    6.83M
.svn-base       3.92M
        3.52M
.txt    2.28M
.csv    1.09M

Answer 3

A shell/gawk solution:

find: Print filename and size for all files

gawk: Create a Assoc Array using type as key and summed up size as value. At the end of input: print size for all types.

find DIR -name "*.*" -type f -printf "%f %s\n" | awk '{sub(/.*\./,"",$1);count[$1]+=$2} END{for (var in count) print var, count[var];}'