Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Since I work in web development and bandwidth is always an issue, I make every effort to reduce the size of my files, especially images. As a result I have at least 10,000 files that are all around 200 bytes in size.

My hard drive's allocation unit size is 4 kB, which means that I am wasting a lot of space. When it comes to backing up in particular, I'd like to avoid wasting space.

What I'd like to know is what the downsides are to setting the smallest AU size. My main concern is what happens to 8 GB files if I have a 512-byte AU size?

share|improve this question
    
IIRC, sustained reading speed is reduced, which is why if you were mainly dealing with very large files (eg. video editors) it would desirable to increase it. –  paradroid Aug 23 '12 at 22:38
    
Oh, and also, by changing from the defaults, you are a bit more susceptible to incompatibility problems with certain software that directly accesses the hard drive, like disk cloning and backup software. –  paradroid Aug 23 '12 at 22:41
add comment

1 Answer

up vote 5 down vote accepted

The downsides of a small disk allocation unit include:

  • Larger allocation table.

    This is the most obvious consequence of reducing the allocation unit. For a given sized volume, reducing the allocation unit from 4KB to 512 results in an allocation table 8 times larger. Note that the filesystem will likely have duplicate or triplicate copies of the allocation table for ensuring filesystem integrity.

  • Allocation occurs more often.

    Since a smaller amount of disk space is allocated per unit, more filesystem overhead will be incurred when writing sequential files (the most common method). In order to allocate a cluster, the filesystem must acquire a mutex lock to ensure exclusive access, modify the allocation table, release the mutex, and then copy out the allocation table from memory to the disk.

  • Possible limit in volume size

    Probably not an issue today with 32- and 64-bit processors. But back in the days of 8- and 16-bit processors and filesystems such as FAT, the number of allocation units combined with the allocation size actually imposed a real limit on the size of hard disk volumes/partitions. One obvious result was the FAT32 filesystem that increased the number of possible allocation units.

  • More file fragmentation

    For a given file size, there will be obviously more allocation units assigned to that file. There are no guarantees that allocation units can/will be contiguous. For reading an 8MB file, the worst case scenario for 4KB clusters would involve 2048 seek plus rotational latency intervals, or one complete disk access for each cluster in sequence. The worst case scenario for 512B clusters would involve 16,364 seek plus rotational latency intervals! Obviously this (possible) fragmentation will impact data throughput.

As disk drives get larger, the allocation size is often increased to mitigate these downsides. The rational is that there is more disk space available to waste, but that is circular logic. Ideally the disk drive should have several partitions, with each partition formatted with an allocation sized for the "typical" file. For instance I leave the C: drive/partition with its default 4KB size. But the partition where TV recordings are written have 64KB clusters since the typical one-hour recording is about 6GB.

share|improve this answer
    
I like that logic, but it might be a bit difficult to implement with my current backup ("scan all files, check modification dates, copy as needed"). However I could improve my backup program by having a "small file" partition, a "big file" partition, and a lookup file to see if a given file is in the small or big partition. –  Niet the Dark Absol Aug 24 '12 at 1:02
    
If I remember correctly, ZFS is able to use different allocation sizes for separate files... other file systems are also capable of packing multiple small files into a large allocation unit. –  grawity Aug 24 '12 at 16:22
    
Hello, I found this Q by own interest. By larger allocation table, what do you mean? What's the downside of a larger allocation table? –  Karolinger Nov 5 '12 at 17:23
1  
@Karolinger - The filesystem has to maintain the status of each allocation unit, whether it is free or in use or marked bad or relocated. The size of this allocation table is directly proportional to the number of allocation units. For a disk partition of a given size, if you halve the allocation size, then the number of entries in the allocation table is doubled. The larger allocation table could potentially require longer search times to find a free unit, since there are simply more entries to scan. Some of these concerns are more pressing for small capacity media, e.g. floppies. –  sawdust Nov 5 '12 at 21:41
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.