Please explain wasted space on an exFAT formatted external hard drive [duplicate]

Question

I recently bought a 750 GB 2.5" hard drive and put it in an external enclosure. It has Advanced Format Technology (AFT), but I use Windows 7, so I assumed I would be safe.

1. I decided to try exFAT, because I was enticed by the reduced overhead
2. For "Allocation unit size" I selected "Default allocation size"
3. Tried with "Quick Format" un-selected first, but it was taking too long, so I stopped and started over with "Quick Format" selected
4. After drive was formatted, looking at the format options again shows that the "Allocation unit size" is 256 KB

The drive works fine, but I had a question about the difference between "Size" and "Size on disk."

On my 120 GB NTFS formatted internal hard drive, the difference is only 0.16%. However, on my much larger, exFAT formatted external drive, the difference is 17.38%. At that rate, the most I will fit on this 750 GB drive is 637 GB (with 0 free space).

What format options should I use to maximize usable disk space?

What impact will my current settings have on performance? What impact will your recommended settings have on performance?

Format settings

Internal hard drive

External hard drive (with big space difference!)

Answer 1

The default allocation unit (cluster) size in exFAT is much larger than the default allocation unit size in NTFS. As you've noted in your question, exFAT defaults to 256KB, but NTFS defaults to 4KB for volumes larger than 2 GB.

Because allocation of storage must occur in whole clusters, a file that is not a multiple of the cluster size will result in wasted space on the storage device. This wasted space depends on the size and is generally larger when the cluster size is larger and when there are many files on the device. This condition is called internal fragmentation, and the wasted space is called slack space.

For example, a 257KB file will require two allocation units to store on your exFAT volume, taking up 512 KB of space. The same file would require only 260KB on an NTFS volume with 4KB clusters.

The Wikipedia article on data clusters explains it in more detail:

A cluster is the smallest logical amount of disk space that can be allocated to hold a file. Storing small files on a filesystem with large clusters will therefore waste disk space; such wasted disk space is called slack space. For cluster sizes which are small versus the average file size, the wasted space per file will be statistically about half of the cluster size; for large cluster sizes, the wasted space will become greater. However, a larger cluster size reduces bookkeeping overhead and fragmentation, which may improve reading and writing speed overall. Typical cluster sizes range from 1 sector (512 B) to 128 sectors (64 KiB).

Try formatting the drive with a smaller allocation unit size. This may slightly slow I/O operations, but will greatly reduce the amount of wasted disk space.

Answer 2

The size of the allocation unit will impact the usable space on the disk. A larger allocation unit will be better if you have large files. A smaller allocation unit is better for small files.

A way to think about it is to think of your hard drive like a blank notebook, and each page is an "allocation unit" with a set size it can hold. A small notebook page (i.e. small allocation unit) is very convenient if you have a lot of small bits of information to save. The same area of paper can be cut into many more pages and lots of small notes can be stored. If a note happens to be larger than a single page, the last line of every page is reserved to point to the next page to continue the note. If your note is only a single page, the last line is never used, but it is always reserved just in case that note gets larger.

If you use a large notebook page size, each page stores a lot more information without needing to flow onto another page. If it does, it still only uses the last line of the page to point to the next page of the note. The drawback is that if you have a lot of files that only use a small part of the page, still take up a full page in the notebook.

If you want to get the most usable space on this drive, I'd suggest using the largest allocation unit that makes sense for your data.

Answer 3

The link DragonLord provided above is not good in any way. It talked about FAT12/16 which is much inferior to exFAT. 16 bit is not enough for addressing clusters so FAT16 volume size tops out at 4GB, 4-16GB is only supported with sectors larger than 512 bytes. Current implementation of exFAT uses 25 bit for cluster indexing so it's most comparable to FAT32. But exFAT uses 64 bit for file size so it supports files larger than 4GB.

The default cluster size for exFAT is described here

In general cluster size above 8KB is not recommended. And use exFAT for HDD is not recommended too, except for exchanging files with other systems which can't understand NTFS, because exFAT is specifically designed for flash memories which have limited write cycle. exFAT doesn't have journaling to reduce flash wear, which trades for some less reliability. The appropriate cluster size for most case is 4KB (which is one of the reasons that NTFS's default cluster size is almost 4KB), except for partitions that most stores very large files. So if the external hard drive is not for plugging into some old HTPCs, media players or devices which can't read NTFS, you should use NTFS