25

Referencing this page: https://support.microsoft.com/en-us/kb/140365

NTFS doesn't go over 4KB cluster size until the volume goes over 16TB, and FAT32 maxes out at 16KB in the 16GB–32GB volume range on modern versions of Windows.

exFAT however only defaults to 4KB in the 7MB–256MB volume range. After that it jumps to 32KB in the 256MB–32GB range, and 128KB beyond that.

Why is that? A relatively high cluster size seems wasteful, especially in a format designed for smaller external devices, like flash drives. Doubly so if you're formatting flash memory, because as I understand it, the main benefit to larger cluster size is faster IO due to less fragmentation and less overall clusters to read. Correct me if I'm wrong, but flash memory is much, much less prone to slowdowns caused by fragmentation. So why make the cluster size so high?

4
  • 1
    "Size" and "high" do not go together. Numbers can be "high". "Large" is the bettor word instead of "high".
    – sawdust
    Commented Sep 13, 2018 at 21:42
  • Possible duplicate of Downsides of a small allocation unit size
    – sawdust
    Commented Sep 13, 2018 at 21:44
  • 1
    just wanted to mention that I just formatted a 1TB drive using default cluster size, and windows used a cluster size of 2048KB!!! This is not only defying the specs on their page, this is always WAYY too high
    – woojoo666
    Commented Sep 16, 2018 at 9:23
  • 1
    For anyone who reached this by Googling & might ever back up a git directory to the drive/card in question, it's important to know downside of large sectors, too: each ~200 byte file in a git directory takes up 1 full allocation unit. Large sectors can make dev dirs (which can have 10k+ tiny files) take up a large multiple of their normal HDD sizes. Ex: superuser.com/questions/704218/… superuser.com/questions/1370780/…
    – kcrumley
    Commented Jul 8, 2019 at 18:09

2 Answers 2

14

Because exFAT is used primarily for things like large capacity SD cards, and on SD cards you must erase a sector before you can write to it. If you were to use small clusters on a card with a large erase sector size it would result in many erase and write commands to the same sector write a several sequential clusters worth of data to disk not only reducing performance but also prematurely wearing out the flash cells. The erase sector size is generally not documented in card datasheets but can be found in the CSD register of the card. The contents of this register will vary from card to card depending on its internal design. Since SD cards are generally used in things like cameras which store large files, the wasted space of a large cluster size isnt important, and only would be if you were storing very large numbers of very small files which generally does not happen.

This webpage has the contents of this register for about a dozen cards:

http://goughlui.com/2014/01/03/project-read-collect-decode-sd-card-csd-register-data/

And if you enter its contents in to the following calculator you can see for a few of the 32/64GB cards the erase sector size is 128 blocks with a block being 512 bytes. And for a 2GB card the erase sector size is 32 blocks with a block being 1024 bytes.

http://goughlui.com/static/csdecode2.htm

Wether or not Windows is smart enough to query the CSD register and suggest a cluster size, or if it simply guesses based on the partition or disk size is unknown. If you were to emulate a SD card with a microcontroller, you could find out.

4
  • 1
    i did not know about needing to erase flash memory before you could write to it. That makes sense now. Thanks! Commented Dec 2, 2016 at 21:36
  • 2
    You seem to be assuming that SD cards have no flash translation layer, i.e. that adjacent logical sectors will always be physically adjacent in NAND. Otherwise the logical sectors in a cluster would not have to belong to the same eraseblock. What evidence do you have to support this assumption?
    – sawdust
    Commented Sep 13, 2018 at 21:33
  • Indeed some high performance SD cards and flash drives do have a translation layer and yet still come pre-formatted with 32kb clusters?? I have verified that 4kb clusters improves performance for random writes and reads Commented Mar 17, 2020 at 0:25
  • "Since SD cards are generally used in things like cameras... the wasted space of a large cluster size isn't important" exFAT is used on more than just SD cards (even in 2016), and outside the example quoted, such a large cluster size decreases the total amount of data that can be stored since so much space is wasted by a large sector size (32KB has 8x less effective storage, 128KB has 32x less ). Due to this, even with the finite lifespan of flash storage, the cons of large cluster sizes outweigh the pros in all but a handful of specific use cases, such as large RAW files.
    – JW0914
    Commented Nov 1, 2020 at 13:47
7

The earlier answer to this question is wrong and shouldn't have been upvoted or accepted. exFAT isn't a flash file system. It's never used directly on "raw" flash. That would be horrendously inefficient, even if the cluster size matched the flash erase block size.

What actually happens is that exFAT runs on top of a block-device emulation layer, which is typically implemented in firmware on the flash device itself. Suppose the exFAT cluster size is 128K, the (fake) block device's sector size is 4K, and you write a 5K file to the exFAT volume. The exFAT driver allocates 128K for the file, but (ignoring metadata) it only writes 8K: the two sectors that actually contain file data. That's the same amount of data that would have been written if the cluster size was 4K. The difference is that if you write a bunch of 5K files, they will be separated by 30 unused fake sectors that are never read or written to. But that doesn't matter, because the real location where the data goes on the raw flash is unrelated to the sector numbers of the virtual block device. The exFAT cluster size probably has little effect on the amount of data written to the raw flash, or the number of block erase operations. When you include exFAT metadata overhead, larger clusters should be somewhat more efficient – but that has nothing to do with the size of the erase blocks of the raw flash, which is completely hidden by the emulation layer.


Of course, large clusters mean you run out of space on the fake block device if you try to store a lot of small files, and there's no way to coerce the emulation layer into letting you use the rest of the real space (short of using those sector numbers you skipped), so large clusters still seem like a bad idea. I don't know who at Microsoft decided that the default exFAT cluster size should be so large, or why, but I'll speculate.

exFAT has a much larger per-cluster overhead than NTFS: 65 times larger, in fact, because NTFS just has one bit per cluster for the free space bitmap, while exFAT has one bit in the bitmap plus 4 bytes in each of two FATs. On the other hand, NTFS has some significant other fixed overhead (notably, several megabytes for the journal), which is not directly tied to the number of clusters.

My guess is that they wanted exFAT to appear to have less overhead than NTFS. On small volumes, NTFS always consumes at least a couple of megabytes, so the 0.2% overhead of 4K clusters didn't seem so bad. But a freshly formatted 1TB volume would have about 2GB of exFAT metadata with 4K clusters, while NTFS would only need about 40MB. That would look bad (and would be bad, I suppose), so they had to use a much larger cluster size.

2
  • 2
    exFAT is obviously not a flash file system but it tries to optimize a few things to be used on flash memory while still extending and maintaining many old FAT structures
    – phuclv
    Commented Jul 20, 2021 at 5:26
  • It seems that at least in the GUI 'format" command, if you try to format a large drive (4 TB) with exFAT the minimum allowed size is 256k (256,000 bytes). On an archive volume I'm trying to work with that has an average file size of 100k (100,000 bytes) this will kill me (and the default of 1024k almost killed me allocating 10x the actual file size in practical use). Honestly 100k files are not really "small"...
    – ebyrob
    Commented Oct 9, 2023 at 13:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .