I'm formatting a 1TB external hard drive as a single NTFS partition. This drive is mainly meant for storing media.
Should I choose something other than the default allocation unit size? The options range from 512 bytes to 64K. Are there any guidelines that I should use with partitions storing other data? Should I stop poking around and just leave it at "default"?
If you are a "Standard User" by Microsoft's definition, you should keep the default 4096 bytes. Basically, the allocation unit size is the block size on your hard drive when it formats NTFS. If you have lots of small files, then it's a good idea to keep the allocation size small so your hard drive space won't be wasted. If you have lots of large files, keeping it higher will increase the system performance by having fewer blocks to seek.
But again, nowadays hard drive capacity is getting higher and higher making small differences by choosing the right allocation size. I suggest you just keep the default.
Also keep in mind that the majority of files are relatively small, larger files are large in size but small in units.
In terms of space efficiency, smaller allocation unit sizes perform better. The average space wasted per file will be half the chosen AUS. So 4K wastes 2K per file and 64K wastes 32K. However, as Jonathon points out, modern drives are massive and a little wasted space is not worth fussing over and this shouldn't be a determining factor (unless you are on a small SSD).
Compare 4K vs 64K average case waste (32K-2K = 30K), for 10,000 files that only comes out to 300,000KB or around 300MB.
Instead think about how the OS uses space. Let's say you have a 3K file which needs to grow 2K. With a 4K AUS the data needs to be split over two blocks - and they may not be together so you get fragmentation. With a 64K AUS there are a lot fewer blocks to keep track of and less fragmentation. 16x the block size means 1/16th the number of blocks to keep track of.
For a media disk where your photos, music and videos are stored (files typically 1MB+) I use the biggest AUS. For a windows boot partition I use the Windows default (which is 4K for any NTFS drive smaller than 16TB).
To find out what the cluster size is on an existing disk:
fsutil fsinfo ntfsinfo X:
I am setting up four Seagate ST3000DM001 3TB drives in a RAID 10 configuration. After reading up on how to format it for the Allocation Unit Size, I found this article: General Guidelines for Improving Operating System Performance.
When configuring drive arrays and logical drives within your hardware drive controller, ensure you match the controller stripe size with the allocation unit size that the volumes will be formatted with. This will ensure disk read and write performance is optimal and offer better overall server performance.
I am using the ASUS P8Z77-V DELUXE motherboard's built-in RAID capability from Intel chipset. Since the Default Stripe size for RAID 10 on the motherboard is 64KB, I am formatting my RAID array using the "64 kilobytes" option in Windows.
Anyway, I hope this information I found helps for other people researching for this topic and add more information to it.
Just be aware the larger cluster size you use, the better performance you will have. But the efficiency decreases.
If you have a 3TB disk and store thousands of pictures, you would want a smaller cluster size for space efficiency, but if you make it too small you will take a severe performance hit.
For any disk 500gb or bigger, I recommend 8k cluster size or larger.
It can make a difference.
A file can only be a multiple of whatever you select, so a finer grained format will mean you will waste less space on files - but make your allocation table bigger.
If you're going to have thousands of files under 64k it's worth considering a lower size unit. However, if you're going to fill the drive with huge, hundred megabyte videos, then a higher unit size is more applicable.
If you don't know, just go with the default.
You point out disk usage as video and music. Majority data file sizes then should be 2MB+ for music and 100MB+ for video, some playlist (less then 5% of overall file quantity) of few kB. You point out the device is external USB disk.
The best solution for such specification is exfat(vfat on linux) or NTFS with 64kB block/allocation unit size. Some hints on what file system you should chose: - xxfat should be used for faster writes and compatibility (ie. you plan/ already use media streaming device for your home/office usage) - NTFS for security and advanced features. (security is still source of problems for home users - be sure you need it)
Have a fun.
For spindle based storage, unless you need compression I would be inclined to bump the cluster size to 64k.
The benefits of larger cluster size are less fragmentation over time, less metadata and lower i/o's for the system to manage.
You do get more wastage but now days it would be a very unusual workload to have the majority of your files small enough for this to matter and is offset by the metadata savings.
I would keep the boot drive on 4k, and SSD's probably should be kept on 4k as well.
I have been tinkering with allocation sizes for years now.
I currently have a parity pool configuration of 3x4TB and another 3TB HDD. The OS is on a separate 512GB SSD. All the HDDs are formatted with 1MB allocation size with NTFS.
The biggest difference is write performance which is consistent and higher. There are no notorious freezes while writing or deleting files. Yes, you never get to see that odd freeze while deleting files that windows explorer always gets stuck into while deleting many small files.
I frequently change contents of my drives and still fragmentation does not pose an issue with standard Windows defragmentation.
1 MegaByte is the best for NTFS, for 6000 files you will loose 6000 MBs. On today's 1TB disks, it's not a problem, even for a 512 GB disk it won't be a problem. Considering that not the whole megabyte is lost, but let's say half, 3GB are lost. However the maximum possible is 64kb for NTFS, use it, it's not a problem for the disk or OS.
