4

I have millions of files on a large disk. I want to minimize the average wasted space by correctly formatting the drive.

Is there any way to optimize the BlockSize, Sectorsize, and cluster size for a pre-existing disk that has many files?

I'm assuming I need to analyze the data I have, format a new disk to spec, and copy the folders over.

I'm also assuming that if there are some large files, this will have an impact on the $NTFS internal tracking metadata, increasing their size (as large files consume many segments)

  • many file systems have a feature called tail packing to minimize wasted space. However NTFS doesn't support that. You might look into packing files into a single file like how tar and 7z do if your files aren't accessed frequently – phuclv Sep 16 '18 at 10:13
2

You can't change the block size; it's whatever the drive presents (512 bytes unless it's a 4K native drive, in which case it's 4096 bytes).

Sectors are the same thing as blocks, just a different name.

Clusters are the file allocation granularity. The default for NTFS is 4096 bytes. Tiny files will disappear into the NTFS file record (always 1024 bytes) so for them, it doesn't matter what the cluster size is.

Large files do not necessarily consume many extents (not if they're contiguous), nor does the cluster size much affect this.

You cannot change the cluster size on an already-formatted volume. You'll need to create a new volume with the parameters you want and copy the data to that volume.

Let's say you have 3 million files and due to the default cluster size each wastes an average of 2048 bytes. Then your total wasted space is 6 GB. This is 0.6 percent of a 1 TB disk, a smaller percentage of a larger disk. Suppose it's a 2 TB disk that cost you $100; the total cost of the wasted space in the last allocation clusters of all of the files is then 30 cents. Less if a lot of them are stored completely in the file records.

Most of us stopped worrying about it long ago and just let NTFS use its default. Besides, having the cluster size match the processor's page size is convenient for Windows' Memory Management code.

  • Thank you. I'm seeing IO slowness when 1,000,000 + files are stored at the same folder level. I suspect the NTFS internal management is to blame, IO perf counters are within reason. that is why I'm thinking cluster size... – halfbit Dec 21 '14 at 0:39
  • Related: superuser.com/q/855088/47507 – halfbit Dec 21 '14 at 0:39
  • 1
    There is nothing you can tweak in NTFS to make it perform well with on the order of a million files in one directory. It simply wasn't designed for such a case. The wasted space due to the file allocation cluster size is not a factor here. (N.b.: Folders are an artifact of Explorer and other things that use the "shell" interfaces. In NTFS there are directories.) – Jamie Hanrahan Dec 21 '14 at 1:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.