Downsides of 8Kb cluster size for NTFS on top of ZVOL?

Question

I am trying to maximize performance of Windows VM on a ZVOL. The official recommendation is that allocation unit size (volblocksize) on the zvol should be aligned with the allocation unit size of what I want to put on the ZVOL. In the case of Windows, that would be a 4Kb volblocksize: http://open-zfs.org/wiki/Performance_tuning#Dataset_recordsize

However, it is also said that small volblocksize harms ZFS performance, and in this thread people give mixed recommendations: http://list.zfsonlinux.org/pipermail/zfs-discuss/2017-September/029227.html

I am wondering if it might be best to leave ZFS at the default 8Kb blocksize, and then format my Windows installation with 8Kb blocks instead of the usual 4kb. I realize this means no NTFS file compression, and I am willing to at least try forgoing it.

My thought was that if the ZVOL is going to address 8kb blocks, there won't be a net storage space penalty for setting the NTFS to 8kb. Is this correct? Are there other reasons why this might be a terrible idea?

Answer 1

FWIW, it's pretty much impossible to trust anecdotal performance numbers: too much can be different between that individual data point and your setup. (Were they using dedup? Did they have enough RAM in the system? Was one of their disks on the verge of failing? What data redundancy configuration were they using? Was this before or after sync writes got 2x faster? The list goes on and on.)

The guidelines you linked to here are correct:

Partial record writes require that data be read from either ARC (cheap) or disk (expensive). recordsize can be set to any power of 2 from 512 bytes to 128 kilobytes. Software that writes in fixed record sizes (e.g. databases) will benefit from the use of a matching recordsize.

And just below...

Zvols have a volblocksize property that is analogous to record size. The default size is 8KB, which is the size of a page on the SPARC architecture. Workloads that use smaller sized IOs (such as swap on x86 which use 4096-byte pages) will benefit from a smaller volblocksize.

However your interpretation of that is slightly off -- it's saying that you should try to match the size of the IO your database will issue (up the IO stack), not the block size of the underlying hardware (down the IO stack).

That advice is correct (and matches your desire to use 8KiB volblocksize if I understand right) assuming you want to do compression inside ZFS (which is basically the only option if the guest is Linux since ext4 and XFS don't do compression). There will not be any storage overhead for this configuration, either due to read-modify-writes or due to extra space being consumed. If you believe the mailing list message you linked to, maybe this would also marginally reduce whatever bottleneck ZFS hits when writing a lot of 4KiB blocks.

However, since NTFS can do compression, if you want to use NTFS's compression instead for some reason, you should make volblocksize equal to the underlying disks' block size, to give NTFS the smallest possible unit to compress stuff into (wasting the least amount of space when it has to round up the compressed data to the next block size), and make the logical block size in NTFS match the database's desired block size. This will also not create any storage overhead through read-modify-write or extra space.