We are redesigning out current ZFS file server. It currently only has one pool of over 70TB.

I see this is a problem since, if there are enough disk failures within the pool to make it non-functional, we have to restore all 60TB of currently existing data.

A debate has risen as to how to divide up the storage into smaller pools, and how big/small each of those pools should be.

As I see it, the only drawback of using pools that are "too small" is a lot of wasted disk space. We are using 4TB 7500 spinning disks with 4TB SSD cache disks. So, I'm leaning towards 20TB raidz vdisks. That means we get about 77% of the disk space usable.

Q: Are there any other advantages/disadvantages to making the pool sizes too large/too small?

  • 1
    I would ask this on the FreeNAS forum, as it's where you'll get the most technical advice regarding ZFS. Numerous iXsystems' devs routinely reply (usually multiple times daily - iXsystems is the company that maintains FreeNAS) and you'll get the best help and advice form them and other forum users who are highly knowledgeable about ZFS – JW0914 Oct 1 '19 at 11:55
  • 2
    Large pools don’t imply large vdevs. In fact, you should not have them. Large vdevs will cause super long resilver times. – Daniel B Oct 2 '19 at 14:01

Q: Are there any other advantages/disadvantages to making the pool sizes too large/too small?

One distinct and significant advantage of a single large ZFS pool is that space allocation is much more flexible in a single large pool.

If, for example, you have three separate sets of data and 60 TB of disk space (ignoring RAID losses), if you create one ZFS pool, the total data stored in all three sets of data is limited to 60 TB and can be split in any manner between the three sets of data. (I'm assuming you'd create separate ZFS datasets in the one large ZFS pool - one for each set of data.)

But if you split the storage into three 20 TB pools and put one set of data in each pool, the total is still limited to 60 TB, but you've also limited each set of data to 20 TB.

That's a huge loss in flexibility. If you don't properly predict data growth, you'll have wasted disks in the pools that didn't grow as much as you thought they would and have to buy extra disks for the pool that did grow. Or remove disks from one pool and move then to another via a likely-to-be-nasty backup, destroy, restore.

Second, are you sure your split pool is actually more reliable than a single large pool? Another advantage of a large pool: all hot spare disks are available to replace any failed disk in the pool (assuming all disks are identical). If you use multiple pools, hot spares in pool A can't help the zdevs in pool B survive a disk failure. Three pools each with two hot spares may very well be less reliable than a single pool that has access to six hot spares.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.