5

All the resources say ZFS doesn't have fsck, or recovery tools, use battery backed SSD for ZIL, etc.

If the plug is suddenly somehow pulled (total power loss despite UPS etc, but assuming no physical damage, no head crashes etc), the SSDs will write cache to nvram and then go quiet....

What chance does ZFS have of being in a consistent state (even if some data was lost) and the pool being usable/readable, when it reboots?

update

I realise I actually mean to ask something closer to, what events would lead to a situation where ZFS gives up on being able to read the pool, despite the data basically being intact? Its not clear what ZFS can recover from (or can recover given the right hardware) and what it can't (or can't without the right hardware), because it does so much internally to self check and fix things. Clearly insufficient redundancy+ disk failure (or other major hardware issue) is one case, and complete wipe/overwrite due to firmware/software bug is another. But assuming the storage media, hardware and software are still working reliably/properly, what else has to have gone wrong, for the result to be loss of a pool? Where are its limits on pool fixing? Which situations have to arise before it can't, and what has to happen to give rise to them?

  • The suggestions to do that are mainly because of those few transactions that would be lost in the crash - the data at rest is always save (except bugs or complete hardware failure). In the special case of SSDs it could also be that data is lost inside the SSD, because the controller silently loses it on power loss, but has already signaled successful write. Then ZFS cannot do anything, and if you don't have sufficient redundancy, your pool may become corrupt. – user121391 Oct 14 '16 at 9:29
  • Could you give examples of what you mean? Most corrupt pools come from either bugs in ZFS (like illumos.org/issues/6214), hardware failure (all redundant copies are damaged or root node metadata is damaged or device lies about data safety) or user error/misconfiguration (accidental zpool destroy, striped pool without redundancy). – user121391 Oct 14 '16 at 14:20
  • Yes. I wander up to my ZFS system and without warning trip up and accidentally yank out the P3500 ZIL midway through a very heavy incoming data session, and the system immediately freezes. Thanks to a good PSU and MB, the other HDD/SSDs aren't affected by the electrical transients. Every other disk/vol was redundant except the ZIL. Have I just lost some recent data, the whole pool, or "it depends", and if it depends, then on what? )OK not the most likely incident but that's the point - at some point I have to choose what to design against, when I spread my money across the hardware spec. – Stilez Oct 14 '16 at 18:26
  • @Stilez: You will lose the uncommitted data in the ZIL, but it's no worse than pulling the machine's power cord. ZFS has had a graceful way to remove the ZIL since pool version 19, realeased in 2009. – Warren Young Oct 14 '16 at 21:30
  • Thanks. That's not quite the same as coping reliably with ungraceful removal though. To bring it to a more realistic level, if one doesn't spec the ZIL with a mirror + supercap and it fails (and power simultaneously fails, which isn't an implausible coincidence if the two have a common cause), is the user imperilling their entire pool, or just risking a limited amount of data in flight? This will affect the decision to get double, or premium, SSDs, in cases where loss of a small amount of data in flight can be accepted because its rare, but loss of pool is much more serious and cannot. – Stilez Oct 15 '16 at 7:05
4

What chance does ZFS have of being in a consistent state (even if some data was lost) and the pool being usable/readable, when it reboots?

ZFS operates like a transactional database management system in that old data is not overwritten in place when being updated, as with traditional filesystems. Instead, the new data is written elsewhere on the disk, then the filesystem metadata structures are updated to point to the new data, and only then is the old data's block freed for reuse by the filesystem. In this way, a sudden power loss will leave the old copy of the data in place if the new data updates are not 100% committed to persistent storage. You won't have half the block replaced or anything like that, causing data corruption.

On top of that, ZFS uses a sophisticated checksumming scheme that allows the filesystem to detect miswritten or corrupted data.

If you're using ZFS with redundant storage, this same scheme allows the filesystem to choose between two or more redundant copies of the data when repairing the filesystem. That is, if you have two copies of a given block and only one of them matches its stored checksum, the filesystem knows that it should repair the bad copy/copies with the clean one.

These repairs may happen on the fly, when you try to read or modify the data — whereupon the filesystem may realize that the requested blocks aren't entirely kosher — or during a zfs scrub operation. It is common to schedule a scrub to run periodically on ZFS pools that have files that are rarely accessed, since the filesystem wouldn't otherwise discover hardware data loss in the normal course of operation. It is common for ZFS pools running on dodgy hardware to show some number of fixed blocks after every scrub.

Scrubbing is kinda sorta like fsck for other Unix type filesystems, except that it happens online, while the filesystem is mounted and usable; it happens in the background and only when the pool is otherwise idle. Also, fsck implementations typically only check metadata, not data, but ZFS checksums both, and so can detect errors in both. If these integrity mechanisms decide that one of the blocks needs to be replaced, it can use the checksums to decide which copy to replace the corrupted copies with.

assuming the storage media, hardware and software are still working reliably/properly, what else has to have gone wrong, for the result to be loss of a pool?

As far as I'm aware, there is no such case. Either one of the three things you mention has failed or ZFS will mount the pool and read from it.

Clearly insufficient redundancy+ disk failure (or other major hardware issue) is one case

Yes, though that can happen in a subtler case than I think you're considering.

Take a simple 2-way mirror. I think you're thinking about one of the disks being physically removed from the computer, or at least inaccessible for some reason. But, imagine sector 12345 being corrupted on both disks. Then all the clever checksums and redundancy in ZFS can't help you: both copies are damaged, so the whole block containing that sector cannot be read.

But here's the clever bit: because ZFS is both a filesystem and a volume manager — as opposed to lash-up like hardware RAID + ext4 or LVM2 + ext4 — a zpool status command will tell you which file is irrecoverably damaged. If you remove that file, the pool immediately returns to an undamaged state; the problem has been removed. The lash-ups that separate the filesystem from the RAID and LVM pieces can't do that.

Which situations have to arise before it can't, and what has to happen to give rise to them?

The only case I'm aware of is something like the above example, where data corruption has damaged enough of the redundant copies of key filesystem metadata that ZFS is unable to read it.

For that reason, with today's extremely large disks — 100 trillion bits! — I recommend that you configure ZFS (or any other RAID or LVM system for that matter) with at least dual redundancy. In ZFS terms, that means raidz2, 3-way mirrors, or higher.

That said, ZFS normally stores additional copies of all filesystem metadata beyond the normal levels of redundancy used for regular file data. For example, a 2-way mirror will store 2 copies of regular user data but 4 copies of all metadata. You can dial this back for performance, but you can't turn it off entirely.


There is a chapter in the ZFS manual on ZFS failure modes which you may find enlightening.

  • I think my question was really closer to, "what are the circumstances that cause a pool to be irrecoverable" (apart from the obvious case of "not enough redundancy+too many disk fails"). What things have to go wrong with the pool, to create a situation where ZFS can't do anything to fix it? Not obvious to me as it's not clear which kinds of events ZFS can handle (or can handle with the right HW helping it) and which it can't (or can't unless it has the right HW). Title edited + question updated for clarity. – Stilez Oct 14 '16 at 10:24
  • Spot on this time. Especially the link to modes of failure (and noting the section that clarifies the different types of data corruption/events, and their effect), and the distinction /implication of being both volume manager and filing system. Thank you! – Stilez Oct 14 '16 at 18:19
  • I wouldn't call raidz2 "3-way redundancy". The common term in the ZFS community would rather seem to be "double redundancy", as opposed to "triple redundancy" (raidz3), "single redundancy" or "no redundancy", referring to how many disks in the vdev you can lose before there is no redundancy in your storage setup and consequently your data is at actual risk. A three-way mirror or raidz2 both provide double redundancy, because you can lose two disks before any further losses or problems risk causing actual data loss. – user Nov 5 '16 at 14:51
  • @MichaelKjörling: Answer edited. – Warren Young Nov 5 '16 at 23:29
  • 1
    @i486: ZFS metadata is written at least 2 times, and that few only for non-redundant ZFS pools. (i.e. single-disk or striped pools.) The default ZFS configuration multiplies that redundancy by the redundancy of the pool itself. If you're using raidz2 (3 copies of all data) you get 6 copies of all metadata. You can dial this back by configuration, if you like, but you have to choose to do it. Also realize that the metadata is itself checksummed, so if you have only two copies of the metadata and one is bad, ZFS knows which one without having to guess. – Warren Young Apr 3 '19 at 8:18
1

As my comments are getting to long, this answer seems useful. Warren Young has already correctly outlined all the basic considerations in his answer, so I'll just focus on the part "to mirror or not to mirror the SLOG device?".


The situation is as follows:

I wander up to my ZFS system and without warning trip up and accidentally yank out the P3500 ZIL midway through a very heavy incoming data session, and the system immediately freezes. Thanks to a good PSU and MB, the other HDD/SSDs aren't affected by the electrical transients. Every other disk/vol was redundant except the ZIL. Have I just lost some recent data, the whole pool, or "it depends", and if it depends, then on what? )

If you think about it, normally the ZIL is stored on all the pool disks and therefore enjoys the same redundancy the pool does. If you move it outside on a separate device for speed purposes, you need to establish another mirror yourself if you want redundancy. But even if you don't have it, you will just lose the tiny amount of data in the ZIL (restore from backup is only needed if sync writes are required and the application data is corrupted) and not make your entire pool inconsistent (which would be restore from backup in all cases).


Now, for the question as to what to choose:

at some point I have to choose what to design against, when I spread my money across the hardware spec.

It depends on your situation (as always):

  • If you have just plain data storage (classic file server), you don't get much (or anything) from moving the ZIL to an SLOG device, because SMB is async and can handle sudden power loss. I believe for NFS it depends on your choices/software, but nowadays most people use SMB on all three major systems.
  • If you need speed and integrity (mostly for databases and VM storage) you will (should) run with sync=always and you will need an SLOG device for the ZIL or it will be very very slow. In these cases you can either mirror the SLOG device or decide that the event "sudden SSD/controller hardware failure or removal AND sudden power loss" is rare enough to run it without. You can then decide if the cost is justifiable or not (in most cases it is, as the rest of the hardware is quite expensive, but still much cheaper than commercial offerings).
  • If you want peace of mind but are on a budget, I can recommend the Intel SSD 730. It is sold as a "gamer" or "enthusiast" product, but is very similar to the smaller 3700 line internally, if you compare the datasheets. It also has the super-capacitors, as several sources on the web state. Of course, officially Intel will not admit that because then nobody would buy the expensive ones.

Edit: in regards to your comment:

NFS/ESXi/sync will be a major use case. As the cost and risk are on my shoulders, I'm trying to understand the risk rather than get a recommended approach - if the separate ZIL fails as part of the power outage (whether or not it was intended to be redundant, loss protected, etc), but nothing else is affected, is possible loss/corruption limited to data received by the ZIL and not yet written to pool (last few seconds data at worst), and recoverable, or are there ways that sudden ZIL+power failure (assuming no other kind of failure at the same time) can cause the pool to be irrecoverable?

All points only valid under the assumption of your example and neither of the following is true: (a) bugs in ZFS, (b) complete hardware failure of all your pool disks, (c) human error/malice.

  • Your pool data will be safe and integrity of data at rest will be preserved, that means you can import the pool and it will not be damaged from ZFS' point of view. That is the normal behavior of a power loss in ZFS and part of its design.
  • After power is restored, normally the ZIL would be read to redo the lost transactions (similar to a RDBMS). Now the following is possible:
    • Your SLOG device is not corrupted or the corrupted parts can be restored from SLOG mirror: everything works as usual (after eventual resilver), so your last 5 seconds are written back to the pool.
    • Your SLOG device is corrupted: ZIL can not be rolled back properly. I don't know if partial rollback is tried, but from your point of view it would not matter much (because you need all the transactions), so I assume your last 5 seconds are discarded.

From a pool perspective, even this worst case is pretty good - 5 seconds lost, but pool is importable (if its version is at least 19). But from application point, this may be a critical error - the application just wrote 5 seconds of sync data, got confirmation that it was written successfully and after the reboot the data is missing, but the application does not know this. The exact error depends on the application. A DBMS might have become inconsistent and need to be repaired, a large data file might be unreadable, system files might cause difficult to find crashes, an encrypted storage partition might be totally unrecoverable - all because part of it is missing/wrong.

Another point that is seldom mentioned: SSDs might die unexpectedly, so mirroring becomes more important than with HDDs, but if you put two identical SSDs factory-new into the system, your failures might happen at the same time.


You can read a good summary on Solaris ZFS, Synchronous Writes and the ZIL Explained and some details on the portion of data loss situations on The effects of losing a ZFS ZIL SLOG device, as I understand them. Oracle documentation is a bit short, but also mentions that under normal operation the ZIL moves from SLOG to pool devices automatically on SLOG failure (of course, you have 5 seconds of vulnerability there).

The man page also offers information about importing pools without ZIL:

 zpool import -a [-DfmN] [-F [-n]] [-c cachefile|-d dir] [-o mntopts] [-o
         property=value]... [-R root]

     -m      Allows a pool to import when there is a missing log
             device. Recent transactions can be lost because the log
             device will be discarded.
  • NFS/ESXi/sync will be a major use. As the cost and risk are on my shoulders, I'm trying to understand the risk rather than get a recommended approach - if the separate ZIL faila as part of the power outage (whether or not it was intended to be redundant, loss protected, etc), but nothing else is affected, is possible loss/corruption limited to data received by the ZIL and not yet written to pool (last few seconds data at worst), and recoverable, or are there ways that sudden ZIL+power failure (assuming no other kind of failure at the same time) can cause the pool to be irrecoverable? – Stilez Oct 17 '16 at 16:50
  • @Stilez I've added the last two sections in regards to your comment. In summary: ZFS will cope with a pool without ZIL just fine (version 19 onwards), your applications might not. – user121391 Oct 18 '16 at 7:38
  • Thanks, the latest info helps. The main use will be domestic: usual movies,mp3s, documents, + ESXi + snapshots. I'm migrating from VMware workstation + windows file shares to ESXi+ FreeNAS + offsite replication. A worst case for "new data" on the ZIL being lost, will be snapshot being written or new files being copied at the time. At the worst I can handle those if ZFS can get me back to "5-15 seconds before power loss". So my question is - will it? I don't really want to have to restore several TB of replication across my phone line, or have to figure where I'd got up to, if avoidable:) – Stilez Oct 18 '16 at 9:24
  • @Stilez Use of the ZIL depends on the size of the data - large junks of blocks bypass it, only small writes are recorded in it, so you might end up with parts of files okay and parts missing (difficult to say because it depends on the situation). You can compare the resulting situation after the crash with a traditional hard drives were certain sectors are dead - you might not notice it or it might crash all your applications, depending on where it happens. – user121391 Oct 18 '16 at 9:44
  • @Stilez Also, another possibility might be to split the pools: ESXi datastore with SLOG device as first pool, data storage/windows fileshares without SLOG device as second pool. – user121391 Oct 18 '16 at 9:47
1

I use ZFS on 4 servers and also my laptop for 5+ years. I had few power failures on write intensive servers (broken UPS firmware reporting false data) and did not notice ANY* data errors/pool mount problems (which does not mean there was no data loss from latest transaction that had not finish writing as explained before/CoW)

* except one single event when I deviated from ZFS manual: I had ZFS on single disk (iSCIS SAN LUN mapped on host) inside KVM Guest and after initial data copy I forgot to change Cache mode from WriteBack to WriteThrough. Pool (5TB) was readable but had 20k+ errors reported. I had to recreate pool using data from backup server - thanks to zfs snapshots and zfs send/receive I lost only (meaning it could be much worse) 2 min of data. Use ECC memory, disable all write buffering (at least without BBU/FBU – subject for another story), RTFM and ZFS is rock solid.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.