3

Summary

One of my ZFS pools can no longer be imported due to the following message:

status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
...
    data1       UNAVAIL  insufficient replicas

I would like to understand what "insufficient replicas" means, and if there is any way to recover from this safely before I pull out the backups and rebuild the storage pool.

I would also like to understand what caused this and what I can do to prevent this situation in the future. I understand ZFS rocks but as a ZFS newbie it's scary that I need to pull out backups for something as innocuous as pulling a drive out of a box.

Background

I recently powered down this box and pulled out the SATA sda disk. I'm 99% sure that I let the system power down fully before unplugging it from the wall, but it is possible I might've done a hard power off if the shutdown was taking too long.

When plugging the drive back in, I believe I plugged it back into the same SATA port on the motherboard, but it's possible I might have plugged it into a different port.

The box also contains an older PATA sdb disk on which Debian is installed as the only operating system on an ext partition.

In the past, I would boot up, all my pools would be imported and volumes mounted. This time I got nothing.

Today

When I booted up the box, I was at the usual GDM log-in screen and realized I couldn't log-in. I later discovered I had no user home directories since they're all on a pool.

From the command line, I discovered the following:

# zfs list
no datasets available

# zpool list
no pools available

# zpool status
no pools available

My drives were available; fdisk produced expected output and I had my /dev/sdaX and /dev/sdbX devices.

zpool import gave me interesting results. The other pools reported ONLINE, but the one in question reported this:

   pool: ...
     id: ...
  state: UNAVAIL
 status: One or more devices contains corrupted data.
 action: The pool cannot be imported due to damaged devices or data.
    see: http://zfsonlinux.org/msg/ZFS-8000-5E
 config:

        data1       UNAVAIL  insufficient replicas
          sda2      UNAVAIL

I was able to import the other pools at this point using zpool import poolname and then see everything was fine with them with zfs list.

No so with this one:

# zpool import data1
cannot import 'data1': one or more devices are already in use

zpool import showed the same output for this pool.

I thought about deleting the /etc/zfs/zpool.cache file. I exported everything and then checked the directory:

# zpool export -a
# ls /etc/zfs

There was no zpool.cache after the export, so I simply rebooted. (I don't know what would happen if I deleted the zpool.cache while pools are imported, so I didn't try.)

Same results after reboot. The other pools are fine and this one can't be imported.

1
  • ZFS is generally insensitive to device names, and will go looking for a device if it's not where the cache file says it is. (Each ZFS backing device stores a unique identifier, IIRC 64 bits, and all devices in a pool store a list of that pool's devices by ID.) That said, can we see the output of zdb /dev/sda and/or zdb /dev/sda2 (assuming that the problematic ZFS disk is actually being identified by the OS as /dev/sda, otherwise adjust as appropriate), please? – user Nov 17 '16 at 13:02
2

I would like to understand what "insufficient replicas" means

It means ZFS encountered an error reading or writing to the pool, then subsequently discovered that there is not enough redundancy in the pool to fix the problem. Since it seems you configured ZFS for single-drive pools, this simply means it ran into a problem and now can't recover on its own.

In the simple case, zpool status -v will report individual files that are now irretrievably corrupt. Restoring them from backups or removing them will return the pool to a non-corrupt state. This is one of the advantages you get with ZFS as opposed to separate file system, logical volume manager, and RAID subsystems. An error at the disk level affecting a single file can be reported as an error in that specific file, whereas with FS + LVM + RAID lash-ups, an error at the lower levels cannot maintain semantic context as it propagates up the stack.

According to your question, however, zpool status is not reporting a problem with particular files, which means something more fundamental has gone wrong with the lone disk in the problem pool. At this point, I would be looking at diagnosing the disk itself: smartctl, badblocks, SpinRite, vendor diagnostic tools, etc.

In the future, strongly consider using at least mirrors with ZFS. ZFS can either fix such problems for you when given some redundancy to work with, or at least let you limp by until you can arrange disk replacement.

any way to recover from this safely before I pull out the backups

If ZFS could do it, it would. ZFS doesn't give up easily, but when it does, it means you cannot safely use the data on the pool.

At this point, your best hope is that the problem is that the underlying storage medium has a soft error that will allow SpinRite, badblocks, or similar to come along and force the disk to pull up a clean copy of the bad sector one last time and copy it to a spare sector.

Beware that ZFS is stricter than other filesystems in this regard, in that it will not accept a partial sector recovery. The strong data checksumming means that only a complete restoration of every sector can cause ZFS to accept that the filesystem is now clean.

If you're of the persuasion that partial data is better than no data, this means ZFS is a problem for you. If instead you believe that incorrect data is incorrect, ZFS is doing exactly what you want it to.

one or more devices are already in use

That's not good.

It either means you've let some other kernel driver attach to that disk before ZFS could get to it, or ZFS is finding the same disk by multiple paths.

Your best hope is that you're in the second case, and forcing ZFS to search for disks by only one path can slide you past this problem:

$ sudo ZPOOL_IMPORT_PATH="/dev/disk/by-id" zpool import -a

If that works, then exporting and re-importing the problem pool may work without the force, since ZFS usually remembers the path by which its vdevs were last imported.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.