One of my ZFS pools can no longer be imported due to the following message:
status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. ... data1 UNAVAIL insufficient replicas
I would like to understand what "insufficient replicas" means, and if there is any way to recover from this safely before I pull out the backups and rebuild the storage pool.
I would also like to understand what caused this and what I can do to prevent this situation in the future. I understand ZFS rocks but as a ZFS newbie it's scary that I need to pull out backups for something as innocuous as pulling a drive out of a box.
I recently powered down this box and pulled out the SATA sda disk. I'm 99% sure that I let the system power down fully before unplugging it from the wall, but it is possible I might've done a hard power off if the shutdown was taking too long.
When plugging the drive back in, I believe I plugged it back into the same SATA port on the motherboard, but it's possible I might have plugged it into a different port.
The box also contains an older PATA sdb disk on which Debian is installed as the only operating system on an ext partition.
In the past, I would boot up, all my pools would be imported and volumes mounted. This time I got nothing.
When I booted up the box, I was at the usual GDM log-in screen and realized I couldn't log-in. I later discovered I had no user home directories since they're all on a pool.
From the command line, I discovered the following:
# zfs list no datasets available # zpool list no pools available # zpool status no pools available
My drives were available; fdisk produced expected output and I had my /dev/sdaX and /dev/sdbX devices.
zpool import gave me interesting results. The other pools reported ONLINE, but the one in question reported this:
pool: ... id: ... state: UNAVAIL status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. see: http://zfsonlinux.org/msg/ZFS-8000-5E config: data1 UNAVAIL insufficient replicas sda2 UNAVAIL
I was able to import the other pools at this point using
zpool import poolname and then see everything was fine with them with
No so with this one:
# zpool import data1 cannot import 'data1': one or more devices are already in use
zpool import showed the same output for this pool.
I thought about deleting the
/etc/zfs/zpool.cache file. I exported everything and then checked the directory:
# zpool export -a # ls /etc/zfs
There was no zpool.cache after the export, so I simply rebooted. (I don't know what would happen if I deleted the zpool.cache while pools are imported, so I didn't try.)
Same results after reboot. The other pools are fine and this one can't be imported.