testing ZFS - how to destroy an UNAVAIL pool without reboot

Question

I am testing ZFSonLinux and i have an experimental pool named usbcka. There is only testing data. I am simulating HW failure to deterimine behaviour in this scenario. I know this setup has no redundancy, that is accepted as a risk (decision not made by me). EDIT: additional info: OS will be on redundant storage

root@carbon:~# zpool status -xv
  pool: usbcka
 state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: http://zfsonlinux.org/msg/ZFS-8000-HC
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    usbcka      UNAVAIL      0     0     0  insufficient replicas
      usb1      ONLINE       0     0     0
      usb2      FAULTED      0     0     0  too many errors

errors: List of errors unavailable (insufficient privileges)

The devices usb1 and usb2 are LUKS devices mounted with cryptsetup.

** As a test, i have intentionaly physicaly removed the underlying device for usb2 ** Suppose the faulted device was a HDD which was beyond repair, HW failure.

I tried

root@carbon:~# zpool destroy -f usbcka
cannot open 'usbcka': pool I/O is currently suspended

root@carbon:~# zpool set failmode=continue usbcka
cannot set property for 'usbcka': pool I/O is currently suspended

Also

zpool clear usbcka

does nothing,empty output

Also

zpool export usbcka

Hangs

How to destroy the pool without reboot?

I want to use the still working devices for creating a new pool

The reason why i don't want a reboot is that i am testing how ZFS fails when a device is faulted, my intention is to use ZFS in work on companys server. That server simply cannot be rebooted at my whim if one of 10 disks fails,and it has hot-swap trays.

I've searched for solutions, but all of them eventually end with a reboot solution.

root@carbon:~# zfs list usbcka
cannot open 'usbcka': pool I/O is currently suspended

root@carbon:~# zpool get all usbcka
NAME    PROPERTY                    VALUE                       SOURCE
usbcka  size                        174G                        -
usbcka  capacity                    1%                          -
usbcka  altroot                     -                           default
usbcka  health                      UNAVAIL                     -
usbcka  guid                        1317589842010265379         default
usbcka  version                     -                           default
usbcka  bootfs                      -                           default
usbcka  delegation                  on                          default
usbcka  autoreplace                 off                         default
usbcka  cachefile                   -                           default
usbcka  failmode                    wait                        default
usbcka  listsnapshots               off                         default
usbcka  autoexpand                  off                         default
usbcka  dedupditto                  0                           default
usbcka  dedupratio                  1.00x                       -
usbcka  free                        170G                        -
usbcka  allocated                   3.26G                       -
usbcka  readonly                    off                         -
usbcka  ashift                      13                          local
usbcka  comment                     -                           default
usbcka  expandsize                  -                           -
usbcka  freeing                     0                           default
usbcka  fragmentation               1%                          -
usbcka  leaked                      0                           default
usbcka  feature@async_destroy       enabled                     local
usbcka  feature@empty_bpobj         enabled                     local
usbcka  feature@lz4_compress        active                      local
usbcka  feature@spacemap_histogram  active                      local
usbcka  feature@enabled_txg         active                      local
usbcka  feature@hole_birth          active                      local
usbcka  feature@extensible_dataset  enabled                     local
usbcka  feature@embedded_data       active                      local
usbcka  feature@bookmarks           enabled                     local
usbcka  feature@filesystem_limits   enabled                     local
usbcka  feature@large_blocks        enabled                     local

Answer 1

I had the same problem on debian testing with kernel 4.14 and the zfs module 0.7.6. But I think this behavior is the same on all Linux boxes.

To destroy an UNAVAILpool, where the status looks like:

user@system: ~ $ sudo zpool status -xv
pool: icy_tank
state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
see: http://zfsonlinux.org/msg/ZFS-8000-HC
scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    icy_tank    UNAVAIL      0     0     0  insufficient replicas
      sdb       UNAVAIL      0     0     0

errors: List of errors unavailable: pool I/O is currently suspended

you have to restart the zed, the zfs event daemon first by restarting the whole systemd zfs-target.

sudo systemctl restart zfs.target

After this step the pool can be cleared.

sudo zpool clear icy_tank

If the device (e.g. sdb) is available at this time and sdb is part of the pool, zpool immediately starting a scrub on the device.

pool: icy_tank
state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
see: http://zfsonlinux.org/msg/ZFS-8000-8A
scan: scrub in progress since Wed Mar  7 20:27:55 2018
    316M scanned out of 121G at 9,89M/s, 3h28m to go
    0B repaired, 0,26% done
config:

    NAME        STATE     READ WRITE CKSUM
    icy_tank    ONLINE       0     0     0
      sdb       ONLINE       0     0     0

errors: 4 data errors, use '-v' for a list