I have configured a ZFS pool with two drives as a mirror. The operating system is Ubuntu 16.04 and I have been using the zfs 0.6.5 as packaged by the vendor. The drives are 3T WD Green and 3T WD Red (probably not optimal for performance, but this not a consideration), which are of equal size in bytes and sectors. I do not use partitions, but zpool create has made two on each drive for me, as is usual. Per default the OS performs scrub on the pool once a month, and I have ran scrub manually a few times.

Several times the scrub process has found checksum errors on the WD Red drive, but not on every run. They have been repaired automatically and have not caused issues as far as I know. The number shown in the CKSUM column has indicated 3, 5 and 9, and now, after a recent upgrade to the upcoming Ubuntu 18.04 and ZFS 0.7.5, also 31 (with extra information "too many errors" if I remember the message correctly).

Alarmed, I detached the drive from the pool, and exported the pool. Without importing the drive, I ran badblocks -b 4096 -s -v -w on it, but it reported (0/0/0) errors. Also smartctl -a /dev/sda indicated nothing out of the ordinary, if I understand correctly (| grep -i error):

  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

I re-attached the disk to the pool and it is currently re-silvering. But I remain mystified: what might have caused the reoccurring scrubbing errors? What should I do in the future to either figure out better what the issue might be, or avoid it altogether? I am not particular eager to look into purchasing replacements drive(s), especially since the WD Red is manufactured only in 2016.

(I'm not sure if this is relevant, but at some point either operator error or software bug caused the non-problematic WD Green drive partition table to get corrupted. I could not find any other action to return it to the pool than detaching it, clearing the partition table and re-attaching it. During re-silvering process some blocks failed to be read from the WD Red drive and I restored the affected one file from backups. Scrubbing detected checksum errors both before and after this incident.)


There's no easy way to tell what the checksum failures were caused by, since they happen independently of the filesystem (unless they're caused by bugs in the FS itself, but I don't think that's what's happening here). The smartctl and badblocks successes make me hopeful that the issue isn't a failing disk.

This is the page that's meant to help you understand the error: http://illumos.org/msg/ZFS-8000-9P. Quoting from that:

For example, the following cases will all produce errors that do not 
indicate potential device failure:

- A network attached device lost connectivity but has now recovered
- A device suffered from a bit flip, an expected event over long
  periods of time
- An administrator accidentally wrote over a portion of the disk
  using another program

I think at this point, checking the connectivity to the drives and running resilver is the right path.

| improve this answer | |
  • Thanks. I have unplugged the drives at least twice, and switched the SATA cables at the same time. I doubt a faulty cable or them being poorly attached could be the case for the issues. – taneli Apr 20 '18 at 4:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.