0

I'm running nas4free v. 10.3.0.3., using ZFS, with a four 2TB drive RaidZ1. Once a year, I do a complete backup, off-line the oldest drive, replace it with a new drive, and issue the "zpool replace mytank /dev/ada0" command, and it starts the resilver, automatically.

This weekend, I started the process, just like always. At the time, the time-to-completion was about 8 hours. However, now after more than 24 hours, the resilver still isn't done. Looking at the Disks|ZFS|Pools|Information screen, it appears the resilvering process keeps restarting. Right now, its 0.14% done. I know its been at this point several times before.

So, knowing that resilvering puts a strain on the drives, I ran smartctl on each individual drive. The three "old" drives show no errors at all. Everything's "normal".

However, the "new" drive isn't. Here are a few relevant lines from the smartctl output:

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   001   001   005    Pre-fail  Always   FAILING_NOW 0
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   001   001   000    Old_age   Always       -       2000

SMART Error Log Version: 1
ATA Error Count: 270 (device log contains only the most recent five errors)

Yeah, I think the drive is failing.

Now my questions:

1) How do I offline the bad disk? Just like I always do?

2) Will "offline-ing" the bad disk stop the resilver process?

3) After replacing the bad disk, and issuing the zpool replace command, will the resilver start up again, automatically?

Thanks for your help.

0
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   001   001   005    Pre-fail  Always   FAILING_NOW 0
197 Current_Pending_Sector  0x0022   001   001   000    Old_age   Always       -       2000

Looks to me like you got a lemon. Such things happen; RMA the disk and get a replacement. With a recent purchase, presumably low power-on hours count and failures indicated by SMART, that shouldn't be a problem. If the reseller does make a fuss about it, find a different reseller and start shopping there instead.

1) How do I offline the bad disk? Just like I always do?

You can always offline a disk using zpool offline <pool> <dev>. Just be careful to stay above the redundancy threshold of the pool. (I don't think ZFS will let you remove a disk that will bring a pool below its redundancy threshold without being forced to, and it might not even do it then, but it's easy to fall into the trap of adding the -f without thinking through the consequences.)

2) Will "offline-ing" the bad disk stop the resilver process?

It should, since there's now no resilvering that needs to be done. Do note however that you won't have any redundancy, so any failures (even sector-level I/O errors) while the pool is in that configuration are potentially critical.

3) After replacing the bad disk, and issuing the zpool replace command, will the resilver start up again, automatically?

You might possibly need to zpool online the new disk since the old one was taken offline, but I don't think so. Conceptually, zpool replace <pool> <old> <new> is the same thing as zpool attach <pool> <new> followed by zpool detach <pool> <old> (but you can't attach/detach devices in a raidz vdev).

As far as ZFS is concerned, the new replacement disk will need a resilver, so a resilver will be started.

That said, this part of your question stood out to me:

four 2TB drive RaidZ1. Once a year, I do a complete backup, off-line the oldest drive, replace it with a new drive, and issue the "zpool replace mytank /dev/ada0" command,

I would encourage you to not remove the old disk before installing the new one, especially if it's working tolerably well. (I've had a disk start spewing errors from the mere thought of putting I/O on it, and in such a case, offlining or removing the offending disk may be a reasonable choice.) Rather, hook up the new disk alongside the old one, and zpool replace <pool> <old> <new>. By doing that, you get the benefit of the redundancy you have in the pool; then, if any of the other disks encounter a problem during the resilver, chances are much better that the system will be able to handle that gracefully and without loss of data.

At the very least, run a zpool scrub to completion on the pool immediately before offlining the disk you're replacing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.