External HDD formatted to ZFS, another external HDD as (offline) backup. How do I protect my data against silent corruption?

Question

I am an amateur photographer using a base model MacBook Air (128 GB SSD) with a macOS/Ubuntu 16.04 dual boot. Because the SSD is too small to store my photo library on, I used to store it on a 500 GB 2,5" external HDD that I previously used as a backup drive. Obviously I no longer had a backup of my pictures anymore. Because of this I purchased a 2 TB internal HDD that I use with a 3,5" dock. The latter has now become my 'main' external HDD, with the 2,5" being a backup drive once again.

I formatted the 2 TB drive to ZFS yesterday, because I want to protect my photos against silent data corruption (bit rot). The 3,5" inch HDD was previously formatted to HFS+ and I used Darktable on macOS to manage/edit/whatever :) my photos. Now I will be using Darktable on Ubuntu 16.04 (including its new official ZFS support) with my photo library. I read that ZFS automatically detects and attempts to heal corrupted data by calculating checksums of the data. Then it dawned on me that the self-healing bit is only possible with a mirrored pool. But ZFS could also notify me of a particular photo being corrupted and I could then manually replace it with a copy from the 2,5" backup drive before the backup is polluted with the corrupted version.

Unfortunately I have read that ZFS doesn't appear to notify the user when it detects corrupted files during an automatic checksum calculation/comparison. So my idea isn't possible. Of course I could just mirror the pool with the 2,5" drive, but I don't want to have it plugged in all the time because:

1. my MacBook Air has only 2 ports which would both be used then, which means I wouldn't be able to use my external keyboard (my cheap USB hub died a while ago and I don't really feel like replacing it ;) )

2. I don't want to wear down the backup drive unnecessarily

So what would be the best solution for protecting my data against bit rot with the gear I already own? Surely my situation isn't that unique ;)

Answer 1

I have a similar workflow. You should periodically scrub the external disk:

zpool scrub poolname

This will take several hours, depending on your pool size. You can check on status:

zpool status poolname

In your case where the pool has no redundancy, the scrubbing process will not be able to correct any bit errors but it will tell you which files are affected and you can manually copy them from your good volume to the affected volume.

Alternatively and at the expense of disk capacity, you can give your single-disk pool redundancy at the ZFS filesystem level by using the copies property to tell ZFS to create multiple copies:

zfs set copies=2 poolname/filesystem

Setting this property will only affect new data written, so you should set it during or immediately after creating your pool and filesystem(s).

Answer 2

I could then manually replace it with a copy from the 2,5" backup drive before the backup is polluted with the corrupted version. Unfortunately I have read that ZFS doesn't appear to notify the user when it detects corrupted files during an automatic checksum calculation/comparison. So my idea isn't possible.

Actually, ZFS does (in a way) notify the user of any errors encountered in ordinary operations. It does so by, if the data cannot be read or doesn't validate against the checksum stored elsewhere, throwing an I/O error. How this propagates to the user software depends entirely on how that software responds to an I/O error; in the worst case, it might simply crash; in a better case, it would somehow report that the file is not readable.

what would be the best solution for protecting my data against bit rot with the gear I already own?

If you want at least partial actual protection against bit rot with a single drive, then you should set copies=2 on your pool. Note that this effectively halves the storage capacity and should be done before you store data on the pool. (You can also cause such a setting to take effect by rewriting the files, say by copying them to a separate directory and then copying them back before deleting the second copy.) It's even better to set properties such as checksum, copies, compression and so on during pool creation, if possible.

With only a single copy and no redundant storage, ZFS can detect but not repair most storage-level errors. (Metadata is always stored with an additional copy, so if you have a single-vdev pool with no redundancy and copies=1, you get two actual copies of the critical metadata.) To do this, scrub the pool by running sudo zpool scrub poolname. If you need to cancel a running scrub, run sudo zpool scrub poolname -s (the -s stands for "stop"). When the scrub finishes, you can get a list of all files affected by storage-level errors by running zpool status poolname -v which will hopefully print no known data errors. You can then decide whether you want to throw those files away, or restore them from backups. Do note that it's possible to run ZFS with checksum=off, which disables checksumming. (The zfs(8) man page does caution however that disabling checksums is NOT a recommended practice, emphasis original.)

You should run scrubs regularly. How long a scrub takes depends heavily on how much data you have and your write patterns (writing large files once which are never written to again is much better in this regard than small files that are regularly updated in-place), because of ZFS's copy-on-write architecture eventually leading to fragmentation. I recommend running scrubs no less often than every few weeks. With a few hundred gigabytes of data (based on your statement that you used to store your photos on the 500 GB drive), a scrub should not take more than a few hours.

I don't want to wear down the backup drive unnecessarily

Generally, what kills magnetic rotational HDDs is spin-ups and physical force. A magnetic rotational hard disk doesn't suffer from anything like SSDs' flash wearout (which generally isn't a problem with modern SSDs, anyway). Both HDDs and SSDs are designed and built to be used and as long as they are healthy can take quite a proverbial beating. (It's when a HDD is marginal that you should start to worry, and copy your data off it as soon as possible.)