I'm running a file server using Linux software raid, set up as raid6. As filesystem I've chosen btrfs. However, I'm not using btrfs own raid6 implementation as it's not very stable yet. So basically, I'm using raid on the block level of things. Additionally, I'm making backups to another location of really important data. Lastly, I've connected a UPS to the server to protect against power loss and surges.

My questions:

  1. I'm aware raid6 gives quite a good protection against harddrive failure, but will btrfs on top of it really detect bit rot and correct it? If so, how is that possible without filesystem raid1?

  2. Do md go through the raid6 block from time to time to find and correct corruptions? Do I need to schedule it?

  3. If I've done things wrong, what do you recommend I do instead, if I want good protection against data failure and bit rot?


1: In your configuration, btrfs should detect data corruption but won't correct it as there is no data redundancy as far as btrfs is concerned.

2: RAID6 doesn't detect nor reliably protect against bit corruption. It only protects against disk failure and in some cases might detect unused still bad blocks.

3: If btrfs RAID6 is not ready, you might consider either btrfs mirroring or ZFS RAIDZ2.

  • 1
    (Answering a question that disappeared: will RAIDZ6 protect against bit-rot) Yes, it does it by design. There is still a risk of corruption due to RAM bit-flip. End-to-end data integrity requires ECC RAM and is recommended at least for production systems. By the way, that's RAIDZ2, not RAIDZ6. – jlliagre Jun 16 '14 at 20:13

BTRFS RAID-6 is (though still experimental) actually pretty stable now. The current version even manages to fix many typical errors, including replacing a failed/missing drive.

Like ZFS, BTRFS does checksumming, which means that you can always (and you should, periodically) run a scrub to verify your data. If data (data or metadata) on a drive is damaged, BTRFS will be able to detect the errors and if it has redundancy (using BTRFS RAID-6), it can fix the affected files. After this, it will know if the repaired files are correct because it has checksums.

Classic (hardware) RAID systems, including MD (software) RAID, do not have checksums. Those systems rely on parity only, so an unfortunate combination of errors on several drives could very well lead to corruption. Since there are no checksums, the RAID system (like md) won't be able to verify that all errors are gone after a scrub (i.e., that the files are correct). There are examples (also some videos) that demonstrate data corruption on a classic RAID system.
It's important to be notified as soon as the first parity error is detected and fix the issue as soon as possible (scrub). Also, since the parity is the only (not very reliable) way for the RAID system to know if your data is ok or not, battery backups should be used to prevent losing all this valuable parity data to the write hole when the power goes out.

Now, if you use an advanced filesystem like BTRFS that does checksumming as a single filesystem (without redundancy) on top of a dumb RAID-6 system, it's up to this RAID system to detect and fix errors because once too many errors have corrupted the RAID system, BTRFS will NOT be able to fix them. It will detect errors and it will help you decide what to restore from your backup by telling you which files (path) are corrupt. But otherwise, it would be too late by then. This is why this setup might not be such a good idea after all.

If you use BTRFS the way it's supposed to be used, by creating a BTRFS RAID-6 filesystem (without MD RAID) with direct access to your drives, it will be able to fix errors reliably and it will know if the errors are actually gone because it has checksums. It will tell you on which drives these errors occurred, so you know which drive is bad (you can replace it using btrfs commands). The point is, no matter if a drive is dead/missing and you have to replace it or if a drive is partially corrupted (because it's about to die), BTRFS will detect the errors reliably. Of course, periodic scrubs are as important as with other RAID systems, to detect silent corruption (hint: cronjob).

So again, BTRFS RAID-6 is still considered experimental in some ways, but by having checksums it already offers a reliable way to detect (and then fix) errors. It's experimental, so it could crash in certain cases, then you should try a newer kernel version, which is the fix for many BTRFS issues. Make sure to stay up to date with your kernel (4.3 at the time of writing, don't use anything older than that for RAID-6). But typical RAID use cases (just storing lots of data and at some point replacing a drive - a multi-drive failure might be different) already work with BTRFS.

You've tagged your question with . ZFS isn't included in the Linux kernel, so it would have to be installed manually - the ZFSonLinux port works very well. It might be necessary to reinstall it or do some other fixing after a new kernel version has been installed, but that's not important now. Of course, there are some things that work differently in ZFS. Also, unlike a BTRFS filesystem, a ZFS RAIDZ2 (like RAID-6) zpool cannot be resized (there are "tricks", but bottom line is that a raidz2 vdev cannot really be grown by adding a drive), so an existing system cannot easily be expanded with more drives. However, as far as stability is concerned, ZFS is probably the best choice of all. With checksums, it offers reliable data protection as described and it's mature enough to handle pretty much everything (multiple drive failures, shaky controller, ZFS survives almost everything and can protect/fix your data as long as you have enough good drives). Given that (like BTRFS) ZFS needs to access your drives individually (to be able to fix errors), you definitely should not use it on top of an MD RAID volume unless you have a good reason for it.

  • 1
    Worth noting: RAID 5/6 are more or less feature-complete, but RAID 5 at least is a LOT slower than RAID 1/10 to scrub. My 4x4 TB RAID 5 array takes 2 days to scrub where before with the same drives on RAID 1 it would take about 10 hours. I'm not sure how the IO benchmarks compare nowadays, but they used to be quite a but slower there as well. Not unusable for a media partition, but possibly an issue for the OP's use case. – Mikkel Jul 16 '16 at 1:49
  • @Mikkel: Yes, you are correct. Scrubbing a RAID-5/6 fs takes longer than scrubbing a RAID-1 fs. The high load on the drives also increases the risk of another drive failing during the scrub, which is why RAID-6 should be used instead of RAID-5. However, the system stays usable at all times. – basic6 Jul 22 '16 at 17:08
  • It looks like btrfs’s RAID1 will auto recover from one mirror being corrupted as long as the data is available on another mirror. The wiki says that if a read encounters this it will even automatically repair the broken device. But you still need to do manual scrubbing to be safe because the RAID1 read optimization may read the good copy and not notice the bad copy and fix it magically. – binki Nov 25 '16 at 17:33

In your setup it would not.

What I would recommend is using ZFS built-in RAID-z2, which is then able to perform bit-rot recovery. Enabling lz4 compression will also provide better performance, as well as space savings.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.