16

What exactly does btrfs scrub do? According to the manual page, which is totally unclear, it does some error checking. What kind of error checking? How reliable is it? Is it able to recover some errors? How does it work? Does it work on every btrfs disk?

  • 3
    Context: BTRFS stores checksums, so it's always able to tell if a file (or metadata) is ok or if it has been corrupted. Almost all other filesystems like ext4 do not have checksumming, so they won't stop you from reading a file that has been corrupted by a bad drive (which is about to die and has already started corrupting data). This is an important data protection feature in BTRFS and this makes scrubbing possible. – basic6 Nov 12 '15 at 11:58
23

I don't know if it does anything else, but I know that at minimum btrfs scrub does full-disk data scrubbing. Basically, it reads all data* on the disk, recomputes its checksum, and compares the recomputed checksum to the stored one. When the stored and recomputed checksums don't match, the system knows there's corruption.

Once corruption is detected, behavior depends on your disk setup. For example, if you have RAID 1 (mirroring), then btrfs scrub can fix corrupted data by copying an uncorrupted version from another disk. If all copies of some data are corrupted (e.g., multi-disk damage or not having redundant copies in the first place), then there's not much btrfs scrub can do besides warn you.

The reason this is important is that hard drives are only about 99.999999999999% reliable at reading and writing bits. So, every few terabytes of data I/O, there is likely to be an error. Although errors can be and are detected (and fixed, assuming a redundant copy is still valid) during normal disk access, routine full-disk scrubbing is able to find and fix errors before enough accumulate that all copies of the same data are corrupted.

* I'm using "data" instead of "file" to include metadata as well. Btrfs stores files and corresponding metadata (including checksums) in data blocks, all of which are checksummed and checked by btrfs scrub.

See also:

  • Btrfs -> Checksum tree and scrubbing at Wikipedia: Technical information about btrfs's data scrubbing.
  • Birthday problem -> Probability table at Wikipedia: Treating "hash space" as "number of data blocks" and "number of hashed elements" as "number of corrupted data blocks", this gives the probability of there being a data block with both copies corrupted in a RAID 1 setup.
  • I haven't counted, but I'm willing to guess that your reliability figure is off by a few orders of magnitude. Consumer HDDs are usually spec'd to a UBE rate of 10^-14 bits. In other words, one unrecoverable read error per 10^14 bits read. The problem is that this is for a full sector; you either get the full sector or you get nothing at all (or that's the idea; silent errors is another cup of tea entirely). So the error is amplified by the sector size, which with Advanced Format drives is 32,768 bits. Hence, the real error rate is more like 10^-10 to 10^-11 readbit-errors. – a CVn Dec 2 '15 at 20:09
  • @MichaelKjörling I don't think sectors matter here.... I have records of the last 29 btrfs scrubs of my computer's two internal 1TB hard drives. The amount of data has varied between 270 and 300 GiB (for a total of 1.35*10^14 to 1.49*10^14 bits read for all scrubs combined). There have been 3 errors found during these scrubs. Assuming that non-scrub I/O neither caused nor fixed bit rot, that's only 2 to 2¼ times the expected error rate of "99.999999999999% reliable" drives. Even with only 4096-bit sectors, I think your argument would expect my drives to have had thousands of errors by now. – Mark Haferkamp Dec 20 '15 at 10:21
  • @MichaelKjörling So far as I understand manufacturer's spec sheets (Seagate and WD), it's bit errors and not whole sectors that die. And the number of nines in the answer is even optimistic: 100-1/10^14 has 16 nines and the post's has only 14 (corresponding to 10^12). – Luc Jul 28 '18 at 8:52
  • @Luc It's good if sector death is less common; dying sectors mean the drive is (maybe) actually failing and might need replacement. Bit errors just lead to silent data corruption, which can be made noisy enough to restore from backups. Math note: The actual calculation is 1-10^n, which is then converted to percent because humans don't like leading decimals. Also, I neglected to mention in my previous comment that the drives are in RAID 1 (hence the same 270-300 GiB data is on each of them), which fixes another apparent calculation error. – Mark Haferkamp Jul 29 '18 at 15:38
5

Expanding on Mark Haferkamp's excellent answer, btrfs scrub reading all data instead of all files is a critical property and is actually what makes it so useful. Remember, btrfs has builtin RAID support. Say you have a btrfs filesystem spanning two drives that you've configured to use RAID1. In this case, when you write to a file, that write is replicated to both disks. (It gets more complicated with a more complex example but for this simple case, this is always what happens.) However when you read from that file, the read will hit only one disk (because it is a waste to read the file in twice unless the first copy is unusable for some reason).

Now say your second btrfs drive is degrading and starting to corrupt data in your filesystem. When you read blocks from this disk, btrfs will notice that the checksum does not match and will restore the block in-band from a known-good copy - the copy on the first drive. It'll return the data to the application calling read() (or whatever) as if nothing happened.

But what if btrfs doesn't decide to read from the second disk? Remember, there's two copies, so it can read from either the first or the second disk. If it reads from the first disk, it won't notice anything wrong. The only time it'll notice anything's wrong is when the first disk degrades, too. Now you're really hosed as it's too late to recover the data - the second disk's copy has been corrupted for a while, and the first copy (which is what you would've used to restore the second disk) is now corrupted too!

This is where btrfs scrub comes in. It reads all data, not all files. This includes metadata, but also secondary copies of files that wouldn't normally be in the read path. When it reads these secondary copies, that creates an opportunity for btrfs's in-band error correction to kick in and restore the data from a redundant copy.

  • 2
    Are you sure that in RAID1 reading is done only from 1 disk? At least with mdadm RAID this shouldn't be the case. Reading should happen in parallel from both disks, but of different data, meaning it should be twice as fast as reading from one disk. Read performance boost is one of major features of RAID 1. – Petr Mar 15 '17 at 9:08
  • @Petr yes, you're correct. individual blocks are only read from one disk. – strugee Mar 15 '17 at 14:40
  • @Petr: To elaborate, - under normal use neither MDADM nor BTRFS read both copies of the same data from both disks. They only read one copy. - MDADM is able to load-balance and distribute reading across copies to obtain twice the reading speed. (because all copy A go onto disk 1, and all copy B go onto disk 2. As mdadm will use exactly 2 disks). - BTRFS has more difficulties. (because copy A and B will go onto whichever 2 disks (out of 2 or more disks) happened to have most free space at the moment - i.e.: the 2 copies will get randomly spread among all the present disks) – DrYak Aug 8 '17 at 13:47
  • and regarding security : - both MDADM and BTRFS will try the other copy in case of problem. - for MDADM the only problem it can detect is "disk doesn't answer" (because it has no actual idea of the files stored above) - for BTRFS it can in addition detect silent data corruption (because the data is checksumed, and so the BTRFS' RAID layer knows if one copy can be trusted or if it must fetch another copy) – DrYak Aug 8 '17 at 13:48
  • last but not least regarding RAID5/6 : - in MDADM it just works. (But won't detect silent corruptions) - in BTRFS silent corruption for now aren't handled (because it's easier to just fetch another copy (in RAID1) rather than doing the Erasure Coding computation to guess which member of the stripe is corrupt and should be rebuilt from remaining data/parity). In other words : as of today (august 2017) don't use btrfs' RAID5/6. – DrYak Aug 8 '17 at 13:52

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.