1

How can we sure that a readonly snapshot is not corrupted due to a disk failure?

Is the only way calculating the checksums one on another and store it for further examination, or does BTRFS handle that on its own?

Rationale

I'm backing up my snapshots regularly as a prevention to a possible disk failure. Days ago, I couldn't make btrfs send | btrfs receive for a specific snapshot. When I deleted it, rest of the operations went as normal. Moreover, btrfs scrub says there are a few uncorrectable errors. That made me think that my snapshots on primary disk might be corrupted before I backed them up to the external disk and if I'm not aware of this, I would end up with already corrupted backups on external disk.

That's what I'm looking to prevent from happening. I want to be sure that if I (can) backup a snapshot, then it isn't corrupted.

1

There are two possible answers depending on what you mean by 'corrupted by a disk failure'.

If you mean simple at-rest data corruption

BTRFS handles this itself, transparently to the user. It checksums everything, including data in snapshots, internally and then verifies the checksums as it reads each block. There are a couple of exceptions to this though:

  • If the volume is mounted with the nodatasum or nodatacow options, you will have no checksumming of data blocks. In most cases, you should not be mounting with these options, so this should not e an issue.
  • Any files for which the NOCOW attribute is set (C in the output of the lsattr command) are also not checked. You're not likely to have any truly important files with this attribute set (systemd journal files have it set, but that's about it unless you set it manually).

If you mean non-trivial destruction of data on the volume because of loss of too many devices

You can't protect against this except by having another copy of the data somewhere. Pretty much, if you've lost more devices than however many the storage profiles for the volume can tolerate, your data is gone, and nothing is going to get it back for you short of restoring from a backup.


Regarding your specific case

The issues you're talking about with send/receive are probably a side effect of those uncorrectable errors reported by scrub. When BTRFS can't transparently fix an error (usually because the block is stored using a profile that can't do recovery, like single or raid0), it returns an I/O error, which will cause the send operation to fail. So, you won't end up with already corrupted backups if you're using send/receive (and actually, you won't with most other tools either, any good backup software will throw an error if it can't read a file).

In this case, it sounds like the uncorrectable errors are entirely in data exclusive to snapshots or that isn't being snapshotted. You can rather easily (though not very quickly) figure out what files are having issues by mounting the source volume somewhere by itself and running the following command from where it's mounted:

find . -exec cat '{}' \; > /dev/null

That will try to read every file on the volume, and you'll see any read errors on the console, with the name of the file in the error message. This is potentially very slow, so you might want to parallelize it if you have a big volume.

Once you've found and dealt with those files, you should have no further issues. If you see these issues happen again in the near future after fixing them, you should look into checking your hardware, as something is silently corrupting data.

  • Thanks for the answer. This covers a few edge cases. I added my rationale. Please take a look. – ceremcem Aug 15 '18 at 23:42
  • @ceremcem I've updated my answer with some more info based on your updated question. – Austin Hemmelgarn Aug 16 '18 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.