5

Btrfs offers these commands to verify data integrity/checksums:

btrfs scrub start <path>|<device>
btrfs check --check-data-csum

However, AFAIK those always verify whole filesystems; the path argument is to identify a filesystem on a device, not file/directory within filesystem.

Now, I have a 3TB Btrfs filesystem. Scrubbing it takes hours. Sometimes I need to make sure that only certain file/directory has not yet been affected by bitrot — for example, before using an *.iso installation image or restoring a backup. How do I use Btrfs for this — without falling back to keeping manual hash files per each file?

I am aware that Btrfs does not store checksums for individual files — it stores checksums for blocks of data. In this case what I am looking for is a command/tool that identifies all the blocks used for storing certain files/directories and verifies those blocks only.

I read somewhere that Btrfs allegedly verifies checksums on read. That is, if a file has been bit-rotted, reading it would fail or something like that. Is this the case?

  • Do you regularly have a problem with files becoming corrupted? That's not normal, I'd suspect a bad drive or ram first, maybe bad filesystem drivers second. Instead of keeping hashes or checksums in a separate database to verify files, you could keep files in an archive that stores a checksum itself, and would save some space too. – Xen2050 Jan 13 '18 at 7:57
  • @Xen2050 no, no problems so far, just want to be able to get 100% confidence. Archives with built-in checksums are an option, but I'd like to employ Btrfs. – Greendrake Jan 13 '18 at 8:43
6

The answer is: simply try reading the whole file. If it reads differently from what has been checksummed, there will be an Input/output error. So yes, Btrfs indeed verifies checksums on read!

To find out this answer, I put together the following test:

  1. Allocate a 1 Gb file to be used as a block device for testing Btrfs partition, mount it as a loop device and format Btrfs on it;
  2. Create a dummy 800 Mb file containing a known unique sequence of bytes in the middle (token1);
  3. Write the file to the Btrfs and record its sha256 for later reference;
  4. Unmount and patch the block device file so that one byte is changed. For this, we sed-replace token1 with token2;
  5. Mount again and try getting sha256 of the 800 Mb file on the Btrfs. See Input/output error;
  6. Unmount, patch back, mount and see that the 800 Mb file is readable again and sha256 is the same as in step 3;
  7. Profit!

Here is the script:

#!/bin/bash
f="btrfstestblockdevicefile"
ft="btrfstestfile"
loop="/dev/loop0"
mount_dir="btrfstestdir"
size="1g"
token1="36bbf48aa6645646fbaa7f25b64224fb3399ad40bc706c79bb8276096e3c9e8f"
token2="36bbf48aa6645646fbaa7f25b64224fb4399ad40bc706c79bb8276096e3c9e8f"

f_mount() {
    echo "Mounting..." && \
    sudo losetup $loop $f && \
    if ! [[ -z $1 ]] ; then
        sudo mkfs.btrfs -q $loop
    fi
    mkdir $mount_dir && \
    sudo mount $loop $mount_dir
}

f_umount() {
    echo "Unmounting..." && \
    sudo umount $loop && \
    sudo rmdir $mount_dir && \
    sudo losetup -d $loop
}

echo "Allocating file for test block device..." && \
fallocate -l $size $f && \
f_mount 1 && \
echo "Generating test file..." && \
dd if=/dev/urandom of="${ft}1" bs=1M count=400 status=none && \
echo $token1 > "${ft}2" && \
dd if=/dev/urandom of="${ft}3" bs=1M count=400 status=none && \
sudo sh -c "cat ${ft}1 ${ft}2 ${ft}3 > ${mount_dir}/${ft}" && \
rm "${ft}1" "${ft}2" "${ft}3" && \
echo "Calculating original hash of the file..." && \
sha256sum "${mount_dir}/${ft}" && \
f_umount && \
echo "Patching the file in the block device file..." && \
sed -i "s/${token1}/${token2}/g" $f && sync && \
f_mount && \
echo "Trying to read the file..." && \
sha256sum "${mount_dir}/${ft}"
echo "OK, unmount, patch back and try again..." && \
f_umount && \
sed -i "s/${token2}/${token1}/g" $f && sync && \
f_mount && \
sha256sum "${mount_dir}/${ft}" && \
echo "Yay, Btrfs rules! Cleaning up..." && \
f_umount && \
rm $f && \
echo "All clear!"

As expected, replacing mkfs.btrfs with making a non-checksumming filesystem (e.g. mkfs.ext4) allows for the corrupted file to be read. Of course, its sha256 is different from the non-corrupted one.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.