30

I have a question concerning unrecoverable errors on a BTRFS file system. Specifically, I've run a BTRFS Scrub recently after experiencing a problem with one of my RAM sticks and it seems to have discovered 4 uncorrectable errors. This is the output:

scrub status for <UUID>
    scrub started at Thu Dec 25 15:19:22 2014 and was aborted after 89882 seconds
    total bytes scrubbed: 1.87TiB with 4 errors
    error details: csum=4
    corrected errors: 0, uncorrectable errors: 4, unverified errors: 0

Luckily I have everything backed up in a tertiary backup so I am not particularly concerned about losing the files (I'm well aware of the issues associated with the experimental status of BTRFS, I have multiple backups to keep my data safe, and determined to continue using it so please no: "Solution; don't use BTRFS" posts).

I would like to know, however, how to determine which files are associated with the uncorrectable errors? I want to find them, delete them, and replace them with their backed up copies.

If anyone has information on how to do this, I would love to hear from you.

Thank you in advance.

Improve this question

7 Answers 7

Reset to default
21

I have found the following method useful...

btrfs scrub the volume.

You will be presented with any number of csum errors as you've shown above.
Using your example error details: csum=4 . Use that number in the tail directive of the following statement:

dmesg | grep "checksum error at" | tail -4 | cut -d\  -f24- | sed 's/.$//'

It is handy to pipe this out to a file (e.g. > csums.txt)

I've tried a number of the suggested inode search approaches and they've all met with limited if any success.

Improve this answer
1
  • 4
    as far as I understand, you are using tail to limit the number of lines displayed and to ignore duplicates. I would recommend using sort | uniq to get rid of the duplicates like so: dmesg | grep "checksum error at" | cut -d\ -f24- | sed 's/.$//' | sort | uniq
    – niklasfi
    Sep 2, 2018 at 20:21
5

Yes, mapping from INODE or Block Number back to a filename can be difficult. If you are really interested, you can try something like this and see which file files to copy...afterall if the file is bad it should throw an error during the copy. I have previously used this type of technique.

 find /mount-point -type f -exec cp {} /dev/null \;

 where mount-point is the ROOT node/mount-point of the affected filesystem
Improve this answer
6
  • Running it now, hopefully it will turn something up. Thank you for your advice, I will update you as to the result.
    – RedHack
    Dec 29, 2014 at 22:00
  • 1
    Sorry to say it does not seem to work =/ it found the first file causing the uncorrectable error, but then it spams the message: "stale file handle" to the terminal unless I terminate it. Granted it found the file, but now I cannot figure out how to get rid of it. Gonna have to contact the BTRFS mailing list.
    – RedHack
    Dec 30, 2014 at 20:04
  • You can move it to a special directory and then exclude it from a further search.
    – mdpc
    Dec 30, 2014 at 20:12
  • 1
    It won't move or copy, it just keeps telling me that the file handle is stale. I cannot even ls.
    – RedHack
    Dec 30, 2014 at 22:04
  • If you use cp -v, you may also monitor the progress: find / -type f -exec cp -v {} /dev/null \; 2> corrupted-files.txt. However, /proc/kcore file might be huge (mine was 128TB) so the copy operation will likely hang. Since /proc directory contains special magical files, we don't need to check for them. Exclude the /proc directory: sudo find / -type f -and -not -path /proc -exec cp -v {} /dev/null \; 2> corrupted-files.txt
    – ceremcem
    Dec 12, 2019 at 13:23
4

dmesg will give you details about the files involved in the uncorrectable checksum errors. The messages typically look like this: "BTRFS: checksum error at logical [...] on dev [...], sector [...], root [...], inode [...], offset [...], length [...], links [...] (path: [...])"; the last piece of information is the absolute path to the file that's corrupted.

Improve this answer
3

I came here looking for the "Uncorrectable error" from BTRFS too. The above grep didn't work for me; I had to use instead:

$ dmesg | sed -n -r 's#.*BTRFS.*i/o error.*path: (.*)\)#\1#p' | sort -u
somepath/somefile.txt

Note how the path is relative to the start of the subvolume - no indication of which subvolume it's in. This luckily wasn't a problem for me.

Improve this answer
4
  • 1
    What is somepath/somefile.txt?  It looks like you are typing it as a separate command — or is it the output from the command you typed?  If it is all supposed to be one command line, please don’t break command lines apart for display purposes — just put it into the answer as one long line.  But, what is it?  Are you providing two inputs to sort (a pipe and a file)?  Or is somepath/somefile.txt meant to be an output file?  (It’s not very helpful to specify output files, unless they are intermediate files that you are using again.  People know how to handle results; e.g., by piping.)
    – Scott - Слава Україні
    Nov 10, 2018 at 17:53
  • Does this answer the original question? I can't tell.
    – I say Reinstate Monica
    Nov 10, 2018 at 17:55
  • @TwistyImpersonator Well, it’s (IMO) clearly meant to be an alternative to Mark’s answer, and that got eight votes (and is an expansion of arrrr’s answer).
    – Scott - Слава Україні
    Nov 10, 2018 at 19:28
  • 1
    @Scott the second line was a sample output of the command.
    – crusaderky
    Nov 11, 2018 at 18:21
2

I also found this thread trying to figure out what to do next after finding BTRFS checksum errors.

The listed answers with dmesg didn't work for me; my scrub took a long time, and by then the oldest messages visible with sudo dmesg were too recent.

One solution could have been to leverage the -W, --follow-new flag to dmesg, something like starting the command below before the scrub:

$ sudo dmesg --follow-new | grep --line-buffered "checksum error at" >> checksum_errors.txt

And then do some post-processing on its output.

However, I found that searching using journalctl's -k, --dmesg and --grep flags were sufficient, and went back far enough to find all the errors I was experiencing. I printed out only the offset and the filename (and just ignored the trailing )) with the command below; in my case I had multiple errors in this one file.

$ sudo journalctl --dmesg --grep 'checksum error' | awk '{ print $25, $31 }' | sort -u

From there, I ran sha256sum on the bad file, confirmed it resulted in an error, and confirmed I had a backup of this file.

Improve this answer
1

Run this after the scrub:

sudo dmesg | grep -e "BTRFS warning.*path:" | sed -e 's/^.*path\: //' | sort -u

It will give you a nice list of all files affected.

Improve this answer
0

Can be simplified a little with:

sudo dmesg | awk -F": " '/BTRFS warning.*path:/ {print $NF}' | sort -u

.

Also, as an aside, I clarified the output of btrfs scrub status with the following:

while read -r f1 f2; 
  do { 
    echo -en "\n$f2\t";
    df -h $f2 | awk 'NR==2 {printf "%s\t%s\t%s\n",$2,$3,$4}';
    sudo btrfs scrub status $f1;
  }; 
done < <(mount | awk '/btrfs/ {print $1 " " $3}')

especially when you have several btrfs filesystems.

Improve this answer
1
  • 1
    Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    May 19 at 12:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .