0

I am trying to understand the implications of bitrot and data corruption on my data on disks.

Currently for my family photos (JPEG and MOVs), aside from storing them on my main pc, I have them on a USB attached 2TB sata drive on my dd-wrt router. This uses an Ext3 filesystem. I also have a nightly rsync backup to another identical drive on the router.

Would rsync catch/report/fail to copy any corrupted data on the first disk that has been corrupted by a bad sector Or by a flipped bit when reading the data from disk and would a bad sector prevent reading of the file for rsync to copy it?

Improve this question
1
  • Hi and welcome to SuperUser. You have several questions and unless they are very tightly connected we only allow one question per question. Please edit your question to comply with our guidelines. Otherwise this question stands a high risk of being closed.
    – Nifle
    Dec 30, 2014 at 16:14

2 Answers 2

Reset to default
3

Ok, I'll take a stab at it...

Firstly, bitrot is a term usually meaning 'silent bit corruption'. How serious a problem is it? It is debatable. Typically a sector(s) goes bad and the HDD replaces it/them without you knowing. So for example, an image falls on a sector that goes bad and you get a partially corrupt image or worse, on a ZIP file, you lose the entire file(s) in it since you can no longer uncompress it.

rsync is not designed for the purpose of recovering from bitrot. As the wiki nicely puts it, it is a widely-used utility to keep copies of a file on two computer systems the same. If bitrot occurs, which of the two copies is the correct one? For that you need 3 or more copies.

BTW, on a disk with files that have had sectors replaced, rsync will still work as expected. It read the file and see it as changed. It will copy it just fine. It is only when the sector cannot be replaced automatically by the HDD, because the pool of replacement sectors is exhausted, will it get stuck.

Since you have 3 copies of your data you could stitch something together with rsync to do a deep scan using checksums of all three copies, compare and update accordingly. I do not know if anyone has done anything like that but I think it should be possible.

The best way to go if you are seriously worried about bitrot is to store your files on a ZFS filesystem on your storage device since it provides detection and recovery of bitrot. I would suggest you read up on setting up FreeBSD as a NAS server. Alternatively I recommend you research RAIDZ disk solutions that do the same thing. These, I warn you, would be expensive.

Hope this helped.

PS: bitrot can happen in other ways not just on HDD. Non-ECC memory for example.

Improve this answer
3
  • Thanks for your reply. To clarify, you said the HDD replaces the bad sector without me knowing, and any data that was on that replaced sector is lost ? Or does the disk have a chance to recover that sector (when can this occur)? I think i understand better now after i saw that link that you posted that rsync is not a good solution against bitrot at least with a simple replication that i am doing. An incremental backup may be better i think as then i would at least be able to identify that a file changed and go back the the last known good version
    – skimon
    Dec 30, 2014 at 20:47
  • Yes the bad sectors are unreadable and the data is usually lost. This often manifests by applications/windows crashing for no reason or locking up.
    – Dude named Ben
    Dec 30, 2014 at 20:57
  • "Alternatively I recommend you research RAIDZ disk solutions that do the same thing. These, I warn you, would be expensive." I've never heard of anyone mention raidz outside of the context of ZFS. Can you elaborate?
    – Josh Enders
    Sep 18, 2023 at 18:27
1

kinda vague Q... but when I get read errors there's some sort of message, like "read error on ..." and the read fails.

If you're worried about backups going bad, make 2 backups, and keep a checksum like crc32, or if concerned about tampering md5, sha, etc...

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .