30

Is it likely that a physical degradation of a hard disk could cause bits to 'flip' in file contents without the OS 'noticing' and telling you about it when reading the file? e.g. could a 'p' in an ASCII text file (binary 01110000) change to a 'q' (01110001) and then a user (me) be able to open the file and see 'q' without being aware that a failure has occurred?

I'm interested in answers relating to FAT, NTFS, or ReFS... if it makes a difference.

I want to know if the OS protects me from this, or if I should be checking my data for invariance between copies/over time.

  • Not specifically related, but similar: superuser.com/questions/613702/… – Michael Frank May 11 '16 at 22:02
  • I suppose it could be possible for a damaged read/write head to accidentally apply the wrong charge to the disc, although I've never seen data corruption on that small of a scale. Also, I wouldn't trust Windows to tell me if a drive is failing or not (The message on log-in). I've seen drives die slowly over time without any warning from Windows. – CConard96 May 11 '16 at 22:11
  • Of course... the data is stored as bits with relative values of 0 or 1, entropy occurs in all systems, including magnetic and solid state storage. All data degrades over time. – acejavelin May 11 '16 at 22:46
  • 2
    @Moab : I would be more impressed with your second ("Also...") comment if the URL was, in any way whosoever, different from the URL in your first comment. – TOOGAM May 11 '16 at 23:08
  • 1
    If you use ReFS on top of a mirrored volume (not parity!) and configure it properly it will checksum file data as well as file system metadata. It will be verified on read (and fixed if necessary) and there is also a scheduled job that will periodically scan the entire volume for detected errors. – davidbak May 12 '16 at 18:36
24

Yes, there is a thing called bit rot.

But no, it won't affect you unnoticed.

When a drive writes a sector to the platters, it doesn't just write the bits in the same way they're stored in RAM - it uses an encoding to make sure there aren't any sequences of the same bit that are too long, and it adds ECC codes, that allow it to repair errors that affect a few bits, and detect errors that affect more than a few bits.

When the drive reads the sector, it checks these ECC codes, and repairs the data, if neccesary and possible. What happens next depends on the circumstances and the firmware of the drive, which is influenced by the designation of the drive.

  • If a sector can be read and has no ECC problems, it's passed to the OS
  • If a sector can be repaired easily, the repaired version may be written to disk, read back, and verified, to determine if the error was a random one (cosmic rays ...) or if there is a systematic error with the media
  • If the drive determines there is an error with the media, it reallocates the sector
  • If a sector can be neither read nor corrected after a few read attempts, on a drive that's designated as a RAID drive, the drive will give up, reallocate the sector, and tell the controller there was a problem. It relies on the RAID controller to reconstruct the sector from the other RAID members, and write it back to the failed drive, which then stores it in the reallocated sector that hopefully doesn't have the problem.
  • If a sector can't be read or corrected on a desktop drive, the drive will do a lot more attempts to read it. Depending on the quality of the drive, this might involve repositioning the head, checking if there are any bits that flip when read repeatedly, checking which bits are the weakest, and a few other things. If any of these attempts succeed, the drive will reallocate the sector and write back the repaired data.

(This is one of the main differences between drives that are sold as "Desktop", "NAS/RAID" or "Video surveillance" drives. A RAID drive can just give up quickly and make the controller repair the sector to avoid latency on the user side. A desktop drive will retry again and again, because having the user wait a few seconds is probably better than telling them the data is lost. And a Video drive values constant data rate more than error recovery, as a damaged frame typically won't even be noticed.)

Anyway, the drive will know if there has been bit rot, will typically recover from it, and if it can't, it will tell the controller which will in turn tell the driver which will tell the OS. Then, it's up to the OS to present this error to the user and act on it. This is why cybernard says

I have never witnessed a single bit error myself, but I have seen plenty of hard drives where entire sectors have failed.

the drive will know there's something wrong with the sector, but it doesn't know which bits have failed. (One single bit that has failed will always be caught by ECC).

Please note that chkdsk, and automatically repairing filesystems, do not address reparing data within files. Those are targeted at corruption withing the structure of the filesystem; like a file size being different between the directory entry and the number of allocated blocks. The self-healing feature of NTFS will detect structural damages and prevent them from affecting your data further, they will not repair any data that is already damaged.

There are, of course, other reasons why data may become damaged. For example. bad RAM on a controller may alter data before it's even sent to the drive. In that case, no mechanism on the drive will detect or repair the data, and this may be one reason why the structure of a filesystem is damaged. Other reasons include plain software bugs, blackout while writing the disk (although this is addressed by filesystem journaling), or bad filesystem drivers (the NTFS driver on Linux defaulted to read-only for a long time, as NTFS was reverse engineered, not documented, and the developers didn't trust their own code).

I had this scenario once, where an application would save all its files to two different servers in to different data centres, to keep a working copy if the data under all circumstances. After a few months, we noticed that on one of the copies, about 0.1% of all files didn't match the MD5 sum that the application stored in its database. Turned out to be a faulty fiber cable between the server and the SAN.

These other reasons are why some filesystems, like ZFS, keep additional checksum information to detect errors. They're designed to protect you from a lot more things that can go wrong than just bit rot.

  • 2
    +1 for pointing out that other hardware issues besides drive media degradation can cause corrupted data to be read and written. I personally have had the problem with bad cables within a case. And, FYI, in addition to ZFS, Windows ReFS file system (for Server 2012+) when configured properly and running on top of Storage Spaces will checksum file data as well as file system metadata and recover it, plus it will do periodic whole volume integrity scans to detect and fix many such errors. – davidbak May 12 '16 at 18:00
17

Yes, hard drives can and do degrade with no warning from the OS. It's called bit rot. I have never witnessed a single bit error myself, but I have seen plenty of hard drives where entire sectors have failed.

Windows has no built-in protection of the file contents beyond the structure of the NTFS filesystem. Think of NTFS as a book: well it only protects the table of contents and verifies that things match up. However, if the damage is in the middle of a page, it offers no protection. FAT has nothing. Hard drives employ ECC error correction on a per-sector basis, but the drive doesn't tell Windows. Some file types specifically have CRC, MD5, or SHA hashes to detect corruption, but they don't fix anything.

Even then the hash only tells you there is a problem, but it doesn't know where the error is located.

The hard drive has SMART which monitors the health of the hard drive, but unless the drive is on death's door, BIOS won't warn you. Worst yet, SMART is often disabled by default in your BIOS. You can monitor the numbers through software, but different drives have different issues. If you have a bunch of relocated sectors, or your ECC errors constantly go up. If you have 100,000 new ECCs everyday, it is a bad sign.

Many files types have no protection against bit rot. Like, TXT and BMP, which have no protection at all. Winrar has an optional option to add parity data to the archive which will make the file bigger, but it can detect (proportional to amount of parity data added) and repair this kind of error.

All other compression programs that I know of detect errors, but are helpless to do anything about it.

Eventually, the errors in a sector will be so bad ECC can't correct it, and the drive will give you what it reads even if it is wrong.

You can use QuickPar or similar to create parity data files, but as far as I know there is no way to automate it. For example, you actually change the file yourself when you have to manually update the parity. You can also have parity data for a group of files, but you change 1 file and the whole parity set has to be recreated. This is a real headache for all, but a small number of files.

  • Windows, chdsk, nor NTFS has detection against bit rot, that's handled by RAID, or a file system with parity. A bad partition isn't nor is it caused by bit rot. I approve of this answer – Ramhound May 12 '16 at 0:15
  • 1
    @Ramhound Sadly, the number of users who protect there data with RAID mirroring, level 5, or level 6 is probably less than 0.01% – cybernard May 12 '16 at 0:18
  • I know I was speaking in general. Bit rot!=bad partitions – Ramhound May 12 '16 at 0:21
  • 1
    NOR = NOT OR; used in a sentence means it's an exclusive list; – Ramhound May 12 '16 at 3:07
  • 1
    I had this 750GB hard drive that did things like this. First of all the computer was slow and freezes all the time. When I some text files part of it is zeroed or has been garbled. The computer later stopped booting. Got a new hard disk (I got HDD not SSD. I think I should have got a SSD) and the problem went and the PC is fast – Suici Doga May 12 '16 at 12:04
2

Yes, it is possible. Windows is just software. Software is a series of instructions for a computer to follow.

Think about another type of a series of instructions: a book. What can those instructions accomplish if they are written in a book that sits on a shelf and nobody bothers to open the book and read those instructions?

Just as those written instructions require a person to read the instructions and start following the instructions, computer software requires hardware to do stuff to be useful. Even if a book has instructions which were written with fabulous accuracy, that doesn't prevent problems if a person decides to read the instructions but then to implement them wrong. Similarly, software cannot prevent hardware from doing bad things. So, broken hardware can physically triumph over what any piece of software can do, including Microsoft Windows.

Now, ReFS may be designed with the intent that software will store details about the data, and to have the software compare those details later. A simple concept is "checksum", where software adds certain values and makes sure that those values match an expected result. When hardware implements that software, then certain bad results may be able to be detected. This may even be highly probable to work. However, since the number of potential problems, that might theoretically exist, is basically an infinite number, there is no guarantee that software will necessarily detect every single problem. (Keep in mind that software is a series of instructions that was created ahead of time.)

FAT is particularly low on features. FAT12 was designed for floppy disks, and FAT16 for systems up to 4GB (although most of Microsoft's implementation of FAT16 tended to not work above 2GB). Without the VFAT extension, neither of them supported filenames longer than 11 characters (some of which would be in a portion called the "extension"). FAT was simply designed to store data in a time when the ability to store data was a novel concept that adults needed to be taught about. When FAT was considered a "leading edge" technology, computer technology was not yet sufficiently prevalent and elaborate for people to be worrying about advanced features.

NTFS added support for some more features, perhaps most notably having the operating system be able to easily keep track of user permissions. There are different versions of NTFS. For instance, Moab points out that Windows Server 2008 added support for self-healing NTFS, which may detect some things. Still, that feature was new to Windows Server 2008, so it isn't something supported at all by Windows XP (or Windows Server 2003, or earlier). Even still, looking over the list of features, it appears that this involved some meta-data that helps the operating system notice problems that are so severe that the disk cannot mount, or other key areas of the disk that affect the operating system's kernel. It did not look like every single piece of data, in every single file, gets affected by this one particular feature.

The software for such operating systems is extremely unlikely to notice such things, unless they cause notable problems for the operating system to accomplish tasks. There may be some exceptions, like the portions of the operating system that check disks (CheckDsk/ChkDsk/ScanDisk/ScanDskW, depending on the operating system), but even they will be rather limited on what they can detect, largely because the filesystems don't store a very large amount of data that was intended to be useful for disk checking.

(RAID5 might be more prone to detect such things, with every bit having a parity bit that would help notice something unusual. Even then, it would be up to the RAID implementation to run a check to notice the problem. If the problem occurred on a part of the disk that isn't actively being worked with, the problem may remain unnoticed until someone tries to start using that data.)

In more recent times, larger numbers of bits meant that small likelihoods, like chances of "1 in 10 million", were more likely to affect things. The general public has also learned about "cosmic rays", which may have a small impact on things. Since bits are being crammed so tightly in newer devices, the physical requirements to represent a bit are smaller, so even small impacts are more likely to mess with how a bit is recognized. ReFS has some features designed to help with them being detected. Wikipedia's article on ReFS refers to this as "automatic integrity checking". As that is described as a notable feature of this filesystem, such features are likely more developed than with NTFS (and certainly more than FAT, which was comparatively simple in nature, and so had virtually no such features).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.