0

As the title says, I am concerning about, how can we avoid data loss due to harddisk (either mechanical HDD or SSD) bad sectors.

\ More information \

Question is well defined. Question ends here. Let me try to provide more information. Storage device failure (hardware failure, excluding virus thingy first) mainly split into two types.

  1. Entire disk fault. Anything mechanically and/or electronically goes wrong inside the disk, makes the disk not accessible.

  2. Bad sector. Only a certain part is fault. The entire disk appears to be working fine. If that certain bad sector is not accessed/checked, we will never know there is a bad sector.

\ RAID cannot avoid data loss due to bad sectors \

RAID1, RAID5 or RAID6 can avoid data loss due to case 1. However, as far as I know, standard RAID will not scan and check all the data regularly, which means RAID cannot avoid data loss due to case 2. For example, in the case of RAID1 mirror. If a bad sector appears in the first disk, a file is corrupted in the first disk. At that time, we still have the mirrored file in the second disk. But we do not know there is a bad sector. Thus, the RAID1 array will not indicate a disk fault, and will not trigger a recovery. Then time goes on and on, more and more bad sectors appears. (Every hardware worn out, it is just a matter of time.) There is definitely a chance that the exact mirrored bit / mirrored part in the second disk suffer from a bad sector too. At this time, there are no more redundant copies. The affected data will be loss forever. Similarly, we will not notice this immediately. We will only find out this loss until we access that piece of data.

\ Is ZFS (a filesystem with checksum) a solution? \

Because of the concern that I wish to avoid data loss due to bad sectors, I started looking into the file checksum thingy. It seems that not many popular filesystems include checksum of every file. I have some knowledge with Gentoo Linux. Thus I am planning to use ZFS on Gentoo Linux.

The ideas of ZFS that "Administration of storage should be simple" and "Redundancy should be handled by the filesystem" are good. It seems to me that if a bad sector occurs in a ZFS disk, it will recovery it silently (Isn't it? I cannot confirm this yet). If more and more bad sectors arise in a ZFS disk, does it mean the size of that ZFS disk will shrink? If not, how can I notice when a bad sector happens? How do I know when the harddisk bad sectors are too many and I have to replace it with another healthy harddisk? I guess I need some ZFS monitors and ZFS utilities, which I cannot find much information on the Internet.

1

Given that you're assuming data storage will always fail eventually, you'll just have to keep more than one copy of data, and occasionally check & make sure you can successfully read the data.

Using some type of RAID sounds like a good way to have at least one backup copy, but additional copies are a great idea, at least for the obvious reason of not keeping all copies in the same building, room, and machine. Online backups are a good way to let someone else worry about their drives failing.

For making sure the data's still readable, just keep your own checksums, and verify them regularly. A basic CRC would work, like cksum from GNU's core utilities, or even md5sum or shaXsum (though they're overkill just for bad sectors IMO).

[And keep a few copies of the checksum files, for obvious reasons.]

Even if ZFS has it's own checksums, you should still keep your own copy, to verify copies stored on other filesystems or online. And I'm assuming you or ZFS will have to read all the files to verify them anyway (I believe it's called ZFS data scrubbing, explicitly called with zpool scrub).


It's also interesting to consider that many drives (especially flash drives) do their own bad block management (& wear leveling), swapping bad & marginal blocks out for new spares, all (mostly) silently without you or the operating system even noticing. I read about flash memory cards once, how out of 16GB of memory there was only enough good memory to make a 512MB or 1G card.

0

Regarding RAID 5/6

RAID 5/6 has parity data.

If it can rebuild a whole disk certainly there is enough parity data to recover a single sector.

Worst case, you detect a bad sector replace the whole drive, and now everything is been rebuilt.

So if a RAID 5/6 detects a bad sector it should automatically use the parity data to repair it.

My Adaptec RAID controller even has a special "verify/fix" option in the GUI.

Also note checksums,sha1,sha2, CRC and etc only offer the ability to detect when an error has occurred. It does offer any ability to fixed them.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.