2

I am thinking of using hardware RAID (levels 5 and 10) on my computers. What I don't get is how is data recovered when something goes bad and the RAID array fails (RAID Card problem or anything else).

When using standalone disks the process is very straightforward, but what does one do when a RAID array fails, how do people repair the array and resume work; and how is data recovered from a RAID array. since we cant just take out the drive and plug it in other computer.

  • 1
    Generally, it's not considered worth it to bother. You just build a new RAID array from a backup. If you're using RAID, presumably your goal is to minimize downtime, and wasting time on recovery hurts you more than it helps. – David Schwartz Nov 11 '15 at 21:25
3

When a drive dies in a HW RAID, you pull it out and plug a new one. An automatic rebuild takes place. In some cases you may have to use the card's GUI or CLI to notify the card that a drive has been replaced. The RAID is, of course, operational during the whole excercise.

If your RAID card dies, then that's much tougher. You should have a spare. If you don't, you buy one from your supplier. If it's no longer in production, you go frantically looking on eBay for a used one.

If you can't find even a used one, then it boils down to reverse engineering the on-disk format and writing some code to recover it. You may be lucky and find that it is stored in a fairly standard format. You either do it yourself or pay a lot to a data recovery company.

In that light, a SW RAID is more future-proof, as the software itself will not break over time and commodity hardware can be easily replaced.

However, SW RAID5 cannot be made fully resilient to power outages. That's the big advantage of HW RAIDs - they can be equipped with battery backed memory and thus be fully safe with regards to pulling the plug.

  • Another factor: In my experience a blue screen (or any other improper shutdown) with software raid causes the drive to fault and rebuild even if there were no pending writes. – Loren Pechtel Nov 11 '15 at 21:53
  • Yes, a system crash or pushing the reset button will damage the SW RAID array in the same way as a power outage. The only way the SW RAID could handle a hard shutdown is to fully serialize all writes, sacrificing all the performance gain from being built from multiple drives. There are some tricks that the more clever SW RAIDs, like the Linux MDRAID do, including a write intent map, to mitigate the damage and only rebuild a very small portion of the array. HW RAIDs handle hard shutdown gracefully. – Vojtech Nov 12 '15 at 9:37
  • Here's an important point to remember: RAID is not a backup solution. Repeat that 10 times so it sinks in. (I'm not saying you had that misconception - I just think it needs to be repeated.) The purpose of RAID is two-fold. 1. Increased performance. 2. Protection against a disk failure (fault tolerance). When you have a RAID setup, you still need to perform backups. Like Vojtech said: if you have an old RAID card that isn't easily replaceable, you could be dead in the water without a backup. – Charles Burge Dec 28 '16 at 21:03
0

Your question is about HW raid recovery, so you look at the features that the HW raid card / controller provides for you.

  1. a BIOS, which you can get into before the OS boots. Configuration can be done here.
  2. A processor or other "accellerator" that takes the work of managing the raid, and deciding where read/writes are directed.
  3. A scheme for identifying which physical disk belongs to what part of the RAID.
  4. A way to inform you about the status of the RAID.

Your RAID card will come with drivers and software that is supported under your OS that allows you to do configuration and status management without having to reboot into that BIOS.

For server hardware, the drives themselves are on hot-swap sleds with additional LEDs. These provide the physical feedback to you if the controller detects a problem with the RAID.

For desktop computers, your drives are hardwired to the RAID card or motherboard.

The driver and the software in your OS may also detect this and alert you some other way (email, etc).

Once a drive has gone bad, the controller stops reading and writing to it, relying on the remaining drives to serve the data. This is called a degraded state, you are still working, but one more failure pushes you over the edge into data loss. (2 more failures if you're running RAID 6)

For a hot-swapable raid controller, you simply pull out the failed drive, and put in a blank one.

How does it know it is blank? That's the job of the (3) scheme. All drives have unused data at the beginning, the partition table has plenty of free space in it. Each manufacturer will use it differently, but this is where the raid controller will store data that tells it which drive belongs to which part of the raid.

Once it sees a new drive that has not been used by this raid card before, it can start the restore process.

This can be automatic or triggered by the user, and can of course completely obliterate the new drive's contents if it was already formatted for something else.

The recovery or rebuild is managed in the background by the raid controller, it reads each sector from the remaining drives and calculates what should be on each sector for the new drive. For RAID 1, it simply copies accross all sectors of the existing good drive to the new drive. For RAID 5 or 6, all the existing drives are read, and the data to write to the new drive can be calculated. Since this work accesses the remaining drives, you can usually set a priority for it, so it doesn't slow down the whole system.

But you'd need to consider whether speed to recover back to full RAID status is more important than ongoing work. Some controllers, e.g. ones built into a regular desktop motherboard, might require you to go into the BIOS and trigger the rebuild there, and not let you boot into the OS until it's ready again. That would be an inconvenience, and would not be a good HW Raid because you want uptime as well as reslilience to failure.

A standalone HW raid card will give you the convenience of a rebuild that doesn't affect your ability to continue working.

If the failure is in the RAID card itself: The computer/server will most likely have crashed hard and is not bootable. At this point you might assume that the drives themselves are still viable, but it is more likely that the drives are in an inconsitent state, i.e. writes to one drive were not fully propagated to other drives. You are at the mercy of the Operating System and its File System error recovery for this. Worst case here is that you need to recover data from a backup as well after repairing the computer/server. If the RAID card is replaceable, an identical model can be installed in its place. Because the individual drives still report the same Identification in a manner that the RAID card recognizes, the full set of drives will work as before without complete loss of data (though File System inconsistencies may be present). If the RAID controller was part of a motherboard, the whole motherboard must be replaced with one with the same model RAID controller. If you attempt to use a different brand of RAID controller, most likely it won't recognize the drives at all, and just ask you how you want them newly configured, which will erase all existing data.

In HP Servers, all the various models of built in and plug-in RAID controllers share the same scheme for disk identification, so replacement of a built in controller with a plug-in, or a plug-in with a more powerful plug-in model is possible without loss of data.

In either case, care must also be taken that the replacement RAID card has its firmware updated to the same or newer version than the one being replaced.

Again with HP servers, I have had a server die, then I pull the whole set of drives in a RAID, and plug them into empty slots in a new server (already powered on), and the data is visible straight away.

  • 1
    This doesn't explain how data is recovered from a RAID array – Ramhound Dec 28 '16 at 20:17
  • I agree with Ramhound. In your answer you covered the case of a faulty disk being replaced. The question is rather: what to do with a full set of healthy disks when the controller dies? – Kamil Maciorowski Dec 29 '16 at 22:03
  • Updated with scenario for card failure. – Tomuo Dec 29 '16 at 23:30
0

With RAID 5 you always have a "parity volume". This is something of a misnomer as parity data is actually distributed among the drives, but it's still the case that you have one more drive than the number you would expect to need for your intended capacity. Call it n for the number of drives that you would need without redundancy, so n+1 is the number your raid 5 array will take.

The RAID 5 concept is that any n drives out of the set will always contain enough data to reconstruct a failed drive's contents.

The way parity lets this happen is similar to the concept of a "batch total" in accounting data. If I have a bunch of transactions to enter into a system, I may well compute a total for the batch before passing it to data entry. The data entry program requires that the computed batch total be entered, and then all of the transactions. The program computes the sum of the transactions and compares this to the batch total that I computed separately. If they're different, something's wrong. So the data entry person checks over the details.

To extend this to RAID 5, imagine that we have a way to know if any of the transactions - or the batch total, for that matter - was entered wrongly (or is unreadable). If only one input is wrong, then we can reconstruct that input with simple arithemetic: Subtract all the numbers that we do have from the total, and there's the missing number.

Fortunately for us, it's very difficult for a bad sector on a hard drive to be "read" without any error indication.

To understand further how the parity volume works, let's think about just one pair of bits in your "end user" data. Say, the first two bits in a sector as presented by the RAID 5 volume. We store one of those bits on drive A, and the other on drive B. On drive C, which is the "parity volume" for these particular bits, we store a bit that is the "exclusive OR" of the other two bits. The "exclusive OR" function is simply the sum of the bits, with carry ignored.

Another way to put it is that the XOR of any number of bits will be 1 if the number of "1" bits in the input is odd, and will be 0 if the number of "1" bits in the input is even. The value of this "XOR" function is what's stored in the "parity volume". Here's what this gives us for two input bits:

    A B   XOR(A,B)
   ----------------
    0 0      0
    0 1      1
    1 0      1
    1 1      0

Now, if we lose one of the bits - any one of the three, even the XOR - we can reconstruct it from the data we do have, as long as we know which bit is wrong - or missing.

And I will state again: the error correction and checking within hard drives is very good about this. It's set up so that, although corrected errors happen routinely, errors that are both uncorrectable and undetected are very, very rare. In the middle we have uncorrectable but detected errors, and we also have the case where the drive just fails and won't read anything (or the drive is missing completely). These "middle" cases are far more common than uncorrectable+undectable errors, and these are the cases RAID 5 protects against. We can reconstruct the missing or bad data from the data we have, simply by computing the XOR - the parity - of the bits that the drive tells us are still readable and good.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.