0

I have received two notifications from our main server today about failed disks in a raid array. This appears to be a software raid setup years before I started at the company by a past staff member.

I have never setup or managed raid on linux before, so before I start making any changes or running commands I am hoping to better understand what is going on and how to potentially fix it.

The two notifications I have received are:

A Fail event had been detected on md device /dev/md/1.

It could be related to component device /dev/sda1.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1] md0 : active raid1 sdb2[1] sda20 480192320 blocks super 1.0 [2/1] [_U] bitmap: 4/4 pages [16KB], 65536KB chunk

md1 : active raid1 sdb1[1] sda10 8188864 blocks super 1.2 [2/1] [_U]

unused devices:

and:

A Fail event had been detected on md device /dev/md/0.

It could be related to component device /dev/sda2.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1] md0 : active raid1 sdb2[1] sda20 480192320 blocks super 1.0 [2/1] [_U] bitmap: 3/4 pages [12KB], 65536KB chunk

md1 : active raid1 sdb1[1] sda1[0] 8188864 blocks super 1.2 [2/2] [UU]

unused devices:

I know that there are two physical disks in the server both 500gb/465gb usable. To me these notifications suggest that one disk is md0 split into two partitions, and the other is md1 also split into two partitions.

Is that correct? Or is this actually showing that one of the physical disks has failed and needs replacing?

0

To me these notifications suggest that one disk is md0 split into two partitions, and the other is md1 also split into two partitions.

It's the opposite. The physical disks are 'sda' and 'sdb' – the RAID arrays are 'md0' and 'md1'.

Each disk is indeed split into two partitions (sda1, sda2, sdb1, sdb2) and each pair of partitions has a RAID array configured on it. If you run lsblk you'll probably see something like this:

NAME     SIZE  TYPE
sda      466G  disk
├─sda1     8G  part
│ └─md1    8G  raid1
└─sda2   458G  part
  └─md0  458G  raid1
sdb      466G  disk
├─sdb1     8G  part
│ └─md1    8G  raid1
└─sdb2   458G  part
  └─md0  458G  raid1

So the physical disk /dev/sda has failed, and it was acting as a member of both the md0 and md1 arrays. (Probably a small portion of the space was used for the '/boot' volume and the remainder for the rootfs.)

To find out which disk it is physically, use lsblk -S or lsblk -do name,tran,model,serial,wwn.

Replacing the disk can likely be done live, using mdadm --manage. (I think the exact commands depend on whether you first connect the new disk as 'sdc', or whether you directly replace the old disk and it remains 'sda'...)

|improve this answer|||||
0

The keys to resolving this issue are:

  md1 : active raid1 sdb1[1] sda10 8188864 blocks super 1.2 [2/1] [_U]

This is telling you that md1 (which represents the second raid) is supposed to be comprised of 2 partitions - sdb1 and sda10. For some reason sda1 has been dropped from the array, so it is now in a degraded state.

sda1 means the first partition on the first disk the OS recognised. The next step is to find out why this failed, and to fox or replace it.

I would run a command like "hdparm -I /dev/SDA" which should tell the make and model of the disk - so you know which disk you are dealing with. I would then look in the log file for errors relating to /dev/SDA. If you find these, its likely the disk needs to be replaced. If not, it is possible that the signature of on the disk saying its part of the array is missing. In this case it can be readded to the array using the mdadm command. You should also investigate why this happened.

If the disk is failing you need to (a) dump the partition info, (b) degrade md0 - ie remove /dev/sda2 (c) replace /dev/sda (d) partition the new sda with partitions no smaller then on the original sda (e) mark these as type FD = raid partitions (f) reads partitions using mdadm. You can watch the rebuild with "watch cat /proc/mdstat"

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.