How to get an inactive RAID device working again?

Question

After booting, my RAID1 device (/dev/md_d0 *) sometimes goes in some funny state and I cannot mount it.

^{* Originally I created /dev/md0 but it has somehow changed itself into /dev/md_d0.}

# mount /opt
mount: wrong fs type, bad option, bad superblock on /dev/md_d0,
       missing codepage or helper program, or other error
       (could this be the IDE device where you in fact use
       ide-scsi so that sr0 or sda or so is needed?)
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

The RAID device appears to be inactive somehow:

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] 
                [raid4] [raid10] 
md_d0 : inactive sda4[0](S)
      241095104 blocks

# mdadm --detail /dev/md_d0
mdadm: md device /dev/md_d0 does not appear to be active.

Question is, how to make the device active again (using mdmadm, I presume)?

(Other times it's alright (active) after boot, and I can mount it manually without problems. But it still won't mount automatically even though I have it in /etc/fstab:

/dev/md_d0        /opt           ext4    defaults        0       0

So a bonus question: what should I do to make the RAID device automatically mount at /opt at boot time?)

This is an Ubuntu 9.10 workstation. Background info about my RAID setup in this question.

Edit: My /etc/mdadm/mdadm.conf looks like this. I've never touched this file, at least by hand.

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR <my mail address>

# definitions of existing MD arrays

# This file was auto-generated on Wed, 27 Jan 2010 17:14:36 +0200

In /proc/partitions the last entry is md_d0 at least now, after reboot, when the device happens to be active again. (I'm not sure if it would be the same when it's inactive.)

Resolution: as Jimmy Hedman suggested, I took the output of mdadm --examine --scan:

ARRAY /dev/md0 level=raid1 num-devices=2 UUID=de8fbd92[...]

and added it in /etc/mdadm/mdadm.conf, which seems to have fixed the main problem. After changing /etc/fstab to use /dev/md0 again (instead of /dev/md_d0), the RAID device also gets automatically mounted!

Answer 1

For your bonus question:

mdadm --examine --scan >> /etc/mdadm/mdadm.conf

Answer 2

I have found that I have to add the array manually in /etc/mdadm/mdadm.conf in order to make Linux mount it on reboot. Otherwise I get exactly what you have here - md_d1-devices that are inactive etc.

The conf-file should look like below - i.e. one ARRAY-line for each md-device. In my case the new arrays were missing in this file, but if you have them listed this is probably not a fix to your problem.

# definitions of existing MD arrays
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=f10f5f96:106599e0:a2f56e56:f5d3ad6d
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=aa591bbe:bbbec94d:a2f56e56:f5d3ad6d

Add one array per md-device, and add them after the comment included above, or if no such comment exists, at the end of the file. You get the UUIDs by doing sudo mdadm -E --scan:

$ sudo mdadm -E --scan
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=f10f5f96:106599e0:a2f56e56:f5d3ad6d
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=aa591bbe:bbbec94d:a2f56e56:f5d3ad6d

As you can see you can pretty much just copy the output from the scan-result into the file.

I run ubuntu desktop 10.04 LTS, and as far as I remember this behavior differs from the server version of Ubuntu, however it was such a long time ago I created my md-devices on the server I may be wrong. It may also be that I just missed some option.

Anyway, adding the array in the conf-file seems to do the trick. I've run the above raid 1 and raid 5 for years with no problems.

Answer 3

You can activate your md device with

mdadm -A /dev/md_d0

I suppose some startup script starts too soon, before one of the RAID member was discovered or some similar problem. As a quick and dirty workaround, you should be able to add this line to /etc/rc.local :

mdadm -A /dev/md_d0 && mount /dev/md_d0

Edit : apparently your /etc/mdadm/mdadm.conf still contains the old configuration name. Edit this file and replace occurences of md0 with md_d0.

Answer 4

I had a similar problem... my server would not mount md2 after I had grown theassociated devices partitions. On reading this thread I found that the md2 RAID device had a new UUID and the machine was trying to use the old one.

As suggested... using 'md2' output from

mdadm --examine --scan

I edited /etc/mdadm/mdadm.conf and replaced the old UUID line with the one output from above command and my problem went away.

Answer 5

md_d0 : inactive sda4[0](S) looks wrong for a RAID1 array. It seems to suggest that the array has no active devices and one spare device ^{_{(indicated by the (S), you would see (F) there for a failed device and nothing for an OK/active device)}} - for a RAID1 array that isn't running degraded there should be at least two OK/active devices ^{_{(and for a degraded array, at least one OK/active device)}} and you can't activate a RAID1 array with no none-failed not-spare devices (as spares do not contain a copy of the data until they are made active when another drive fails). If I'm reading that /proc/mdstat output right, you'll not be able to activate the array in its current state.

Do you have any physical drives in the machine that have failed to spin-up? Does ls /dev/sd* list all the drives and partitions that you would normally expect to see on that machine?