How do I reactivate my MDADM RAID5 array?

Question

I've just moved house which involved dismantling my server and re-connecting it. Since doing so, one of my MDADM RAID5 arrays is appearing as inactive:

root@mserver:/tmp# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
md1 : active raid5 sdc1[1] sdh1[2] sdg1[0]
      3907023872 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

md0 : inactive sdd1[0](S) sdf1[3](S) sde1[2](S) sdb1[1](S)
      3907039744 blocks

unused devices: <none>

It looks to me as though it's found all of the disks but for some reason doesn't want to use them.

So what do the (S) labels mean and how can I tell MDADM to start using the array again?

[Edit] I just tried stopping and assembling the array with -v:

root@mserver:~# mdadm --stop /dev/md0
mdadm: stopped /dev/md0

root@mserver:~# mdadm --assemble --scan -v
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 1.
mdadm: added /dev/sdd1 to /dev/md0 as 0 (possibly out of date)
mdadm: added /dev/sdb1 to /dev/md0 as 1 (possibly out of date)
mdadm: added /dev/sdf1 to /dev/md0 as 3 (possibly out of date)
mdadm: added /dev/sde1 to /dev/md0 as 2
mdadm: /dev/md0 assembled from 1 drive - not enough to start the array.

..and entering cat /proc/mdstat looks no different.

[Edit2] Not sure if it helps but this is the result of examining each disk:

root@mserver:~# mdadm --examine /dev/sdb1

/dev/sdb1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 2f331560:fc85feff:5457a8c1:6e047c67 (local to host mserver)
  Creation Time : Sun Feb  1 20:53:39 2009
     Raid Level : raid5
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 2930279808 (2794.53 GiB 3000.61 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Sat Apr 20 13:22:27 2013
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 6c8f71a3 - correct
         Events : 955190

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       17        1      active sync   /dev/sdb1

   0     0       8      113        0      active sync   /dev/sdh1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       97        2      active sync   /dev/sdg1
   3     3       8       33        3      active sync   /dev/sdc1

root@mserver:~# mdadm --examine /dev/sdd1

/dev/sdd1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 2f331560:fc85feff:5457a8c1:6e047c67 (local to host mserver)
  Creation Time : Sun Feb  1 20:53:39 2009
     Raid Level : raid5
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 2930279808 (2794.53 GiB 3000.61 GB)
   Raid Devices : 4
  Total Devices : 2
Preferred Minor : 0

    Update Time : Sat Apr 20 18:37:23 2013
          State : active
 Active Devices : 2
Working Devices : 2
 Failed Devices : 2
  Spare Devices : 0
       Checksum : 6c812869 - correct
         Events : 955205

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8      113        0      active sync   /dev/sdh1

   0     0       8      113        0      active sync   /dev/sdh1
   1     1       0        0        1      faulty removed
   2     2       8       97        2      active sync   /dev/sdg1
   3     3       0        0        3      faulty removed

root@mserver:~# mdadm --examine /dev/sde1

/dev/sde1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 2f331560:fc85feff:5457a8c1:6e047c67 (local to host mserver)
  Creation Time : Sun Feb  1 20:53:39 2009
     Raid Level : raid5
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 2930279808 (2794.53 GiB 3000.61 GB)
   Raid Devices : 4
  Total Devices : 2
Preferred Minor : 0

    Update Time : Sun Apr 21 14:00:43 2013
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 2
  Spare Devices : 0
       Checksum : 6c90cc70 - correct
         Events : 955219

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8       97        2      active sync   /dev/sdg1

   0     0       0        0        0      removed
   1     1       0        0        1      faulty removed
   2     2       8       97        2      active sync   /dev/sdg1
   3     3       0        0        3      faulty removed

root@mserver:~# mdadm --examine /dev/sdf1

/dev/sdf1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 2f331560:fc85feff:5457a8c1:6e047c67 (local to host mserver)
  Creation Time : Sun Feb  1 20:53:39 2009
     Raid Level : raid5
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 2930279808 (2794.53 GiB 3000.61 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Sat Apr 20 13:22:27 2013
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 6c8f71b7 - correct
         Events : 955190

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8       33        3      active sync   /dev/sdc1

   0     0       8      113        0      active sync   /dev/sdh1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       97        2      active sync   /dev/sdg1
   3     3       8       33        3      active sync   /dev/sdc1

I have some notes which suggest the drives were originally assembled as follows:

md0 : active raid5 sdb1[1] sdc1[3] sdh1[0] sdg1[2]
      2930279808 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

[Edit3]

Looking through the log it looks like the following happened (based on the Update Time in the --examine results):

sdb and sdf were knocked out some time after 13:22 on the 20th
sdd was knocked out some time after 18:37 on the 20th
the server was shut down some time after 14:00 on the 1st

Given that two disks went down (apparently) simultaneously I think it should be reasonably safe to assume the array wouldn't have been written to after that point(?) and so it should be relatively safe to force it to re-instate in the correct order? What's the safest command to do that with and is there a way to do it without writing any changes?

Community · Accepted Answer · 2017-03-20 10:17:03Z

38

The S labels means the disk is regarded as "spare". You should try stopping and re-starting the array:

  mdadm --stop /dev/md0
  mdadm --assemble --scan

to re-assemble the array and if that doesn't work, you may need to update your mdadm.conf, see for example this question for details on how to do that.

edited Mar 20, 2017 at 10:17

CommunityBot

1

answered Jun 4, 2013 at 9:14

Stefan Seidel

10.7k1 gold badge30 silver badges44 bronze badges

Tried that (and added -v to see what was going on) but all the disks which should be added get the responses along the following lines: mdadm: /dev/sdb1 is busy - skipping.
– Jon Cage
Commented Jun 4, 2013 at 9:33
just stop md0 and re-assemble the array
– krizna
Commented Jun 4, 2013 at 9:56
tried that - still no luck (see my edit)
– Jon Cage
Commented Jun 4, 2013 at 10:38
3

Ok, it looks like it thinks the RAID wasn't shut down properly, if you are sure it wasn't, try -R or -f. If that fails, too, re-create the array using mdadm create /dev/md0 --assume-clean <original create options> /dev/sd[dbfe]1. Be warned: All of these options may destroy your data.
– Stefan Seidel
Commented Jun 4, 2013 at 11:04
5

Well I went for it and mdadm --assemble --scan --force worked. The array is back up and running and I have access to my data :)
– Jon Cage
Commented Jun 5, 2013 at 11:41

| Show 4 more comments

Milen · Accepted Answer · 2014-09-13 01:33:49Z

This question is a bit old, but the answer might help someone facing a similar situation. Looking at the event counts from the mdadm --examine output you have provided, they seem close enough (955190 - for sdb1 and sdf1, 955219 for sde1 and for sdd1 you've got 955205). If they are below 40-50, this is OK, and in that case the recommended course of action is to assemble your array manually, forcing mdadm to accept the drives despite the event count difference:

Stop the array:

mdadm --stop /dev/md0

Then try to reassemble the array manually:

mdadm --assemble --force /dev/md0 /dev/sdb1 /dev/sdd1 /dev/sde1 /dev/sdf1

Check the status of the array, to examine if the drive list/structure is OK (bottom of command output will show which drive is at what status and at which position in the array):

mdadm --detail /dev/md0

If the structure is OK, check rebuilding progress:

cat /proc/mdstat

--assemble --force is what I needed to get an array back online today. — Nathan V, Commented Apr 13, 2020 at 0:30

krizna · Accepted Answer · 2013-06-04 09:02:31Z

0

You can activate Raid md0 with the below command

mdadm -A /dev/md0

and this command to update mdadm.conf file

mdadm --examine --scan >> /etc/mdadm/mdadm.conf

answered Jun 4, 2013 at 9:02

krizna

3711 silver badge4 bronze badges

Add a comment |

bbx · Accepted Answer · 2024-07-30 21:22:25Z

Late to the party, but since we needed a different solution... System:

kernel 5.10
raid5

our setup

our_server:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md2 : inactive sdf1[3](S) sde1[1](S) sdd1[0](S)
      11720349142 blocks super 1.2

our_server:~# mdadm --detail /dev/md2
/dev/md2:
           Version : 1.2
        Raid Level : raid5
     Total Devices : 3
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 3

              Name : our_server:2  (local to host our_server)
              UUID : 1aecccd4:7f6838d3:37bf6446:7d9e81ec
            Events : 41287

    Number   Major   Minor   RaidDevice

       -       8       81        -        /dev/sdf1
       -       8       65        -        /dev/sde1
       -       8       49        -        /dev/sdd1


our_server:~# mdadm --examine /dev/sd[def]1 | grep Event
         Events : 41292
         Events : 36092
         Events : 41292

We had two good devices sdd and sdf but could not easily re-run our array

our_server:~# mdadm --run /dev/md2
mdadm: failed to start array /dev/md2: Input/output error

Others solution did not work either:

our_server:~# mdadm --assemble --force /dev/md2 /dev/sdf1 /dev/sdd1
mdadm: forcing event count in /dev/sdf1(2) from 41287 up to 41292
mdadm: clearing FAULTY flag for device 0 in /dev/md2 for /dev/sdf1
mdadm: Marking array /dev/md2 as 'clean'
mdadm: /dev/md2 assembled from 2 drives - not enough to start the array.

our_server:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md2 : inactive sdd1[0](S) sdf1[3](S)
      7813566095 blocks super 1.2

our_server:~# cat /sys/block/md2/md/array_state
inactive

our_server:~# echo "clean" > /sys/block/md2/md/array_state
-bash: echo: write error: Invalid argument

neither:

our_server:~# mdadm --stop /dev/md2
mdadm: stopped /dev/md2
our_server:~# mdadm --assemble --scan
mdadm: /dev/md2 assembled from 1 drive - not enough to start the array.

our_server:~# cat /sys/block/md2/md/array_state
inactive

our_server:~# echo "clean" > /sys/block/md2/md/array_state
-bash: echo: write error: Invalid argument

solution

our_server:~# mdadm --assemble --force --run /dev/md2 /dev/sdf1 /dev/sdd1
mdadm: Fail to create md2 when using /sys/module/md_mod/parameters/new_array, fallback to creation via node
mdadm: /dev/md2 has been started with 2 drives (out of 3).
our_server:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md2 : active (auto-read-only) raid5 sdd1[0] sdf1[3]
      7813565440 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [U_U]
      bitmap: 21/30 pages [84KB], 65536KB chunk


our_server:~# cat /sys/devices/virtual/block/md2/md/array_state
read-auto

our_server:~# echo clean > /sys/devices/virtual/block/md2/md/array_state

our_server:~# mdadm --add /dev/md2 /dev/sde1
mdadm: re-added /dev/sde1

our_server:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md2 : active raid5 sde1[1] sdd1[0] sdf1[3]
      7813565440 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [U_U]
      [>....................]  recovery =  2.8% (110505584/3906782720) finish=445.1min speed=142132K/sec
      bitmap: 21/30 pages [84KB], 65536KB chunk

Stack Exchange Network

How do I reactivate my MDADM RAID5 array?

4 Answers 4

solution

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
linux
ubuntu-12.04
software-raid
raid-5
mdadm
.

Linked

Hot Network Questions

How do I reactivate my MDADM RAID5 array?

4 Answers 4

solution

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged linuxubuntu-12.04software-raidraid-5mdadm.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
linux
ubuntu-12.04
software-raid
raid-5
mdadm
.