Fixing a broken Raid-Array (Hetzner-Root-Server from Germany)

Question

I do not know why but my Hetzner Root Server loses once a quarter its RAID array and the situation is different every time. This time I need further help. Maybe they want that I change to managed-server, it is about 40% more expensive :) .

EDIT: Desired Result:

md3 : active raid1 sda4[2] sdb4[1]
      1822442815 blocks super 1.2 [2/2] [UU]

md2 : active raid1 sda3[2] sdb3[1]
      1073740664 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda2[2] sdb2[1]
      524276 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sda1[2] sdb1[1]
      33553336 blocks super 1.2 [2/2] [UU]

How can I fix the following raid array?

cat /proc/mdstat

Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md124 : active (auto-read-only) raid1 sda1[2]
      33553336 blocks super 1.2 [2/1] [U_]

md125 : active (auto-read-only) raid1 sda2[2]
      524276 blocks super 1.2 [2/1] [U_]

md126 : active (auto-read-only) raid1 sda3[2]
      1073740664 blocks super 1.2 [2/1] [U_]

md127 : active (auto-read-only) raid1 sda4[2]
      1822442815 blocks super 1.2 [2/1] [U_]

md3 : active (auto-read-only) raid1 sdb4[1]
      1822442815 blocks super 1.2 [2/1] [_U]

md2 : active raid1 sdb3[1]
      1073740664 blocks super 1.2 [2/1] [_U]

md1 : active (auto-read-only) raid1 sdb2[1]
      524276 blocks super 1.2 [2/1] [_U]

md0 : active (auto-read-only) raid1 sdb1[1]
      33553336 blocks super 1.2 [2/1] [_U]

unused devices: <none>

Here are the details:

 /dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 79ca4cbd:5d44fcad:01e8ed8e:0bd7009a
           Name : rescue:0
  Creation Time : Mon Aug 20 11:23:55 2012
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 67106816 (32.00 GiB 34.36 GB)
     Array Size : 67106672 (32.00 GiB 34.36 GB)
  Used Dev Size : 67106672 (32.00 GiB 34.36 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : facee938:5e51285d:e49d35a7:4e3ae212

    Update Time : Sun Jan 17 02:23:41 2016
       Checksum : cf49c9d3 - correct
         Events : 504


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)
/dev/sda2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 4cdff7b7:2ec9bae4:8c9cbf02:67bfe971
           Name : rescue:1
  Creation Time : Mon Aug 20 11:23:55 2012
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 1048552 (512.07 MiB 536.86 MB)
     Array Size : 1048552 (512.07 MiB 536.86 MB)
    Data Offset : 24 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 181560d1:abc6d39b:3bd45252:6c5bff30

    Update Time : Sat Jan 23 06:48:30 2016
       Checksum : e5f248df - correct
         Events : 2064


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)
/dev/sda3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 009a5d9b:7a0f238e:3ede62a0:0d2ee0ba
           Name : rescue:2
  Creation Time : Mon Aug 20 11:23:56 2012
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 2147481600 (1024.00 GiB 1099.51 GB)
     Array Size : 2147481328 (1024.00 GiB 1099.51 GB)
  Used Dev Size : 2147481328 (1024.00 GiB 1099.51 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 106e6b66:9365ada0:b0ee61bf:15cf9585

    Update Time : Sat Jan 23 11:20:33 2016
       Checksum : b62dfda7 - correct
         Events : 6901428


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)
/dev/sda4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : aa151e0d:2430bfba:a79d7030:d56a7872
           Name : rescue:3
  Creation Time : Mon Aug 20 11:23:56 2012
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 3644885903 (1738.02 GiB 1866.18 GB)
     Array Size : 3644885630 (1738.02 GiB 1866.18 GB)
  Used Dev Size : 3644885630 (1738.02 GiB 1866.18 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 1a883ec6:768c6884:f8465824:69bddd2e

    Update Time : Sat Jan 23 06:48:30 2016
       Checksum : a114be68 - correct
         Events : 2062


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 79ca4cbd:5d44fcad:01e8ed8e:0bd7009a
           Name : rescue:0
  Creation Time : Mon Aug 20 11:23:55 2012
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 67106816 (32.00 GiB 34.36 GB)
     Array Size : 67106672 (32.00 GiB 34.36 GB)
  Used Dev Size : 67106672 (32.00 GiB 34.36 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 2df660db:6eaab24e:be6a2b7e:6295cc6f

    Update Time : Sat Jan 23 11:20:53 2016
       Checksum : 9734d8ec - correct
         Events : 506


   Device Role : Active device 1
   Array State : .A ('A' == active, '.' == missing)
/dev/sdb2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 4cdff7b7:2ec9bae4:8c9cbf02:67bfe971
           Name : rescue:1
  Creation Time : Mon Aug 20 11:23:55 2012
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 1048552 (512.07 MiB 536.86 MB)
     Array Size : 1048552 (512.07 MiB 536.86 MB)
    Data Offset : 24 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 2594f11b:7e7289b6:8bff6aff:10bf1b05

    Update Time : Mon Jan 25 06:46:16 2016
       Checksum : cc71a538 - correct
         Events : 2078


   Device Role : Active device 1
   Array State : .A ('A' == active, '.' == missing)
/dev/sdb3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 009a5d9b:7a0f238e:3ede62a0:0d2ee0ba
           Name : rescue:2
  Creation Time : Mon Aug 20 11:23:56 2012
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 2147481600 (1024.00 GiB 1099.51 GB)
     Array Size : 2147481328 (1024.00 GiB 1099.51 GB)
  Used Dev Size : 2147481328 (1024.00 GiB 1099.51 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : b838fbbb:1dafe023:afce822f:45c7ba0d

    Update Time : Tue Jan 26 09:27:55 2016
       Checksum : 873de764 - correct
         Events : 7041530


   Device Role : Active device 1
   Array State : .A ('A' == active, '.' == missing)
/dev/sdb4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : aa151e0d:2430bfba:a79d7030:d56a7872
           Name : rescue:3
  Creation Time : Mon Aug 20 11:23:56 2012
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 3644885903 (1738.02 GiB 1866.18 GB)
     Array Size : 3644885630 (1738.02 GiB 1866.18 GB)
  Used Dev Size : 3644885630 (1738.02 GiB 1866.18 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 8e56f14c:a1478ce7:59c6ba88:09d18a60

    Update Time : Mon Jan 25 06:46:11 2016
       Checksum : 20fe7d89 - correct
         Events : 2076


   Device Role : Active device 1
   Array State : .A ('A' == active, '.' == missing)

And more Details:

/dev/md0:
        Version : 1.2
  Creation Time : Mon Aug 20 11:23:55 2012
     Raid Level : raid1
     Array Size : 33553336 (32.00 GiB 34.36 GB)
  Used Dev Size : 33553336 (32.00 GiB 34.36 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Tue Jan 26 09:38:15 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:0
           UUID : 79ca4cbd:5d44fcad:01e8ed8e:0bd7009a
         Events : 508

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       17        1      active sync   /dev/sdb1
/dev/md1:
        Version : 1.2
  Creation Time : Mon Aug 20 11:23:55 2012
     Raid Level : raid1
     Array Size : 524276 (512.07 MiB 536.86 MB)
  Used Dev Size : 524276 (512.07 MiB 536.86 MB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Mon Jan 25 06:46:16 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:1
           UUID : 4cdff7b7:2ec9bae4:8c9cbf02:67bfe971
         Events : 2078

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       18        1      active sync   /dev/sdb2
/dev/md2:
        Version : 1.2
  Creation Time : Mon Aug 20 11:23:56 2012
     Raid Level : raid1
     Array Size : 1073740664 (1024.00 GiB 1099.51 GB)
  Used Dev Size : 1073740664 (1024.00 GiB 1099.51 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Tue Jan 26 09:42:42 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:2
           UUID : 009a5d9b:7a0f238e:3ede62a0:0d2ee0ba
         Events : 7042054

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       19        1      active sync   /dev/sdb3
/dev/md3:
        Version : 1.2
  Creation Time : Mon Aug 20 11:23:56 2012
     Raid Level : raid1
     Array Size : 1822442815 (1738.02 GiB 1866.18 GB)
  Used Dev Size : 1822442815 (1738.02 GiB 1866.18 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Mon Jan 25 06:46:11 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:3
           UUID : aa151e0d:2430bfba:a79d7030:d56a7872
         Events : 2076

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       20        1      active sync   /dev/sdb4
/dev/md124:
        Version : 1.2
  Creation Time : Mon Aug 20 11:23:55 2012
     Raid Level : raid1
     Array Size : 33553336 (32.00 GiB 34.36 GB)
  Used Dev Size : 33553336 (32.00 GiB 34.36 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Sun Jan 17 02:23:41 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:0
           UUID : 79ca4cbd:5d44fcad:01e8ed8e:0bd7009a
         Events : 504

    Number   Major   Minor   RaidDevice State
       2       8        1        0      active sync   /dev/sda1
       1       0        0        1      removed
/dev/md125:
        Version : 1.2
  Creation Time : Mon Aug 20 11:23:55 2012
     Raid Level : raid1
     Array Size : 524276 (512.07 MiB 536.86 MB)
  Used Dev Size : 524276 (512.07 MiB 536.86 MB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Sat Jan 23 06:48:30 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:1
           UUID : 4cdff7b7:2ec9bae4:8c9cbf02:67bfe971
         Events : 2064

    Number   Major   Minor   RaidDevice State
       2       8        2        0      active sync   /dev/sda2
       1       0        0        1      removed
/dev/md126:
        Version : 1.2
  Creation Time : Mon Aug 20 11:23:56 2012
     Raid Level : raid1
     Array Size : 1073740664 (1024.00 GiB 1099.51 GB)
  Used Dev Size : 1073740664 (1024.00 GiB 1099.51 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Sat Jan 23 11:20:33 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:2
           UUID : 009a5d9b:7a0f238e:3ede62a0:0d2ee0ba
         Events : 6901428

    Number   Major   Minor   RaidDevice State
       2       8        3        0      active sync   /dev/sda3
       1       0        0        1      removed
/dev/md127:
        Version : 1.2
  Creation Time : Mon Aug 20 11:23:56 2012
     Raid Level : raid1
     Array Size : 1822442815 (1738.02 GiB 1866.18 GB)
  Used Dev Size : 1822442815 (1738.02 GiB 1866.18 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Sat Jan 23 06:48:30 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:3
           UUID : aa151e0d:2430bfba:a79d7030:d56a7872
         Events : 2062

    Number   Major   Minor   RaidDevice State
       2       8        4        0      active sync   /dev/sda4
       1       0        0        1      removed

What is the problem? What RAID array are you unable to mount? Are you getting an error? It does look like you are missing a disk, is that expected? — Clarus, Jan 29, 2016 at 18:48
Thank you for your feedback (first one ever). I've edited the question und added the desired (probably the correct) result. — user532552, Jan 29, 2016 at 18:57
Thank you but I've already seen that, that's an another configuration (I think) and the solution is not explained clear enough ( to find out which of the two devices has the more recent copy -> how? mount them read-only -> how? Then kill the other raid -> how) and add the device to the correct one -> how? ) — user532552, Jan 29, 2016 at 19:03
It appears that you have two drives (sda and sdb), each with 4 partitions (1,2,3,4). I think the idea is to mirror those partitions. Instead, you have 4 RAID partitions connected to the sda partitions (md0, md1, md2 and md3) with their mirror's removed. Also, you have 4 RAID partitions connected to the sdb partitions (md124, md125, md126 and md127). To help you, we need to understand: 1. what is the role/mount point of each partiton (guess: root, boot, swap, home)? And 2. how did this situation appear? For example, can we assume md0 and md124 have the same data? — agtoever, Jan 30, 2016 at 10:32

S.Haran · Accepted Answer · 2016-01-31 17:16:07Z

RAID member 1 partitions on /dev/sdb are the most current as shown here...

a1
Array UUID : 79ca4cbd:5d44fcad:01e8ed8e:0bd7009a
Events : 504
Update Time : Sun Jan 17 02:23:41 2016
Device Role : Active device 0

b1
Array UUID : 79ca4cbd:5d44fcad:01e8ed8e:0bd7009a
Events : 506
Update Time : Sat Jan 23 11:20:53 2016
Device Role : Active device 1

a2
Array UUID : 4cdff7b7:2ec9bae4:8c9cbf02:67bfe971
Events : 2064
Update Time : Sat Jan 23 06:48:30 2016
Device Role : Active device 0

b2
Array UUID : 4cdff7b7:2ec9bae4:8c9cbf02:67bfe971
Events : 2078
Update Time : Mon Jan 25 06:46:16 2016
Device Role : Active device 1

a3
Array UUID : 009a5d9b:7a0f238e:3ede62a0:0d2ee0ba
Events : 6901428
Update Time : Sat Jan 23 11:20:33 2016
Device Role : Active device 0

b3
Array UUID : 009a5d9b:7a0f238e:3ede62a0:0d2ee0ba
Events : 7041530
Update Time : Tue Jan 26 09:27:55 2016
Device Role : Active device 1

a4
Array UUID : aa151e0d:2430bfba:a79d7030:d56a7872
Events : 2062
Update Time : Sat Jan 23 06:48:30 2016
Device Role : Active device 0

b4
Array UUID : aa151e0d:2430bfba:a79d7030:d56a7872
Events : 2076
Update Time : Mon Jan 25 06:46:11 2016
Device Role : Active device 1

So to get to your desired result first stop the spurious md12X arrays....

mdadm --stop /dev/md124
mdadm --stop /dev/md125
mdadm --stop /dev/md126
mdadm --stop /dev/md127

Then just add each removed RAID Member partition back into it's original RAID1

mdadm --add /dev/md0 /dev/sda1
mdadm --add /dev/md1 /dev/sda2
mdadm --add /dev/md2 /dev/sda3
mdadm --add /dev/md3 /dev/sda4

I think what happened is that at some point mdadm could not see sda, So it removed all the RAID member 0 partitions. Then when mdadm sees sda appear again it creates the md12X arrays with the out-of-sync sda partitions. So the question is why is mdadam at times is not seeing sda? Could be a power issue, a cable problem, or a drive problem. Check the SMART status on the drives. — S.Haran, Feb 1, 2016 at 14:49
There were a failure md2 : active raid1 sda3[2](F) sdb3[1] 1073740664 blocks super 1.2 [2/1] [_U] how could I fix that? Thank you again,.. — user532552, Feb 5, 2016 at 16:03
I suspect you have a problem with /dev/sda. The fix will be similar as last time but to be sure show the output of... cat /proc/mdstat — S.Haran, Feb 5, 2016 at 22:13

Stack Exchange Network

Fixing a broken Raid-Array (Hetzner-Root-Server from Germany)

1 Answer 1

You must log in to answer this question.

Hot Network Questions

Fixing a broken Raid-Array (Hetzner-Root-Server from Germany)

1 Answer 1

You must log in to answer this question.

Related

Hot Network Questions