1

My server lost power yesterday. It automatically booted up when power came back, but my RAID is now nonfunctional.

The RAID array is a software RAID6 built with mdadm. Originally, the array used 8 drives. Several months ago I noticed a drive failing SMART tests, so I replaced it. While replacing the drive, I added an additional 4 drives, which were originally in my previous server and formatted as ZFS (as a part of a FreeNAS RAIDZ), bringing the total drive count to 12.

Unfortunately I don't remember the exact command(s) I used to expand the array, and the command is no longer in my bash_history. I do not know why today mdadm is reporting 8 disks, and fdisk is reporting those 4 drives as being ZFS; I am confident that for the past several months I have been running a 12-disk RAID setup.

As far as I know, my server has not shut down or rebooted since replacing the bad drive and adding the 4 additional drives, until yesterday.

The only mdadm-related commands that I have run since the power loss are mdadm --detail /dev/md0, mdadm --stop /dev/md0, and mdadm --assemble --scan -v. I have not rebooted the server.

I realize that I need a backup and will implement one ASAP. For now, I'm just hoping there's a way to recover my data.

I'm running Debian 9 and mdadm 3.4.

I'm happy to provide any other information. (Please note that I don't consider myself a RAID or mdadm expert by any means.)

Update: I added the output of an mdadm examine for the "good"/"bad" drives. The "good" drives are correctly reporting raid6 and 12 drives.

mdadm --detail /dev/md0

(Note: this was at boot, before running mdadm --stop /dev/md0. Also note that it reports raid0.)

/dev/md0:
        Version : 1.2
     Raid Level : raid0
  Total Devices : 8
    Persistence : Superblock is persistent

          State : inactive

           Name : blackcanary:0  (local to host blackcanary)
           UUID : 7bd22c29:465d927f:d0c1d08f:ba3694c9
         Events : 72036

    Number   Major   Minor   RaidDevice

       -       8       64        -        /dev/sde
       -       8       32        -        /dev/sdc
       -       8        0        -        /dev/sda
       -       8      112        -        /dev/sdh
       -       8       80        -        /dev/sdf
       -       8       48        -        /dev/sdd
       -       8      128        -        /dev/sdi
       -       8       96        -        /dev/sdg

fdisk -l

(Note: /dev/sdb and /dev/sdj are the only two drives that should NOT be in the array.)

Disk /dev/sdd: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sde: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdc: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdf: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdg: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdh: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdi: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdb: 57.9 GiB, 62109253632 bytes, 121307136 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 764C7B0B-574D-40DF-95C0-06D765AEB2D2

Device        Start       End  Sectors  Size Type
/dev/sdb1      2048   1050623  1048576  512M EFI System
/dev/sdb2   1050624  54255615 53204992 25.4G Linux filesystem
/dev/sdb3  54255616 121305087 67049472   32G Linux swap


Disk /dev/sdj: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdk: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 41DFFB5A-7F8C-11E5-9D78-7824AF43D5FA

Device       Start        End    Sectors  Size Type
/dev/sdk1      128    4194431    4194304    2G FreeBSD swap
/dev/sdk2  4194432 7814037127 7809842696  3.7T FreeBSD ZFS


Disk /dev/sdl: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 40C2CEAE-7F8C-11E5-9D78-7824AF43D5FA

Device       Start        End    Sectors  Size Type
/dev/sdl1      128    4194431    4194304    2G FreeBSD swap
/dev/sdl2  4194432 7814037127 7809842696  3.7T FreeBSD ZFS


Disk /dev/sdm: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 402A1394-7F8C-11E5-9D78-7824AF43D5FA

Device       Start        End    Sectors  Size Type
/dev/sdm1      128    4194431    4194304    2G FreeBSD swap
/dev/sdm2  4194432 7814037127 7809842696  3.7T FreeBSD ZFS


Disk /dev/sdn: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 415081DC-7F8C-11E5-9D78-7824AF43D5FA

Device       Start        End    Sectors  Size Type
/dev/sdn1      128    4194431    4194304    2G FreeBSD swap
/dev/sdn2  4194432 7814037127 7809842696  3.7T FreeBSD ZFS

blkid

/dev/sdd: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="91d8e433-aeb9-a4f2-75ca-10cf3ceb0a83" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sde: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="6ccd4cee-9fdd-872c-add0-209a9d074eb5" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sdc: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="11bf8212-7c04-0c8c-a061-168bda9c34a5" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sda: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="6cee76b0-031c-bb6a-6c13-7d15d0b2feee" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sdf: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="508cd116-e0ac-131c-cd0d-3c5c41c1cbba" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sdg: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="0080d32e-7fea-037f-75a8-6e40dfa8fdfa" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sdh: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="f1aa862e-6c4c-0239-5fb4-29d7a9dd7497" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sdi: UUID="7bd22c29-465d-927f-d0c1-d08fba3694c9" UUID_SUB="32472d1f-25c1-dfb3-fa9a-57ab3df17986" LABEL="blackcanary:0" TYPE="linux_raid_member"
/dev/sdb1: UUID="E15C-D029" TYPE="vfat" PARTUUID="dff9fca6-2eef-4084-845c-d780ca7b6cb8"
/dev/sdb2: UUID="bfd3d30c-de34-4e00-89b4-384bcbb7922d" TYPE="ext4" PARTUUID="7696f182-551e-447c-9549-155605cc48a3"
/dev/sdb3: UUID="7f65b504-ffe2-48e4-868f-a9db95865505" TYPE="swap" PARTUUID="48ecdc7c-fb0a-404e-bc02-178063515b6c"
/dev/sdj: UUID="b59275af-0825-4f25-96d2-aff0c3fef5e5" TYPE="ext4"
/dev/sdk1: PARTUUID="41f3ae62-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdk2: LABEL="main" UUID="2939752790805872810" UUID_SUB="11544886583175140825" TYPE="zfs_member" PARTUUID="420ad019-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdl1: PARTUUID="40d655e5-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdl2: LABEL="main" UUID="2939752790805872810" UUID_SUB="15587016790013084755" TYPE="zfs_member" PARTUUID="40ef0474-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdm1: PARTUUID="403ea9c7-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdm2: LABEL="main" UUID="2939752790805872810" UUID_SUB="12390459856885165202" TYPE="zfs_member" PARTUUID="4057cbaa-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdn1: PARTUUID="416507e6-7f8c-11e5-9d78-7824af43d5fa"
/dev/sdn2: LABEL="main" UUID="2939752790805872810" UUID_SUB="16896032374271545514" TYPE="zfs_member" PARTUUID="417ef281-7f8c-11e5-9d78-7824af43d5fa"

mdadm --asemble --scan -v

mdadm: looking for devices for /dev/md0
mdadm: No super block found on /dev/sdn2 (Expected magic a92b4efc, got b1f5ae15)
mdadm: no RAID superblock on /dev/sdn2
mdadm: No super block found on /dev/sdn1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdn1
mdadm: No super block found on /dev/sdn (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdn
mdadm: No super block found on /dev/sdm2 (Expected magic a92b4efc, got fd87a884)
mdadm: no RAID superblock on /dev/sdm2
mdadm: No super block found on /dev/sdm1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdm1
mdadm: No super block found on /dev/sdm (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdm
mdadm: No super block found on /dev/sdl2 (Expected magic a92b4efc, got 946313cc)
mdadm: no RAID superblock on /dev/sdl2
mdadm: No super block found on /dev/sdl1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdl1
mdadm: No super block found on /dev/sdl (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdl
mdadm: No super block found on /dev/sdk2 (Expected magic a92b4efc, got 4de36623)
mdadm: no RAID superblock on /dev/sdk2
mdadm: No super block found on /dev/sdk1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdk1
mdadm: No super block found on /dev/sdk (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdk
mdadm: No super block found on /dev/sdj (Expected magic a92b4efc, got 0000043c)
mdadm: no RAID superblock on /dev/sdj
mdadm: No super block found on /dev/sdb3 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb3
mdadm: No super block found on /dev/sdb2 (Expected magic a92b4efc, got 00000405)
mdadm: no RAID superblock on /dev/sdb2
mdadm: No super block found on /dev/sdb1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb1
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
mdadm: /dev/sdi is identified as a member of /dev/md0, slot 7.
mdadm: /dev/sdh is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdg is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdf is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sda is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 1.
mdadm: added /dev/sdd to /dev/md0 as 1
mdadm: added /dev/sda to /dev/md0 as 2
mdadm: added /dev/sde to /dev/md0 as 3
mdadm: added /dev/sdf to /dev/md0 as 4
mdadm: added /dev/sdg to /dev/md0 as 5
mdadm: added /dev/sdh to /dev/md0 as 6
mdadm: added /dev/sdi to /dev/md0 as 7
mdadm: no uptodate device for slot 8 of /dev/md0
mdadm: no uptodate device for slot 9 of /dev/md0
mdadm: no uptodate device for slot 10 of /dev/md0
mdadm: no uptodate device for slot 11 of /dev/md0
mdadm: added /dev/sdc to /dev/md0 as 0
mdadm: /dev/md0 assembled from 8 drives - not enough to start the array.

mdadm.conf

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This configuration was auto-generated on Sun, 25 Jun 2017 22:28:24 -0500 by mkconf
ARRAY /dev/md0 metadata=1.2 name=blackcanary:0 UUID=7bd22c29:465d927f:d0c1d08f:ba3694c9

mdadm --examine /dev/sdd

(Note: Very similar information returns for any of the "good" 8 drives)

/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 7bd22c29:465d927f:d0c1d08f:ba3694c9
           Name : blackcanary:0  (local to host blackcanary)
  Creation Time : Sun Jun 25 22:53:12 2017
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 7813780144 (3725.90 GiB 4000.66 GB)
     Array Size : 39068897280 (37259.00 GiB 40006.55 GB)
  Used Dev Size : 7813779456 (3725.90 GiB 4000.66 GB)
    Data Offset : 257024 sectors
   Super Offset : 8 sectors
   Unused Space : before=256936 sectors, after=688 sectors
          State : clean
    Device UUID : 91d8e433:aeb9a4f2:75ca10cf:3ceb0a83

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Oct  7 14:23:36 2018
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : b960e905 - correct
         Events : 72036

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)

mdadm --examine /dev/sdk

(Note: The same information returns for any of the "bad" 4 drives)

/dev/sdk:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)

fdisk -b 512 -t gpt -l /dev/sdn

Disk /dev/sdn: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 415081DC-7F8C-11E5-9D78-7824AF43D5FA

Device       Start        End    Sectors  Size Type
/dev/sdn1      128    4194431    4194304    2G FreeBSD swap
/dev/sdn2  4194432 7814037127 7809842696  3.7T FreeBSD ZFS

fdisk -b 4096 -t gpt -l /dev/sdn

Disk /dev/sdn: 3.7 TiB, 4000787030016 bytes, 976754646 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 415081DC-7F8C-11E5-9D78-7824AF43D5FA

Device       Start        End    Sectors  Size Type
/dev/sdn1      128    4194431    4194304   16G FreeBSD swap
/dev/sdn2  4194432 7814037127 7809842696 29.1T FreeBSD ZFS
  • Check whether either sudo fdisk -b 512 -t gpt -l /dev/sdn or sudo fdisk -b 4096 -t gpt -l /dev/sdn shows the expected partitions. – user1686 Oct 9 '18 at 3:56
  • I added the output of both of those to the original post. Some confusion on my end is that I wasn't expecting to see partitions for sd[k-n] considering the other drives don't list partitions. – Uze Oct 9 '18 at 4:11
0

After much research, I was able to restore my RAID array without apparent data loss.

I did ultimately have to use mdadm --create --assume-clean. I opted to use overlay files so that I could non-destructively test various configurations until I found the correct one.

I used mdadm --examine /dev/sd* on the "good" drives to determine their order. Then, I generated the possible permutations of the "bad" drives and ran through them until I had a mountable filesystem. Luckily I only had 24 possible combinations to go through.

I ran through all 24 permutations and didn't have success mounting. I examined one of the mapper drives (mdadm --examine /dev/mapper/sd*) and compared it to the original and noticed that the data offset was different. I added the data-offset param to the config, re-tested the permutations, and had success after the 12th attempt.

These are the commands I ran while testing:

mdadm --stop /dev/md0
mdadm --create --assume-clean --run --level=6 --data-offset=128512 --raid-devices=12 /dev/md0 /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sda /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf /dev/mapper/sdg /dev/mapper/sdh /dev/mapper/sdn /dev/mapper/sdm /dev/mapper/sdl /dev/mapper/sdk
dumpe2fs /dev/md0
fsck.ext4 -v /dev/md0

I used dd to backup the MBR and partition table of the "bad" drives, and then overwrote the MBR & partition table with zeroes to match the "good" drives, so that this shouldn't be an issue on future reboots.

After successfully rebuilding, I generated a new mdadm.conf file and recorded as much data as possible about the array in case I need it in the future. Now, I'm looking into a true backup solution.

  • ... have you also rebooted to confirm that your config holds? – Attie Oct 16 '18 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.