How to make a three disk software RAID by adding the third disk?

Question

The server is running CentOS and it has been successfully running a two disk software RAID. I wish to add a third disk to this for redundancy, so when one disk fails, the system still has two disks operating until a replacement for the failed disk arrives.

When installing CentOS from a DVD-ROM, that's where I have set-up the software two disk RAID in the past. But how do you add a third disk to the existing two disk software RAID in a running system?

Answer 1

2 disks and redundancy suggest RAID1 is already used. To increase redundancy I do not think RAID5 is an option as it offers more space but - with 3 disks it is the same as RAID1 with 2 and allows losing one disk.

So presuming RAID1:

mdadm --add /dev/mdX /dev/sdY
mdadm --grow --raid-devices=3 /dev/mdX

Using lvm2 (either mirror or dm-raid) it would be:

vgextend VG /dev/sdY
lvconvert -m2 VG/LV

Answer 2

You want to reshape your RAID-1 to a three-disk RAID-5 setup. md, the software RAID you're probably using, fortunately supports reshaping.

ＷＡＲＮＩＮＧ： Make sure you have backed up your data just in case something goes catastrophically wrong (not likely, but always a possibility).

---

Full Example

You didn't provide any details about your setup, so I'll do a full example from scratch anyone can try.

Set up RAID-1

root@node51 [/tmp]# truncate -s 1G 1.img
root@node51 [/tmp]# truncate -s 1G 2.img
root@node51 [/tmp]# losetup /dev/loop1 1.img
root@node51 [/tmp]# losetup /dev/loop2 2.img
root@node51 [/tmp]# mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/loop{1,2}
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: size set to 1048000K
Continue creating array? yes
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

How the array looks as RAID-1:

root@node51 [/tmp]# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Wed Mar  9 15:30:18 2016
     Raid Level : raid1
     Array Size : 1048000 (1023.61 MiB 1073.15 MB)
  Used Dev Size : 1048000 (1023.61 MiB 1073.15 MB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Wed Mar  9 15:30:24 2016
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : node51:0  (local to host node51)
           UUID : 9f03708b:dfe27926:16485757:02cbc146
         Events : 17

    Number   Major   Minor   RaidDevice State
       0       7        1        0      active sync   /dev/loop1
       1       7        2        1      active sync   /dev/loop2

Add a filesystems and/or partitions (optional)

I'll demonstrate a simple filesystem setup and resize here, but you can find more detailed resizing tips elsewhere (see "Additional Resources" at the bottom of this answer).

root@node51 [/tmp]# mkfs.ext4 /dev/md0
mke2fs 1.42.9 (4-Feb-2014)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262000 blocks
13100 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

root@node51 [/tmp]# mount /dev/md0 /mnt
root@node51 [/tmp]# df -hT /mnt
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/md0       ext4  992M  1.3M  923M   1% /mnt

Add the third disk

root@node51 [/tmp]# truncate -s 1G 3.img
root@node51 [/tmp]# losetup /dev/loop3 3.img
root@node51 [/tmp]# mdadm --add /dev/md0 /dev/loop3
mdadm: added /dev/loop3

Once you have added the third disk, you should see it as a spare:

root@node51 [/tmp]# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Wed Mar  9 15:30:18 2016
     Raid Level : raid1
     Array Size : 1048000 (1023.61 MiB 1073.15 MB)
  Used Dev Size : 1048000 (1023.61 MiB 1073.15 MB)
   Raid Devices : 2
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Wed Mar  9 15:31:10 2016
          State : clean
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

           Name : node51:0  (local to host node51)
           UUID : 9f03708b:dfe27926:16485757:02cbc146
         Events : 18

    Number   Major   Minor   RaidDevice State
       0       7        1        0      active sync   /dev/loop1
       1       7        2        1      active sync   /dev/loop2

       2       7        3        -      spare   /dev/loop3

Reshape RAID-1 to RAID-5

root@node51 [/tmp]# mdadm --grow /dev/md0 --level=5 --raid-devices=3
mdadm: level of /dev/md0 changed to raid5
mdadm: Need to backup 128K of critical section..

This may take a long time. You can check cat /proc/mdstat to see the progress:

root@node51 [/tmp]# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid5 loop3[2] loop2[1] loop1[0]
      1048000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      [=====>...............]  reshape = 25.0% (262208/1048000) finish=0.1min speed=87360K/sec

unused devices: <none>

You have RAID-5 now!

root@node51 [/tmp]# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Wed Mar  9 15:30:18 2016
     Raid Level : raid5
     Array Size : 2096000 (2047.22 MiB 2146.30 MB)
  Used Dev Size : 1048000 (1023.61 MiB 1073.15 MB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Wed Mar  9 15:31:49 2016
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           Name : node51:0  (local to host node51)
           UUID : 9f03708b:dfe27926:16485757:02cbc146
         Events : 41

    Number   Major   Minor   RaidDevice State
       0       7        1        0      active sync   /dev/loop1
       1       7        2        1      active sync   /dev/loop2
       2       7        3        2      active sync   /dev/loop3

Increase the size of your filesystems and/or partitions (optional)

Here, you can see my filesystem of about 1GiB become 2GiB by the online expanding feature of ext4:

root@node51 [/tmp]# df -hT /mnt
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/md0       ext4  992M  1.3M  923M   1% /mnt
root@node51 [/tmp]# resize2fs /dev/md0
resize2fs 1.42.9 (4-Feb-2014)
Filesystem at /dev/md0 is mounted on /mnt; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/md0 is now 524000 blocks long.

root@node51 [/tmp]# df -hT /mnt
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/md0       ext4  2.0G  1.5M  1.9G   1% /mnt

For other filesystem resizing scenarios, see "Additional Resources" at the end of this answer.

Example cleanup (optional)

Here's how to tear down and delete the example above:

root@node51 [/tmp]# umount /mnt
root@node51 [/tmp]# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
root@node51 [/tmp]# losetup -d /dev/loop1
root@node51 [/tmp]# losetup -d /dev/loop2
root@node51 [/tmp]# losetup -d /dev/loop3
root@node51 [/tmp]# rm -fv /tmp/{1,2,3}.img
removed ‘/tmp/1.img’
removed ‘/tmp/2.img’
removed ‘/tmp/3.img’

---

Additional Resources

I wrote a few other answers covering filesystem resizing. You can use the same concept to grow the filesystem in your RAID.

• Growing an XFS filesystem inside LVM
• Growing an XFS filesystem inside a primary partition