14

How would I grow a RAID10 array using mdadm?

Obviously I would need to add 2 drives at a time, or would it have to be 4 drives?

I have a feeling its just a bad idea.

Or would be just wiser to just opt for RAID5?

6
  • Usually, if someone is asking me if they need RAID 10, that means they don't need it. Why not Raid 5 or Raid 6?
    – surfasb
    Jul 17, 2011 at 14:57
  • I was hoping for better write performance than RAID5.. But I guess the lack of flexibility rules that out..
    – stuartc
    Jul 17, 2011 at 20:49
  • In general, RAID 5/6 will provide better write performance than RAID 10/01. Because you are distributing writes across all drives versus only half the drives.
    – surfasb
    Jul 17, 2011 at 21:15
  • 7
    @surfasb "RAID 5/6 will provide better write performance than RAID 10" no it won't, because distributing the writes over more drives is more that out-weighed by the write penalty of RAID5/6
    – user84636
    Mar 1, 2014 at 21:22
  • 1
    You can reshape RAID10 with mdadm now. Tested with mdadm-4.0-5 in Fedora 27.
    – Nick
    Feb 4, 2018 at 5:16

7 Answers 7

6

Last time I checked, mdadm won't let you --grow raid10. I glanced over mdadm's manpage now and it still says: Currently supported growth options including changing the active size of component devices and changing the number of active devices in RAID levels 1/4/5/6, changing the RAID level between 1, 5, and 6, changing the chunk size and layout for RAID5 and RAID5, as well as adding or removing a write-intent bitmap.

3
  • Thanks! Appreciated, also another good hint this isn't going to happen quickly either is that I can't find one hardware RAID card that does it either.. RAID5 it is..
    – stuartc
    Jul 17, 2011 at 20:42
  • 2
    RAID5 can be painfully slow on some workloads. What do you intend to do?
    – Stingray
    Jul 19, 2011 at 17:12
  • Mostly just storing media, but also mounting an NFS share for photo editing, so was hoping for some good write speeds as well as reads.
    – stuartc
    Jul 20, 2011 at 14:15
22

To grow RAID 10 you need mdadm in version min. 3.3 and kernel version min 3.5. You need also an even number of disks - unpaired ones can only work as a spare or, eventually, to grow into degradated mode (not tested).

Here goes the example of growing RAID 10 from 4 drives to 6 using mdadm 3.3-2ubuntu2 @ Linux 4.2.0-10-generic. Tested with ext4 data on it, filesystem was unmounted, ext4 was extended after the RAID grow without any issue.

~$ cat /proc/mdstat
md126 : active raid10 sdd1[1] sdc1[0] sdf1[3] sde1[2]
976428032 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
bitmap: 0/8 pages [0KB], 65536KB chunk

~$ sudo mdadm /dev/md126 --add /dev/sdi1 /dev/sdj1
mdadm: added /dev/sdi1
mdadm: added /dev/sdj1
~$ sudo mdadm --grow /dev/md126 --raid-devices=6

~$ cat /proc/mdstat
md126 : active raid10 sdj1[5] sdi1[4] sdd1[1] sdc1[0] sdf1[3] sde1[2]
1464642048 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU]
bitmap: 0/6 pages [0KB], 131072KB chunk
2
  • 3
    This should be the accepted answer nowadays. I've done it myself.
    – kyrofa
    Jul 8, 2019 at 23:54
  • as i just found out myself the hard way, this only applies to raid 10,near2 layouts. raid10,far2 is still un-growable. see i.e. here: wiki.archlinux.org/index.php/RAID#Nested_RAID_levels
    – gabtub
    Jul 16, 2019 at 1:14
10

I realize this is over a year old but someone might find this helpful...

You can expand a raid 10 array, but not how you are hoping. You would have to nest multiple levels of raid. This can be done with mdadm on 2 drives in raid 10, which quite nice performance depending on the layout, but you would have to make multiple 2 disk raid 10 arrays, then attach them to logical node. Then to expand add a few more, and stripe across that. If that is your use case (needing to expand a lot) then you would be wise to use a parity array, which can be grown.

These are the limitations you get with raid 10, while maintaining better read/write performance overall. And a clarification, raid 5/6 absolutely does not "In general, provide better write performance...". Raid 5/6 has their own respective pros/cons just as raid 10, but write performance is not a pro for raid 5/6.

Also, you didnt specify the size of your drives but beware of raid 5 on new large drives. Though if you are careful, you can recover from an unrecoverable read error, you risk downtime and the possibility of not being able to recover at all.

--edit to add info-- Use tools like hdparm (hdparm -i) and lshw to get the serial numbers along with the device name (/dev/sda) when you have a failure. This will ensure you remove the correct device when replacing. Up-arrow on Travis' comment as it is very correct and a nice layout, but as usual, weight the pros and cons of every solution.

10

Some great news from the release announcement for mdadm 3.3:

This is a major new release so don't be too surprised if there are a few issues...

Some highlights are:

...

  • RAID10 arrays can be reshaped to change the number of devices, change the chunk size, or change the layout between 'near' and 'offset'. This will always change data_offset, and will fail if there is no room for data_offset to be moved.

...

According to this answer on U&L, you will need at least linux 3.5 as well.

1
  • 3
    I experimented with growing mdadm raid10 using mdadm v3.3.2 with mixed results. I found that I couldn't grow a raid10 that I had created with an earlier version of mdadm. A raid10 that was created with v3.3, I was able to reshape/grow. So if you're in the situation of wanting to grow a raid10 (but can't), it may be a good idea to create the new raid10 using the latest mdadm.
    – adalle
    Jun 24, 2015 at 18:16
6

I know it's more work and could get confusing, but you can always stripe multiple mirrors.

For example I just setup a 4 drive raid 10 array and latter want to add another 4 drive raid 10 array. Just use mdadm to create a new raid 10 array on the new drives. You could then either create another raid 0 array using the two existing raid devices. However I would use the features of lvm to create the stripe, thus keeping mdadm configs and /dev/md devices in an easy to understand state. Either method would work, and there are probably more but that's what I could do off the top of my head.

1
  • Great way of managing a growing dataset, much easier than most. Tricky part is keeping track of which drives have failed in which which mirror set. But yeah great suggestion!
    – stuartc
    Oct 18, 2012 at 17:16
3

You cannot grow a RAID10-array, this is true. But you can start with two drives (at the cost of no redundancy at this point) and then later, add two more. I have used this method to switch from a RAID5 array with four disc drives to RAID10 with four disc drives.

I degraded RAID5 and removed one disc out of the array. This gave me two discs that were free to use. Then I created a RAID10-array with something along the lines of 'mdadm --create --level 10 /dev/md0 /dev/sda1 missing /dev/sdc1 missing'. This was enough to start the array.

When I had moved the data from the degraded RAID5-array to the RAID10-array, I added two of the RAID5-discs to the RAID10-array and added the extra one as a spare.

It is probably a good idea if someone who is more knowledgable than I can talk about any performance impact this may have had. But the primary reason for my post is to show that it is possible to start off with RAID10 and two drives.

Note: do read the following forum post that asks and answers whether the disk order in mdadm matters.

0

There was always and is a manual solution to this. Suppose you have A,B,C,D drives in raid 10. I usually use a R0(R1,R1) setting but lets suppose we have far offset of mdadm s hybrid raid 10. Remove 1 drive (A) and create new degragaded raid 10 array with new drives A,E,F force it to start and then copy your data. Afterwards wipe B,C,D and add it to degragaded array and sync.

If you had R0(R1(A,B),R1(C,D)) you could have removed two drives A and C and do the same copying

R0(R1(,B),(,D)) --> R0(R1(A,),R1(E,),R1(F,))

then adding C,B,D to degragaded R1 arrays and resyncing so that you will have

R0(R1(A,B),R1(E,C),R1(F,D))

I did not experimented on former, but I remember achieving latter. I also remember that R0(R1,R1,R1,...) works faster than hybrid R10 in fio benchmarks.

If you dont have backup it is risky. Don't do it.. However I am not sure if it is riskier than losing power when mdadm is growing the raid 10 array. You may argue that if copying is involved restoring from backup is easier however if you have many disks attached to a sas expander and raid card, restoring from backup especially over network maybe a lot slower. Since you can always dd old raid to new raid and expand the partition so that you can bypass slow read and write speed problems of many small files.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .