13

I have a home media server running on openSUSE 12.2. I'm using eight 2TB drives in a RAID-10 configuration. I deliberately bought two different types of drives: four Seagate Barracuda Green and four Western Digital Red. My goal is to configure the RAID such that each mirrored pair within the array will consist of dissimilar drives (ie, one Seagate drive and one WD drive). YaST2 Partitioner unfortunately did not give me this level of choice in the structure of the array, so now I'm trying to find out what the default RAID10 structure looks like.

I do know the following:

  • sdc, sdd, sde, and sdf are all WD drives
  • sdg, sdh, sdi, and sdj are all Seagate drives

I chose the default 'n2' layout when creating the RAID. My guess based upon info from these two sources is that adjacent drives are mirrored (ie, sdc==sdd, sde==sdf, etc), but I want to know for sure:

Here is the output of 'mdadm --detail /dev/md0':

/dev/md0:
        Version : 1.0
  Creation Time : Sat Mar 16 12:55:11 2013
     Raid Level : raid10
     Array Size : 7814045696 (7452.05 GiB 8001.58 GB)
  Used Dev Size : 1953511424 (1863.01 GiB 2000.40 GB)
   Raid Devices : 8
  Total Devices : 8
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Sat Mar 16 13:09:37 2013
          State : active, resyncing
 Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 2048K

  Resync Status : 1% complete

           Name : aldaris:0  (local to host aldaris)
           UUID : c6cc3943:97394500:b77d44cd:f02ed876
         Events : 149

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1
       2       8       65        2      active sync   /dev/sde1
       3       8       81        3      active sync   /dev/sdf1
       4       8       97        4      active sync   /dev/sdg1
       5       8      113        5      active sync   /dev/sdh1
       6       8      129        6      active sync   /dev/sdi1
       7       8      145        7      active sync   /dev/sdj1

And here are the contents of /proc/mdstat:

Personalities : [raid10] md0 : active raid10 sdj1[7] sdi1[6] sdh1[5] sdg1[4] sdf1[3] sde1[2] sdd1[1] sdc1[0]
      7814045696 blocks super 1.0 2048K chunks 2 near-copies [8/8] [UUUUUUUU]
      [>....................]  resync =  4.8% (375163456/7814045696) finish=1206.5min speed=102751K/sec
      bitmap: 57/59 pages [228KB], 65536KB chunk

unused devices: <none>

So my questions are:

  1. How do I tell which drives are mirrors of each other?
  2. Is there a way to change this, or should I just swap the wires around (since that will swap the drive letters) and then rebuild the RAID?

Thanks in advance.


Tangential note, for anyone wants to know my reasoning for doing this is: Drives of the same model and batch, operated under similar usage loads, uptime, and temperature have little systematic variation, and differences in time to failure between drives will be primarily driven by random variation in the manufacturing process. This increases the risk of multiple drives dying at once. By purchasing drives not just from different batches but completely different manufacturers, I am introducing systematic variation into my array, thus influencing which drives will fail at similar times.

6
  • 1
    Home media servers are off-topic for ServerFault. That said, you clearly put some effort into this and I'm going to flag it for migration to unix.SE.
    – Andrew B
    Mar 16, 2013 at 18:26
  • @AndrewB, why is that?
    – poige
    Mar 16, 2013 at 19:56
  • It's a very interesting question and the reasoning behind is so well understood. You don't have to change wires since just to specify disk in when (re-)creating RAID is much more simpler.
    – poige
    Mar 16, 2013 at 20:03
  • @poige This falls under anything in a home setting. I tried to make it clear that I considered the question adequately researched despite this.
    – Andrew B
    Mar 16, 2013 at 20:19
  • If I were to wildly speculate, then I would guess that you could look at the RaidDevice column. It might be as simple as (0,1),(2,3),(4,5),(6,7). That would make me assume that you need to make RaidDevice line up like WD/Sea/WD/Sea/etc... But I am not 100% certain about that.
    – Zoredache
    Mar 16, 2013 at 20:26

5 Answers 5

10

Recent versions of mdadm show this right in the details of the array. Example from mdadm v3.3 - 3rd September 2013

 $ mdadm --detail /dev/md1

/dev/md1:
        Version : 1.1
  Creation Time : Tue Aug 23 11:45:41 2016
     Raid Level : raid10
     Array Size : 3864803328 (3685.76 GiB 3957.56 GB)
  Used Dev Size : 1932401664 (1842.88 GiB 1978.78 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Fri Aug 26 09:39:28 2016
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 512K

           Name : px4-300r-THXOAP:1  (local to host px4-300r-THXOAP)
           UUID : 5ee06437:83dfdb64:808feaa2:5d57b1e6
         Events : 620

    Number   Major   Minor   RaidDevice State
       4       8       50        0      active sync set-A   /dev/sdd2
       1       8       34        1      active sync set-B   /dev/sdc2
       2       8       18        2      active sync set-A   /dev/sdb2
       3       8        2        3      active sync set-B   /dev/sda2

Note the denotation set-A or set-B. In the above case, sdd and sdb can fail together without data loss. It is possible this data is not available while the array is rebuilding though.

6
  • 1
    The set-A and set-B tips are really helpful. But, if sdd and sdb are in the same mirrored set as given in the above example, then losing both of those drives would result in losing all data. Am I missing something?
    – Curtis
    Dec 18, 2020 at 21:41
  • 1
    I found another conversation about this, that seems to confirm my understanding of how it works. If I'm right, then the above post should be corrected before someone loses their entire array by removing both drives from the same set.
    – Curtis
    Dec 18, 2020 at 21:48
  • 3
    This answer is correct. The other post is wrong. set-A and set-B are mirrors of each other. Try reading them with dd as suggested in the below answer.
    – JoeSlav
    Dec 22, 2020 at 12:27
  • 1
    Sorry, I should have studied the other answers that suggest using dd... you're right, the above answer is correct. I made the bad assumption that it wouldn't be reliable to use dd to check this while the system is in use. In fact, it didn't work for me on the main partitions, but since /boot was a different partition (and not getting written too), the dd trick worked great.
    – Curtis
    Dec 24, 2020 at 16:47
  • 1
    Can someone clarify what set-A and set-B are indicating in this case? The answer indicates that sdd and sdb can fail together. Is it also the case that sdd and sda can fail together?
    – Erotemic
    Jun 14, 2021 at 19:06
3

I had the same issue and after googling a while I didn't find a reliable answer. After giving it some thoughts, I figured that the mirrors have the same data and so we could compare some part of it.

NOTE: BE CAREFUL, IF YOU HAVE MORE THAN 2 DRIVES WITH THE SAME CHECKSUM YOU ARE PROBABLY COMPARING EMPTY DISKSPACE, CHOOSE ANOTHER OFFSET (skip option).

With this few commands, you can figure it out:

for disk in sda sdb sdc sdd
do
  echo -n "$disk = ";
  dd if=/dev/$disk skip=1M bs=1M count=1 2>/dev/null | md5sum;
done

This will output something like:

sda = 7c4ef0f3e0143b35e044d5d65908a3a2  -
sdb = 7c4ef0f3e0143b35e044d5d65908a3a2  -
sdc = e02f7d61ad3791bd691da5b7516928a5  -
sdd = e02f7d61ad3791bd691da5b7516928a5  -

Now we know that sda/sdb is one mirror and sdc/sdd another one. One of each must stay to avoid data loss.

The "dd" command is reading one time (count=1) one Megabyte (bs=1M) at one Megabyte offset from the disk start (skip=1M). Don't skip=0, because the begining of the disk contains different information. The data usually begins after 1MB.

2
  • 1
    dd if=/dev/$disk skip=1M bs=1M count=1 didn't work for me. dd (coreutils) 8.23 from Debian 8 (Jessie) doesn't support skip with a unit. Instead I used skip=1 where 1 is relative to bs. Possibly a typo? Feb 16, 2016 at 12:45
  • 1
    FYI If you're seeing the MD5 sum d41d8cd98f00b204e9800998ecf8427e your dd call fails. That is the hash of the empty string:-) Feb 16, 2016 at 13:17
2

You can always verify which are mirrors by a crude manner of comparing the data, for example:

# dd if=/dev/sda1 bs=1M skip=10 count=50 2> /dev/null | md5sum -
7c01afa434fc74aeddc8ec0546e4c332  -
# dd if=/dev/sdb1 bs=1M skip=10 count=50 2> /dev/null | md5sum -
1f0f8166857710d555cb35db4a23891a  -
# dd if=/dev/sdg1 bs=1M skip=10 count=50 2> /dev/null | md5sum -
7c01afa434fc74aeddc8ec0546e4c332  -
# dd if=/dev/sdf1 bs=1M skip=10 count=50 2> /dev/null | md5sum -
1f0f8166857710d555cb35db4a23891a  -

(if you don't get any matches, you may need to increase skip=, as you're not skipping over RAID superblocksl; and if you get same md5sum for more than 2 disks, you may need to increase count=, as you're probably reading and m5summing zeros - to prevent that you should put some data on the raid first, otherwise they might be full of zeroes)

As for the swapping wires around, you don't need to do that - mdadm should create raid with devices as specified on command line in mdadm --create, so you would just specify drives in different order on command line.

0

I think you are talking about a real raid 10 array (1+0 striped mirroring)

sdc/sdg = md0 raid1 2TB |
sdd/sdh = md1 raid1 2TB |_  md4 raid0 8TB
sde/sdi = md2 raid1 2TB |
sdf/sdj = md3 raid1 2TB |

1. Create your 4 raid1 arrays:

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/{sdc,sdg}
mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/{sdd,sdh}
mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/{sde,sdi}
mdadm --create /dev/md3 --level=1 --raid-devices=2 /dev/{sdf,sdj}

At this time you have 4 raid 1 arrays with 2TB space on each,

2. Let's assemble them!

mdadm --create /dev/md4 --run --level=0 --raid-devices=4 /dev/md{0,1,2,3}

--run options is usefull because some components are active in another array

3. Adapt your mdadm.conf file

You may need (depending of your configuration) to adapt this file to reflect changes about our new array (/dev/md4).

4. Enjoy.. Your new DIY raid10 array!

2
  • No, he's talking about linux's mdadm raid10 personality. I believe your answer doesn't apply (although, to be fair, it does give the OP an alternative to achieve what he needs to do)
    – GnP
    Jun 5, 2014 at 23:49
  • The performance and supported features differ as well. Sep 3, 2018 at 15:32
0

Run "mdadm --examine device" on each component device (i.e. /dev/sda1, /dev/sdb1, etc.). The information there should help you determine which components are each others' mirror.

You must log in to answer this question.