2

I have five raw images of a server's disks (stored on one external HDD). Each of those disks contains two partitions which are "Linux raid autodetect" (fd) and are members of RAID1 (boot-related; 3 devices (2 active + 1 spare)) and RAID5 (other data; 5 devices).

Having mapped devices using kpartx (or losetup, doesn't really matter) I run mdadm to assemble RAID arrays. RAID1 successfully assembled and I can mount it in a regular way. RAID5 is being assembled (actually "active", according to /proc/mdstat) but:

  1. All devices are "faulty" and "removed";
  2. I see additional errors in dmesg (blk_update_request: I/O error, dev loop4, sector 16) after which I get more errors (md: super_written gets error=-5; md/raid:md0: Disk failure on loop5, disabling device.);
  3. I can't mount RAID5 array (fair enough).

I tried all suggestions I could find on the web:

  1. Use --force key -> won't work;
  2. Use --create and --assume-clean -> won't work;
  3. Use --zero-superblock (which is similar to 2 ) -> won't work.
  4. If check with --examine - all member devices are in clean state, checksum is correct and event numbers are equal.

As per 2 and 3 - it seems that I CAN'T write to those mapped devices. I strace'd --zero-superblock and, on write() syscall, I got EPERM (Operation not permitted). Can't imagine what prevents me from writing to mapped devices (according to lsblk --fpm RAID5 device is writable brw-rw----).

Another peculiar detail. Above I mentioned that I get I/O error on sector 16. I checked what's located at sector 16 (offset 8192 bytes) in every loop (mapped) RAID5 device. Appeared that it starts with magic number 0x6d746962 (ASCII: "bitm") - likely bitmap related:

00002000  62 69 74 6d 04 00 00 00  48 a7 15 94 56 79 23 ed  |bitm....H...Vy#.|<br/>
00002010  7d 7b 33 bc b9 f8 71 7a  d0 03 00 00 00 00 00 00  |}{3...qz........|<br/>
00002020  d0 03 00 00 00 00 00 00  00 b0 d8 0d 00 00 00 00  |................|

I'm not sure if it's related (at least I don't know how to play with bitmap while assembling array) but it's clear that system can't succeed with blk_update_request operation (likely it either lacks some rights or can't find what's needed).

I on purpose won't post wall of logs - please, let me know if it's needed. I'll enclose some info which describes situation better.

md0 : active raid5 dm-6[5](F) dm-8[4](F) dm-2[3](F) dm-4[2](F) dm-10[1](F)
      464609280 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/0] [_____]
      bitmap: 0/1 pages [0KB], 65536KB chunk

And here is from --detail output:

Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       2       0        0        2      removed
       4       0        0        4      removed
       6       0        0        6      removed
       8       0        0        8      removed

       1     252       10        -      faulty   /dev/dm-10
       2     252        2        -      faulty   /dev/dm-2
       3     252        4        -      faulty   /dev/dm-4
       4     252        8        -      faulty   /dev/dm-8
       5     252        6        -      faulty   /dev/dm-6

Today is a 3rd day in a row of my "tries harder"... I would appreciate any of your advices. Thanks.

0

So, finally, I managed to assemble the RAID5. As appeared, the problem was in the the following (I haven't mentioned it in my question). Disks' images I had were in E01 format, split to number of parts. Prior mapping partitions I used ewfmount (including -X allow_root) to "assemble" multiple E01s into one file. Turned out that FUSE, despite you see your images mounted as RW, somehow blocks them, making devices read-only (even if you edit /etc/fuse.conf).

One of my tries, while troubleshooting, was to export multiple E01s to five regular dd/raw formatted images and work with them directly, excluding ewfmount from the 'scheme".

Once export was completed I mapped raw disks using kpartx, and had been able to assemble RAID5 in a regular way with no problems.

Multiple Operation not permitted or I/O error problems were obvious and had to giveme a hint but I, due to unknown reasons, denied them (until I tried to modify some sectors on mapped devices with dd directly and failed).

That's it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.