0

For the past 8 hours my mdadm re-shape process has been stuck. The output is:

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] 
md0 : active raid6 sdo[11] sdn[12] sdm[13] sdl[14] sdk[15] sdj[16] sdi[17] sdh[20](F) sdg[19] sdr1[1] sds1[2] sdt1[3] sdu1[4] sdp1[10] sdq1[6] sde1[0] sdd1[7] sdb1[9] sdc1[8] sdf1[5]
      17581607424 blocks super 0.91 level 6, 64k chunk, algorithm 2 [20/19] [UUUUUUUUUUUUUUUUUU_U]
      [====>................]  reshape = 23.2% (453532992/1953511936) finish=6484322.5min speed=3K/sec

The count has been at exactly 453532992 the whole time, with the speed decreasing toward 0 (not reaching it, since it's an average speed and the speed was fine for a long time before). I can see that one of the drives has been marked as failed, which has me a little worried. I can re-build onto that drive (I know it's not a hardware failure) once the re-shape is done, but at this rate that's not happening.

Does anybody have any suggestions? Is there something I can do to get mdadm going again?

Edit: To add to the fun, it looks like something else has gone wrong on the system. I'm getting I/O errors when I try to read anything on the root file system. The RAIDed file system is still up and running, but some processes are stuck that won't let me un-mount it. And the errors on the root file system seem to be preventing a clean reboot.

So I suppose it's possible that all of the problems are on the root device and nothing to do with the RAID, it just can't continue to re-shape because it can't read something it needs? I'm not sure. But if I can't do a clean reboot, what would happen to the re-shape if I had to hard-power-reset the box? What if the root drive is physically dead and I need to install the OS on a new drive? What would happen to the array then?

1 Answer 1

0

What I think ended up being the problem was that the root drive (not part of the RAID), which is a solid state drive, was being heavily over-used by means of the backup file for the re-shape.

Additionally, I found a post talking about that exact subject, which basically recommends:

So learn from my mistake, never ever put the –backup-file on ‘volatile’ media like flash.

The I/O was essentially broken on the root drive and it couldn't be used for anything, including running the commands necessary to do anything remotely useful (such as gracefully shut down). I hard-power-cycled the box and kind of hoped for the best.

When it came back up, the root drive was working fine. At least observably. The RAID didn't assemble correctly because it was only trying to use the original 11 drives and not the full 20. I told mdadm to stop the array and then re-assemble it from the 20 devices I know of. Once I did that, it picked up the re-shape where it left off.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .