I have here a serious problem on my home server: I have a LVM on a RAID5 array. As memory ran short, I decided to enlarge the RAID array (and later the PV+VG+LVs). So far so good.
I stopped the system and installed the additional hard disk physically. It was detected and I closed the server box and went away (I do not reside in the place where the server is located; it's about 200 km distance). Now I logged (from far away) via SSH into the server and did the following:
I created the partition label and a matching partition on the new device and added the partition with mdadm /dev/md1 --add /dev/sde1
to the md1
array as a hot spare. In a second step I wanted to enlarge the array with mdadm --grow /dev/md1 -n 4
(from 3 to 4 devices). The array started to rebuild.
After a quite short time I got the problem:
The array went down. In fact I see in the output of dmesg
some lines mentioning that the sync process was stopped and that the device (/dev/md1
) was not responding and blocked for >120 sec. Since then every access to the array (via mounted LVs) just blocks.
After some research in the net I found out that this might happen due to a too small stripe_cache_size
on rebuild (actual value is 256).
The main problem is now that the PV contains a LV that is my root file system! Thus I cannot do anything as soon as the RAID blocks. I even cannot open any editor or use cat /proc/mdstat
as these commands need access to the root file system where the programs reside. Now I look for a possibility to repair the system. I see only two solutions:
- I go there personally again and restart from a start CD, ensure a large cache and let the rebuild process run till finished. Then I restart the original system that should start up as normal. The disadvantage is that I have to stay next to the machine till the rebuild is done completely.
- I restart the system directly and avoid the resync process to start (1). Then I change the cache size and let the system do the rebuild actually. This might be possible without me having to drive 400 km for a simple restart as I might be able to tell someone up there what to do.
Now the question (1) is: Can I somehow avoid/prevent the automatic rebuild of the array?
I thought of disabling the newly added device (sde1
) but that would (I think) not avoid the reshape from 3 to 4 disks with one missing disk.
Maybe you have a better alternative. In this case please let me know.
Thanks a lot!