Last week at work, our linux server (CentOS 5.5) was unresponsive to log in attempts, so I had to hard shut down it. After pulling out a couple of disks, on boot up it reported a degraded raid array and that fsck -p failed prompting to do a manual fsck. The server has 5x 2TB disks in a hardware RAID 5 array. Software side, I believe this is just arranged into one big logical volume that includes /boot / and /home, and a second logical volume for swap.
I reimported the RAID configurations on the removed disks, at which point the RAID array still showed degraded status, and the machine still returns fsck error on boot. The fifth disk started autorebuilding, but failed, probably due to file system corruption. Fortunately, I was able to recover the 2+ TB of data off the server using rescue mode (whew!). Then I ran fsck -yf on the logical volume, which made some changes. Now fsck returns clean on boot, but when I get to the Cent OS login screen, I am greeted with boxes replacing all of the font. An error of some variety pops up that prevents me from logging in, but I'm unable to read the error, since it is also all boxes. I also can't log in via text terminal (continually reprompted with login:, no chance to enter password) or SSH (server responds, but reports incorrect password).
At this point, I try to run fsck, but it tells me the filesystem is clean. I am still able to get in to the filesystem in rescue mode from the install dvd and the files I have looked at all seem to be OK. I'd really rather avoid a total reinstall, since this would require a lot of reinstallation and copying data back and from rescue mode files look to be intact. Did I totally bork it up by running fsck on the logical volume, or letting RAID auto rebuild? What are your recommendations on how to proceed?