I thought I'd answer myself in the hope that Google indexes this and someone may find it usefull one day. Of course no guarantees and I am not responsible for any consequence. It's going to be a long answer.
I have a Ubuntu 12.04 LTS server (M for master) running two virtual servers (V1 and V2), both Ubuntu 12.04 LTS as well. On the host machine I have three IP adresses, so each machine can be accessed under it's own public IP. The servers are installed on LVM basis.
Yesterday one of the virtual servers stopped accepting SSH connects. Also I could not connect to the database running on it anymore. When I investigated I found:
- machine was pingable
- ssh would result in connection refused
- nmap -sS from another server showed open ports, but not 22
- virsh list --all showed the machine as running
- I had no console (serial) installed (lazy, yeah yeah I know)
- The other virtual server and the master server were running normally
My first instinct was to powercycle the machine, perhaps some process got stuck (have some own software running on the server), so I did:
- virsh destroy virtual2
- virsh start virtual2
This resulted in the exact same status:
- virsh reports virtual2 as running
- virtual2 is pingable
- ssh still connection refused
Oke, so something else is the matter. At this time I started worrying a bit that my server may have been compromised, however I could not access it at all. I was slightly less worried when nmap -sS -v -v did not show any strange open ports, but one never knows.
Now my battle plan changed to two main goals:
- It seems the server is booting but for some or other reason ssh and other processes are not started
- I need console access to see what is going on
In order to get console access I found I need to make two changes:
- install a serial console on the virtual machine
- make a change to the XML of the virtual machine on the master
The problem however was that most of these sites assume a running and accesable server, something I did not have. So I first needed to find a way to gain access to the LVM. Again there are a lot of tutorials, however all of them seemed to work right up to where I wanted to mount the filesystem. It would continously fail with a "mount: you must specify the filesystem type". Even specifying the the filesystem with -t did not resolve this matter.
I finally stumbled onto this site: http://blog.leenix.co.uk/2010/07/howto-mount-kvmxen-virtual-disk-image.html. Perhaps my issue with kpartx was that my LVM contains three partitions (boot, / and var).
Steps I took to mount my logical volumes:
list the LVM's with : lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
OS disk01 -wi-ao 18.62g
virtual1 disk01 -wi-ao 20.00g
virtual2 disk01 -wi-ao 30.00g
swap disk01 -wi-ao 952.00m
list the partitions in the LVM : fdisk -l /dev/disk01/virtual2
Disk /dev/disk01/virtual2: 32.2 GB, 32212254720 bytes
4 heads, 32 sectors/track, 491520 cylinders, total 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00064ee2
Device Boot Start End Blocks Id System
/dev/disk01/virtual21 63 15623046 7811492 83 Linux
/dev/disk01/virtual22 15624192 19529727 1952768 82 Linux swap / Solaris
/dev/disk01/virtusl23 19529728 58591231 19530752 83 Linux
This gave me the start sectors to be able to mount the partions.
Created a mount directory : mkdir /mnt/tempfs
Now I could mount the two partitions by using:
losetup -o32256 /dev/loop0 /dev/disk01/virtual2
losetup -o9999220736 /dev/loop0 /dev/disk01/virtual2
If your start blocks are different you can calculate the offste (-o option) by multiplying the start position (63 for the first partition and 19529728 for the third) with the sector size reported in the fdisk -l output.
Now I could mount the partition : mount /dev/loop0 /mnt/tempfs/
I now had access to my files. I later found there is also a tool called guestfish. You can use this to gain access to the files by issuing : guestfish -d virtual2 -i. If you want readonly access add --ro at the end).
To return the partion to a state were I felt comfortable starting the virtual machine I issued:
losetup -d /dev/loop0
So first I wanted to see if my server was actually compromised, so I mounted both partitions (one a the time) and:
- find /mnt/tempfs -ctime 0 -type f (see which files had changed in the last 24 hours)
- looked at log files and .bash_history files
Fortunately this did not show strange things, so I started to think something was going wrong during boot. The strange thing was that none of the log files actually were updated at boot, so I could not get any information from that. I needed console access.
Again there are lots of tutorials on gaining console access, all seem to be relatively simple like this post:
In the 7th post nullzone describes a fairly simple way to set this up. So I went to work:
- stopped the virtual machine
- mounted the drive
- created a file in etc/init/ttyS0.conf with the parameters suggested
- changed my xml as suggested
- reloaded libvert-bin
- started my virtual machine
- started console
This would continously get stuck after 'Escape character is ^]'. I got no output and no response to any key. I then tried various combinations of serial speeds, different /dev/pts/ devices, nothing worked.
Then finally it dawned on me I had another server that was still running and had ssh access. I made the same change to the XML for that machine and added the /etc/init/ttyS0.conf. On this machine I could simply reload the ttyS0 by issuing 'sudo ttyS0 start'.
Back to virsh, reloaded libvert-bin, destroyed server, started server, console virtual1 and exactly the same. Stuck after the escape character. Fortunately I could check the log files on this machine and noticed an error in /var/log/auth.log : "Nov 22 03:46:11 virtual1 getty: /dev/–L: No such file or directory". It seemed that getty was not accepting the commandline parameters, so I played around with them and finally found that my /etc/init/ttyS0.conf needed to look like this:
# ttyS0 - getty
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.
start on stopped rc RUNLEVEL=
stop on runlevel [!2345]
exec /sbin/getty -L ttyS0 9600 vt102
So the serial device and speed need to be switched in place. When I now reloaded everything I got console access to the virtual1 server, but still not to the virtual2 server.
By this time I was nearly pulling my hair out and it was 4 o'clock in the morning. It dawned on me that maybe the machine was not getting into a stated were it would acutally load the /etc/init/ttyS0.conf file. Some further research showed that one could also change the /boot/grub/menu.lst file and add console output to grub:
title Ubuntu 12.04 LTS, kernel 3.2.0-24-virtual
kernel /boot/vmlinuz-3.2.0-24-virtual root=UUID=ba4c41ed-30b6-4a80-808d-64fe26ad7e17 ro console=ttyS0,9600 earlyprint=serial,ttyS0,9600
After reloading and rebooting everything I finally had console output and of the boot process as well. It turned out that an NFS mount could not be made. Fortunately Ubuntu is so smart that it then halts the boot and starts waiting for input from the user. Hence the machine is pingable (network has started), but it never completes the boot process, so SSH is not started and so on.
Quite a long answer, but I hope that maybe someone someday can solve these issues quicker than me and simply have a nice night of sleep.