Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

have been searching an pulling my hear out for the last 6 hours. I have a virtual machine that has been running fine for the last six months. I was happy ssh'ing into it and it was running a database and some small apps. Tonight ssh stopped working, so I decided to reboot the machine. I now have the following situation:

  • virsh list --all states machine as running
  • I can ping the machine and get a reply
  • When I ssh to the machine I see "ssh: connect to host [myserver] port 22: Connection refused"
  • nmap does not show port 22 as open

I have tried to: - reboot the machine once more (no luck) - mount the filesystem and check /etc/ssh/sshd.conf (has not changed since working situation) - install virsh console, however this does not seem to work

When I mount the fs directly using losetup the strange thing is that file dates seem to be frozen in /var/log/ around the time of the crash. If I look in /var/run/ I can see an sshd.pid, but the time is 6 hours ago (and numerous reboots).

My virsh xml looks like this:

<domain type='kvm' id='21'>
  <name>myserver</name>
  <uuid>09678c8d-a99b-1d18-a7af-88d027cc8f93</uuid>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-1.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/dev/disk01/myserver'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:e3:13:86'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-09678c8d-a99b-1d18-a7af-88d027cc8f93</label>
    <imagelabel>libvirt-09678c8d-a99b-1d18-a7af-88d027cc8f93</imagelabel>
  </seclabel>
</domain>

I'm sort of lost as to where I can look to get the machine up and running again.

On the same instance of kvm I have another server running which is working fine. Both are Ubuntu 12.04.

All help is welcome....

share|improve this question
    
Solved, but cannot self answer for another 7 hours... –  lleto Nov 22 '12 at 4:00
add comment

1 Answer

up vote 0 down vote accepted

I thought I'd answer myself in the hope that Google indexes this and someone may find it usefull one day. Of course no guarantees and I am not responsible for any consequence. It's going to be a long answer.

Situation: I have a Ubuntu 12.04 LTS server (M for master) running two virtual servers (V1 and V2), both Ubuntu 12.04 LTS as well. On the host machine I have three IP adresses, so each machine can be accessed under it's own public IP. The servers are installed on LVM basis.

Yesterday one of the virtual servers stopped accepting SSH connects. Also I could not connect to the database running on it anymore. When I investigated I found:

  • machine was pingable
  • ssh would result in connection refused
  • nmap -sS from another server showed open ports, but not 22
  • virsh list --all showed the machine as running
  • I had no console (serial) installed (lazy, yeah yeah I know)
  • The other virtual server and the master server were running normally

My first instinct was to powercycle the machine, perhaps some process got stuck (have some own software running on the server), so I did:

  • virsh destroy virtual2
  • virsh start virtual2

This resulted in the exact same status:

  • virsh reports virtual2 as running
  • virtual2 is pingable
  • ssh still connection refused

Oke, so something else is the matter. At this time I started worrying a bit that my server may have been compromised, however I could not access it at all. I was slightly less worried when nmap -sS -v -v did not show any strange open ports, but one never knows.

Now my battle plan changed to two main goals:

  • It seems the server is booting but for some or other reason ssh and other processes are not started
  • I need console access to see what is going on

In order to get console access I found I need to make two changes:

  • install a serial console on the virtual machine
  • make a change to the XML of the virtual machine on the master

The problem however was that most of these sites assume a running and accesable server, something I did not have. So I first needed to find a way to gain access to the LVM. Again there are a lot of tutorials, however all of them seemed to work right up to where I wanted to mount the filesystem. It would continously fail with a "mount: you must specify the filesystem type". Even specifying the the filesystem with -t did not resolve this matter.

I finally stumbled onto this site: http://blog.leenix.co.uk/2010/07/howto-mount-kvmxen-virtual-disk-image.html. Perhaps my issue with kpartx was that my LVM contains three partitions (boot, / and var).

Steps I took to mount my logical volumes:

  • list the LVM's with : lvs

     LV        VG     Attr   LSize   Origin Snap%  Move Log Copy%  Convert
     OS        disk01 -wi-ao  18.62g                                      
     virtual1  disk01 -wi-ao  20.00g                                      
     virtual2  disk01 -wi-ao  30.00g                                      
     swap      disk01 -wi-ao 952.00m 
    
  • list the partitions in the LVM : fdisk -l /dev/disk01/virtual2

    Disk /dev/disk01/virtual2: 32.2 GB, 32212254720 bytes
    4 heads, 32 sectors/track, 491520 cylinders, total 62914560 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x00064ee2
    
    Device Boot                         Start         End      Blocks   Id  System
    /dev/disk01/virtual21               63    15623046     7811492   83  Linux
    /dev/disk01/virtual22         15624192    19529727     1952768   82  Linux swap / Solaris
    /dev/disk01/virtusl23         19529728    58591231    19530752   83  Linux
    
  • This gave me the start sectors to be able to mount the partions.

  • Created a mount directory : mkdir /mnt/tempfs

  • Now I could mount the two partitions by using:

    losetup -o32256 /dev/loop0 /dev/disk01/virtual2
    or
    losetup -o9999220736 /dev/loop0 /dev/disk01/virtual2
    

If your start blocks are different you can calculate the offste (-o option) by multiplying the start position (63 for the first partition and 19529728 for the third) with the sector size reported in the fdisk -l output.

  • Now I could mount the partition : mount /dev/loop0 /mnt/tempfs/

  • I now had access to my files. I later found there is also a tool called guestfish. You can use this to gain access to the files by issuing : guestfish -d virtual2 -i. If you want readonly access add --ro at the end).

  • To return the partion to a state were I felt comfortable starting the virtual machine I issued:

    umount /mnt/tempfs
    losetup -d /dev/loop0 
    

So first I wanted to see if my server was actually compromised, so I mounted both partitions (one a the time) and:

  • find /mnt/tempfs -ctime 0 -type f (see which files had changed in the last 24 hours)
  • looked at log files and .bash_history files

Fortunately this did not show strange things, so I started to think something was going wrong during boot. The strange thing was that none of the log files actually were updated at boot, so I could not get any information from that. I needed console access.

Again there are lots of tutorials on gaining console access, all seem to be relatively simple like this post: http://ubuntuforums.org/showthread.php?t=1159220

In the 7th post nullzone describes a fairly simple way to set this up. So I went to work:

  • stopped the virtual machine
  • mounted the drive
  • created a file in etc/init/ttyS0.conf with the parameters suggested
  • changed my xml as suggested
  • reloaded libvert-bin
  • started my virtual machine
  • started console

This would continously get stuck after 'Escape character is ^]'. I got no output and no response to any key. I then tried various combinations of serial speeds, different /dev/pts/ devices, nothing worked.

Then finally it dawned on me I had another server that was still running and had ssh access. I made the same change to the XML for that machine and added the /etc/init/ttyS0.conf. On this machine I could simply reload the ttyS0 by issuing 'sudo ttyS0 start'.

Back to virsh, reloaded libvert-bin, destroyed server, started server, console virtual1 and exactly the same. Stuck after the escape character. Fortunately I could check the log files on this machine and noticed an error in /var/log/auth.log : "Nov 22 03:46:11 virtual1 getty[1272]: /dev/–L: No such file or directory". It seemed that getty was not accepting the commandline parameters, so I played around with them and finally found that my /etc/init/ttyS0.conf needed to look like this:

# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on stopped rc RUNLEVEL=[2345]
stop on run­level [!2345]

respawn
exec /sbin/getty -L ttyS0 9600 vt102

So the serial device and speed need to be switched in place. When I now reloaded everything I got console access to the virtual1 server, but still not to the virtual2 server.

By this time I was nearly pulling my hair out and it was 4 o'clock in the morning. It dawned on me that maybe the machine was not getting into a stated were it would acutally load the /etc/init/ttyS0.conf file. Some further research showed that one could also change the /boot/grub/menu.lst file and add console output to grub:

title           Ubuntu 12.04 LTS, kernel 3.2.0-24-virtual
uuid            ba4c41ed-30b6-4a80-808d-64fe26ad7e17
kernel          /boot/vmlinuz-3.2.0-24-virtual root=UUID=ba4c41ed-30b6-4a80-808d-64fe26ad7e17 ro console=ttyS0,9600 earlyprint=serial,ttyS0,9600
initrd          /boot/initrd.img-3.2.0-24-virtual

After reloading and rebooting everything I finally had console output and of the boot process as well. It turned out that an NFS mount could not be made. Fortunately Ubuntu is so smart that it then halts the boot and starts waiting for input from the user. Hence the machine is pingable (network has started), but it never completes the boot process, so SSH is not started and so on.

Quite a long answer, but I hope that maybe someone someday can solve these issues quicker than me and simply have a nice night of sleep.

share|improve this answer
    
Just noticed that Superuser formatted the change to the /boot/grub/menu.lst a bit strange. Normally you see ro quit splash behind the kernel file location. This should become: ro console=ttyS0,9600 earlyprint=serial,ttyS0,9600 –  lleto Nov 22 '12 at 12:11
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.