Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have nearly 140 vms that were running on Ubuntu 10.04 using KVM. Different OSes in the VMs, but mainly 10.04 VMs.

They ran well under 10.04. I did a full wipe of the OS disks and brought in all the existing VMs with dirsh define of their config files. I have done this on servers running 3-5 VMs and it has worked fine, but these servers run 20-25 vms each.

After the upgrade I have noticed the following things:

  1. Some of the VMs's time went crazy. Like they were booted and thought it was 2 months ago.
  2. Doing a du/df no longer shows how big the img files that were originally created. Only what is used in them. So when you have a 40G Ubuntu install it shows as 2G in du, ls -lh shows properly
  3. Some (not all) VMs act like they can't write to disk almost as if they run out of space but they have 55%+ free. After deleting a file the VM starts working again.
  4. The VMs are usually still pingable, but the console and SSH fail. Some SSH connections hang after putting in the password.

The disk images live on XFS.

Version info:

  • OS: Ubuntu 12.04
  • Kernel: 3.2.0-30-generic
  • kvm 1:84+dfsg-0ubuntu16+1.0+noroms+0ubuntu14.1 dummy transitional package from kvm to qemu-kvm
  • qemu 1.0+noroms-0ubuntu14.1 dummy transitional package from qemu to qemu-kvm
  • qemu-common 1.0+noroms-0ubuntu14.1 qemu common functionality (bios, documentation, etc)
  • qemu-kvm 1.0+noroms-0ubuntu14.1 Full virtualization on i386 and amd64 hardware
  • qemu-utils 1.0+noroms-0ubuntu14.1 qemu utilities
  • seabios 0.6.2-0ubuntu2.1 legacy BIOS implementation which can be used as a coreboot payload
  • vgabios 0.6c-2ubuntu3 VGA BIOS software for the Bochs and Qemu emulated VGA card
  • libvirt-bin 0.9.8-2ubuntu17.5 programs for the libvirt library
  • libvirt0 0.9.8-2ubuntu17.5 library for interfacing with different virtualization systems

I have also tried build 14.7 of qemu/kvm with the same results.

There are no entries in the logs (libvirt, qemu, syslog) on the Hypervisors or the VMS.

<domain type="kvm">
  <name>template</name>
  <uuid>9b5004aa-e4e1-11de-9137-003048dad824</uuid>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <vcpu>2</vcpu>
  <os>
    <type arch="x86_64" machine="pc">hvm</type>
    <boot dev="hd"/>
    <kernel>/kvm/kernels/vmlinuz-2.6.32-24-server</kernel>
    <initrd>/kvm/kernels/initrd.img-2.6.32-24-server</initrd>
    <cmdline>ro root=/dev/vda</cmdline>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset="utc"/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type="file" device="cdrom">
      <target dev="hdc" bus="ide"/>
      <readonly/>
    </disk>
    <disk type="file" device="disk">
      <source file="/kvm/images/template.img"/>
      <target dev="vda" bus="virtio"/>
    </disk>
    <interface type="bridge">
      <mac address="de:ad:3e:5b:47:99"/>
      <source bridge="br2"/>
      <model type="virtio"/>
    </interface>
    <serial type="pty">
      <target port="0"/>
    </serial>
    <console type="pty">
      <target port="0"/>
    </console>
    <graphics type="vnc" port="-1" listen="0.0.0.0" autoport="yes" keymap="en-us"/>
  </devices>
</domain>

Doing an strace on a running VM I see this:

read(19, "\2\0\0\0\0\0\0\0\0\0\336\255\276\357\10\353\0\f\333\350\275\0\10\0E\0\0B\214\212@\0"..., 69632) = 90
ioctl(6, KVM_IRQ_LINE_STATUS, 0x7fff290e65f0) = 0
read(19, 0x7f7967430a54, 69632)         = -1 EAGAIN (Resource temporarily unavailable)
select(20, [7 8 11 15 16 17 18 19], [], [], {1, 0}) = 1 (in [8], left {0, 999819})
read(16, 0x7fff290f6710, 16)            = -1 EAGAIN (Resource temporarily unavailable)

Any help would be appreciated.

EDIT

I have upgraded to the latest kernel build from Ubuntu. Noticed some Denies from AppArmor when only doing an iozone runs, so I completely removed apparmor on my testing system. Upgraded qemu/kvm and support libraries to those found in 12.10. I also tried using the pc-0.12 and pc-0.14 versions.

All still locking up within an hour or two of booting.

share|improve this question
add comment

2 Answers

Something seems like it's blocking the VMs from the KVM host perspective. Couple of things to try:

  • Does the KVM host CPU usage spike when this happens?
  • What does vmstat show (run vmstat 5)?

sample vmstat

% vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 806972 396372 2369348    0    0    13    26   11  197  4  4 90  2  0   
 0  0      0 808908 396380 2367056    0    0     0    18 6453 13546  2  2 95  1  0  
 0  0      0 807508 396380 2370296    0    0     0    22 6438 13662  2  3 95  1  0  
 0  0      0 809892 396392 2367068    0    0     0    56 6273 13189  1  2 95  1  0  
  • What does iostat show (run iostat -dx 5)?

sample iostat

% iostat -dx 5
Linux 2.6.35.14-106.fc14.x86_64 (grinchy)   01/28/2013  _x86_64_    (4 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.61    11.29    1.14    4.07    58.12   116.23    33.46     0.17   31.81   7.18   3.74
dm-0              0.00     0.00    0.83    2.38    34.75    19.02    16.77     0.10   31.11   2.17   0.70
dm-1              0.00     0.00    0.00    0.00     0.04     0.00     8.00     0.00    5.30   2.70   0.00
dm-2              0.00     0.00    0.92   12.37    23.18    97.21     9.06     0.79   59.40   2.46   3.27

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    12.20    0.00    3.00     0.00   112.00    37.33     0.05   16.73  13.87   4.16
dm-0              0.00     0.00    0.00    1.40     0.00    11.20     8.00     0.05   38.29  12.43   1.74
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   12.60     0.00   100.80     8.00     0.06    4.37   1.92   2.42

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    11.60    0.00    2.60     0.00   107.20    41.23     0.04   13.46  13.23   3.44
dm-0              0.00     0.00    0.00    0.60     0.00     4.80     8.00     0.01   15.00  13.00   0.78
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00   12.80     0.00   102.40     8.00     0.13   10.05   2.08   2.66

We had a similar issue when running KVM on CentOS 5.x but it has not shown up since migrating to CentOS 6.x. Also when debugging the issue it seemed to be Guest VM specific. What I mean is that certain Guest VMs seem to cause this issue across then entire KVM Host.

share|improve this answer
    
I have profiled a system when I had a vm hung as well as with that VM shut off completely and there is no difference. I have seen the issue you are pointing out before. But these aren't causing huge I/O waits either. Their CPU goes up for that VM, but I can't tell what the VM is actually doing. –  Joe Coder Feb 1 '13 at 16:35
add comment

I troubleshot this for a while and found that it was related to the kernel used by the VMs themselves. They were using 2.6.32. We upgraded to 2.6.38-10 which has a timer bug fix that looks to be the culprit. They have been running for 11 months without the issue coming back.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.