1

A bit of back story. My Xen experience started out about 5 years ago with an Aaeon EMB-CV1 with 4GB of memory, running four VMs. As I outgrew that hardware, I moved to an Aaeon EMB-KB1 with 8GB of memory. I outgrew that hardware and have moved to an Asrock J5040-ITX with 16GB of memory. These have all worked beautifully and have been very performant, it's amazed me the capabilities of paravirtualization and the efficiency attained from the platform. All have been running with a Debian dom0 and Debian domu's, using the Debian-maintained Xen version. Each dom0 install has been a fresh build, migrating the VMs afterwards.

I recently decided to add an additional board into the mix to help with load so that things requiring more horsepower (like my ELK stack, minecraft server, nextcloud instance, etc.) can live on the 5040, while lower-CPU stuff like DNS, VPN server, mail server, etc. can live on a board with a slower CPU. I wanted to be on the same CPU architecture so that I could live-migrate VMs for maintenance, so I picked up an Asrock J4205 with 8GB of memory. After installing Debian 10 (my standard build currently,) the board was snappy and performant. I then installed xen-tools and xen-system-amd64, and after rebooting, the system took significantly longer to boot, and was very stuttery from the console (and SSH as well.) At this point I wasn't running any VMs, didn't have any custom tweaks, etc. Looking at top, there was a lot of "steal," overall averaging around 10% between all four cores (all physical cores.)

I tried tying the dom0 to only one CPU and at that point the dom0 was consistently performant again. However, any domu's I tried to spin up would be very stuttery, with high "steal." Live-migrating them back to the 5040 they'd be fine again. This was also the case if I didn't live-migrate but just started them up on the 4205.

I thought maybe I'd goofed something up in the build somehow, so I blew away that installation and rebuilt it from scratch, and experience the same thing.

If I reboot and choose the non-Xen grub entry, there isn't any "steal" or stuttering, so I don't believe it's related to other system software that's installed, it seems related to Xen itself.

I've also tried completely disabling speedstep, all the c-states, etc. in BIOS, but it doesn't seem to make a difference.

I started logging performance with sysstat and this is what I see:

On the 5040 with 6 VMs running:

08:25:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
08:35:01 AM     all      0.03      0.00      0.11      0.00      0.05     99.81
08:45:01 AM     all      0.03      0.00      0.11      0.00      0.05     99.81
08:55:01 AM     all      0.03      0.00      0.17      0.00      0.20     99.60
09:05:01 AM     all      0.03      0.00      0.12      0.00      0.05     99.80
09:15:01 AM     all      0.03      0.00      0.13      0.00      0.09     99.75
09:25:01 AM     all      0.03      0.00      0.18      0.00      0.26     99.53
Average:        all      0.03      0.00      0.14      0.00      0.12     99.72

On the 4205 with no VMs running:

08:35:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
08:45:02 AM     all      0.03      0.00      0.07      0.01      7.74     92.15
08:55:02 AM     all      0.03      0.00      0.09      0.00      8.95     90.93
09:05:01 AM     all      0.03      0.00      0.07      0.00      9.19     90.70
09:15:01 AM     all      0.03      0.00      0.08      0.00      7.93     91.96
09:25:01 AM     all      0.03      0.00      0.07      0.00      8.85     91.05
09:35:01 AM     all      0.03      0.00      0.19      0.00      6.73     93.05
Average:        all      0.03      0.00      0.09      0.00      8.24     91.63

# top
top - 09:45:55 up  1:29,  1 user,  load average: 0.15, 0.11, 0.08
Tasks: 161 total,   2 running, 159 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  1.1 sy,  0.0 ni, 96.6 id,  0.0 wa,  0.0 hi,  0.0 si,  2.3 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni, 78.9 id,  0.0 wa,  0.0 hi,  0.0 si, 21.1 st
%Cpu2  :  0.0 us,  1.1 sy,  0.0 ni, 89.4 id,  0.0 wa,  0.0 hi,  0.0 si,  9.6 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni, 62.8 id,  0.0 wa,  0.0 hi,  0.0 si, 37.2 st

Is there any way to tell what's causing the performance degradation, and what the dom0 is doing when it's "stealing" the CPU? I've been googling the issue a lot the last few days and haven't found anything useful so far, only threads saying that this happens when you oversubscribe your domu's, but as I'm not running any domu's at this point I don't see how that could be an issue since it's just sitting there looking cool but not doing any real work.

Local disk storage on both dom0's is a single 20GB Intel 313 SLC SSD. VMs are stored on a Debian nas box, connecting via iscsi.

# uname -a
Linux vhost2 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 GNU/Linux
# cat /etc/debian_version
10.9

# xl info
host                   :
release                : 4.19.0-14-amd64
version                : #1 SMP Debian 4.19.171-2 (2021-01-30)
machine                : x86_64
nr_cpus                : 4
max_cpu_id             : 3
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 1497.612
hw_caps                : bfebfbff:47f8e3bf:2c100800:00000101:0000000f:2094e283:00000000:00000100
virt_caps              : hvm hvm_directio
total_memory           : 8040
free_memory            : 7413
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 11
xen_extra              : .4
xen_version            : 4.11.4
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : placeholder dom0_mem=512M,max:512M no-real-mode edd=off
cc_compiler            : gcc (Debian 8.3.0-6) 8.3.0
cc_compile_by          : pkg-xen-devel
cc_compile_domain      : lists.alioth.debian.org
cc_compile_date        : Fri Dec 11 21:33:51 UTC 2020
build_id               : 6d8e0fa3ddb825695eb6c6832631b4fa2331fe41
xend_config_format     : 4

Edit: I know that the dom0 is technically a special domu, so technically there is a domu running on the box, and at that point, anything I do from the console is technically being run in that domu. Is there any way to get a view of the perspective from the hypervisor? Eg., what the hypervisor is doing at the time that's so important that it has to interrupt the dom0/other domu's. I've noticed that there are things that exacerbate the problem, like just holding down one of the cursor keys when viewing a large file in vim - that tends to make the steal (and stuttering) much more pronounced. Stuff like that had made me suspect possibly something with acpi/speedstep/c-states, but disabling them in the BIOS didn't seem to make a difference. And if it was related to that, I would think it would happen in a non-xen Debian install as well, not just when booted with xen

It's also odd that that issue goes away (from the dom0 perspective anyhow) when I only grant the dom0 one core (but any domu's running still have the issue even if they only have one vcpu assigned.).

I've been wracking my brain and exhausting my Google-fu to try to figure out what's going on, because it's really rather exasperating. I know that it's possible that xen might just not work with this board, which would be fine, I'd just like to have a better understanding of what's going on and why, if possible, before surrendering to that.

0

You must log in to answer this question.

Browse other questions tagged .