6

My setup is:

  • L0: KVM, ubuntu
  • L1: Windows 10
  • L2: Virtualbox (ubuntu) or Bluestacks (Android Emulator)

Symptoms:

  • Bluestacks complains that Hyper-V is enabled and won't start, even though it is not in windows 'additional features'.
  • Virtualbox will reboot the entire Windows guest once I run a new VM. I've tried changing basically every setting I could find.

I do have nested enabled:

$ systool -m kvm_intel -v | grep nested
    nested              = "Y"
$ cat /sys/module/kvm_intel/parameters/nested
Y

Inside my config for my VM ('virsh edit ')

<features>
  <acpi/>
  <apic/>
  <vmport state='off'/>
</features>
<cpu mode='custom' match='exact' check='partial'>
  <model fallback='allow'>Haswell</model>
  <feature policy='require' name='vmx'/>
</cpu>

systeminfo in the L1 guest says:

Hyper-V Requirements: A hypervisor has been detected. Features required for Hyper-V will not be displayed.

Any idea why I can't get nested virtualization working at L2 with a windows host?

4
  • Start a command prompt or PowerShell as administrator, and run the command systeminfo. At the end will be about five lines of "Hyper-V" related information. Paste that section into your question. Apr 30, 2019 at 0:40
  • Thanks for your help Micheal. I've pasted. It appears that it is already a Hyper-V host from this: docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/…
    – Joe
    Apr 30, 2019 at 15:12
  • I also double-checked and Hyper-V (and all subitems) are unchecked in 'additional features' control panel.
    – Joe
    Apr 30, 2019 at 15:14
  • That message does not mean that you have Hyper-V installed, it means that your OS knows it is in a virtual machine and refuses to allow use of Hyper-V (or any further virtualization). Apr 30, 2019 at 16:10

2 Answers 2

2

To get Hyper-V to work in a KVM virtual machine, you need to both pass through the host CPU model as-is, and disable the hypervisor CPU feature (of the virtual CPU). Your virtual machine XML should appear as follows:

  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <feature policy='disable' name='hypervisor'/>
  </cpu>

In addition, you should create the VM with Q35 chipset, not i440fx. And ideally the guest should boot with UEFI. If you use BIOS boot you need SeaBIOS 1.10 (I think, it's been a while) or later. For Ubuntu that means Ubuntu 18.04 LTS or later.

2
  • thanks for this, not at home at the moment and dont wnat to risk it messing up my remote connection but will try on friday and then accept your answer if it works. thanks!
    – Joe
    May 1, 2019 at 14:35
  • so im not exactly sure the degree to which this helped. What seemed to make the biggest difference is enableing 'vt-d' in my bios
    – Joe
    May 6, 2019 at 8:48
13

After spending entirely too long on this particular problem I found a solution. I too was having problems getting nested virtualization working on my Windows Server 2019 VM Guest. After installing HyperV my VM would no longer boot up.

In this particular situation I'm running Linux with KVM/Qemu as my L0 hypervisor, and run a Windows Server 2019 VM that I need to run docker on. One of our clients has a set of PowerShell scripts that they use to setup the docker dev environment (very Windows-centric which I strongly dislike).

Firstly, I did confirm that my kvm-intel kernel model had nested virtualization support enabled:

cat /sys/module/kvm_intel/parameters/nested

So that wasn't the issue, but it's worth mentioning that this is required to support nested virtualization for KVM/Qemu setups.

The actual solution was to change my CPU configuration in Virt Manager from (the default)

<cpu mode="host-model" check="partial"/>

To:

<cpu mode="custom" match="exact" check="partial">
    <model fallback="allow">Skylake-Client-noTSX-IBRS</model>
    <feature policy="disable" name="hypervisor"/>
    <feature policy="require" name="vmx"/>
</cpu>

Which forces my CPU model to be Skylake-Client-noTSX-IBRS instead of Skylake-Client-IBRS (which was the default). For some reason TSX caused serious problems for me.

<feature policy="disable" name="hypervisor" /> makes windows "think" it's not running as a VM.

<feature policy="require" name="vmx" /> forces the vmx CPU feature allowing the Windows Server 2019 VM to itself be able to run other VMs.

If you look at your Windows task manager (from within the VM), you'll see that Windows believes the processors are bare metal processors (as opposed to virtual processors) while simultaneously having virtualization support. At this point I I could install Docker for windows and begin working with it. The setup was:

  • L0: Linux (KVM/Qemu)
  • L1: Windows Server 2019 (HyperV)
  • L2: Docker

I hope this helps other people running into problems with Nested Virtualization on Windows Guests running in KVM. It's admittedly an edge case, but I could see others running into the same challenges.

3
  • Thank you so much for your post. It seems like not too many people are doing this. I used your solution on my laptop (i7-8550U CPU) with Ubuntu(L0) Windows 10(L1) WSL&Docker(L2) worked great. Now I am trying it on an Ryzen 2700X machine and hitting the same brick wall. Not sure where to go. Any advice? Oct 17, 2020 at 17:42
  • This should definitely be the accepted answer. Thanks so much for sharing. Apr 19, 2021 at 14:46
  • And with this, my Windows VM boots and has WSL2 installed - it failed as soon as I tried WSL2 otherwise. Note that to get all desired vCPUs recognized you may still need to pass a topology line in the CPU that maps your desired pass-through cores into less than 2 sockets. Thanks for this answer!
    – Mike Hardy
    Dec 12, 2021 at 4:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .