Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to get my head around virtualisation and hypervisors.

So the top OS, say Linux, runs in ring 0, with userspace programmes running in ring 3. The user boots up VMware, with Linux running within it. Which rings/modes is the virtual Linux running on? And say the user boots up another virtual Linux OS within the virtual Linux OS? Are further modes emulated by VMware? How are all of these layers kept separate to protect from faults and malicious behaviour?

And is the above implemented differently on different platforms? Intel VT, AMD-V, ARM, pure software virtualisation, etc?

share|improve this question
+1 I'd like to know the answer to this too. – Mehrdad Dec 1 '12 at 6:10
up vote 2 down vote accepted

This article should provide a lot of information along the lines you are looking for. However, it is specific to VMware.

The general case is that it really depends on a number of things:

  • Is VT-x / AMD-V in use?
  • Is EPT in use?
  • Is VT-d or some type of IOV in use?

The above questions are hardware instruction usage questions. Depending on which AMD/Intel CPU hardware instructions are used to assist with virtualization, the structure of the hypervisor is usually quite a bit different. If no hardware instructions are in use, then the virtual machine is said to be "Fully Emulated" or "running in software", which is often the slowest mode. However, even without hardware support, paravirtualization (where the guest OS knows that it's running in a VM) can significantly accelerate even a software emulation layer, by providing a clear-cut programming interface between the guest and the host.

Some virtual machines, such as qemu without kqemu or kvm, can run entirely in ring 3 -- but doing so has several limitations, such as performance, and the lack of low level hardware access. In general, even those hypervisors without any hardware acceleration at all traditionally run in ring 0, i.e., within the host kernel as a driver.

The use of rings 1 and 2 is fairly rare, but it does happen, as noted in the Anandtech article. Also, rings 1 and 2 are commonly used for bare metal hypervisors such as Xen. To the best of my knowledge, the "dom0" host operating system kernel in Xen runs entirely in ring 1, and only the Xen "microkernel" runs in ring 0. So if Linux were the dom0, then it would run in ring 1, and guest (domU) kernels would be hypervised by Xen in ring 0 and run in ring 2.

Of course, the "rings" of the processor aren't the only security or isolation mechanism, and the hardware instructions provide a lot of support for safely separating VMs without having to do everything in software. The details, if you want them, are best obtained by reading an Intel processor's programming manual, specifically on the topic of VT-x.

Nested virtualization is not generally possible except in the case of VMware (it is the only hypervisor I'm aware of that can do it). It does this by emulating the VT-x instructions (and possibly EPT) within the guest, thus making the guest think it has real VT-x/EPT support. This is likely a hypervisor trick, though I don't know the implementation details. EPT, however, is often called "Nested Page Tables", so I wonder if the "nested" aspect implies that you can create further levels of nesting than the one level deeper that is required to separate the page tables of the host from the guest(s).

Far more common for nested virtualization is that you're stuck running "fully emulated" or, at best, paravirtualized -- which incurs a great performance hit.

And speaking of performance, I don't think very many people use nested virtualization at all if they can avoid it, except maybe nested container virtualization (which carries zero CPU/memory/hardware overhead). The performance impact, even with VMware's virtualized VT-x, is so great that any potential isolation benefits are wiped out. Trust me, one level of virtualization is enough, until/unless we get to a hardware node where it is actually practical to nest more deeply and be able to eat the performance impact of the levels of indirection.

share|improve this answer
One useful use case for nested visualization that I have personally used was to create a test environment of a Hyper-V server that I could snapshot and roll back. I was able to run a Hyper-V server (win2k8R2) with several VM's running inside a VMware VM running on a windows 7 box. – Scott Chamberlain Dec 1 '12 at 9:21

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.