How to Enable Nested KVM | rdoxenham.

com

http://www.rdoxenham.com/?p=275

rdoxenham.com
Rhys Oxenhams' Cloud Technology Blog

How to Enable Nested KVM

Posted on June 26, 2012
If youve arrived at this blog post Id have to assume youre familiar with what KVM is,
but for the benefit of those who are unaware or are just interested in reading more, Ill
give a bit of a background Kernel-based Virtual Machine (KVM) is a kernel module
that was originally developed by an Israeli organisation called Qumranet to provide
native virtualisation technology for Linux-based platforms; essentially turning the
kernel into a Tier-1 hypervisor. It has since been ported to multiple other platforms
and architectures other than 32/64-bit x86. It got initially adopted into the upstream
Linux kernel as of 2.6.20 (back in 2007).
Typically KVM is designed to run on-top of a bare-metal Linux machine with a CPU
that supports virtualisation extensions, i.e. Intel VMX and AMD SVM. This allowed a
physical machine to run multiple virtual machines on-top (using associated
components such as libvirt and qemu), but theres a new neat technology known as
nested KVM, in other words, KVM support within a KVM-based guest or a hypervisor
within a guest. You may ask the question why do we need this? well, in my
position Im often running into situations where I have to carry out product
demonstrations or debugging hypervisor environments, having another layer of
virtualisation abstraction with nested-KVM is great, especially when on the train or on
a plane!
There are, of course, some performance problems with doing this but for debugging
or wanting to spin up a test environment with technologies such as Red Hat
Enterprise Virtualisation or VMware on a single machine its quite a nice solution. So
lets look at how to enable it first, by default its usually disabled, at least on my
Fedora 16 machine (you can replace Intel with AMD here if you have an AMD-based
processor)
$ cat /sys/module/kvm_intel/parameters/nested
N

1 of 5

11/07/16 06:53

How to Enable Nested KVM | rdoxenham.com

http://www.rdoxenham.com/?p=275

To enable it, we need to make sure the KVM architecture specific module is loaded
with the nested option. There are a few options for enabling it, the first way to do this
is just update your boot loader to specify the nested option; that way it persists with a
reboot or kernel upgrade. Assuming youre using Fedora with GRUB2, (as root) you
need to update the /etc/default/grub file and append kvm-intel.nested=1 to the
GRUB_CMDLINE_LINUX line. For reference, mine is specified below, remember to
replace intel with amd if required.
# cat /etc/default/grub | grep CMDLINE
GRUB_CMDLINE_LINUX="rd.lvm.lv=vol0/swapVol rd.md=0 rd.dm=0 KEYTABLE=us quiet rd
Once this is specified, youll need to rebuild your GRUB configuration files so that
when you next reboot the command line arguments you just specified are loaded,
note the GRUB2 configuration location may be different on a non-Fedora machine
# grub2-mkconfig -o /boot/grub2/grub.cfg
Alternatively, (thanks to Dominic Cleal for the suggestion) this can be modified using
the modprobe configuration files, making things slightly easier$ echo options kvm-intel nested=1 | sudo tee /etc/modprobe.d/kvm-intel.conf
I would now recommend rebooting your machine to verify the changes have been
made. Once again, you can re-run the previous command to check this and you
should see that the changes to the module have been made.
$ cat /sys/module/kvm_intel/parameters/nested
Y
And thats it, youve successfully enabled nested-KVM. Next, when you create new
virtual machines, e.g. with virt-manager, you will need to require vmx or svm to be
presented to the virtual machine; that way the guests can make use of the underlying
nested-KVM features that have been enabled. This can also be done via direct
modification of the libvirt XML definition of a given virtual machine, an example of one
of my VMs is shown below<cpu match='exact'>
<model>Westmere</model>
<feature policy='require' name='vmx'/>
</cpu>
Any questions please feel free to get in touch, Id be happy to help out.
Further reading: https://github.com/torvalds/linux/blob/master/Documentation/virtual
/kvm/nested-vmx.txt
Like 12

Tweet

26

This entry was posted in KVM, Linux by admin. Bookmark the permalink

2 of 5

11/07/16 06:53

How to Enable Nested KVM | rdoxenham.com

http://www.rdoxenham.com/?p=275

[http://www.rdoxenham.com/?p=275] .
9 THOUGHTS ON HOW TO ENABLE NESTED KVM

James Read
on June 26, 2012 at 01:14 said:

Really good article Rhys, Id never heard of a hypervisor with this feature until
now.
At first it strikes you as a really peculiar option you can see why it is
disabled by default. Clearly, in the example you gave, such as debugging a
hypervisor or similar, it would be really useful to run virtual.

Dominic Cleal
on June 26, 2012 at 11:34 said:

Thanks Rhys. Slightly easier than updating GRUB is just to use modprobe:
echo options kvm-intel nested=1 | sudo tee /etc/modprobe.d/kvm-intel.conf
Looking in virt-manager, I see I didnt have the CPU model set up to be the
same as the hosts (theres a simple button under Processor that sets it up). I
wonder how performance would have been affected with/without the extra
SSE3/4 type extensions that its now enabled?

Dale Macartney
on June 26, 2012 at 12:15 said:

Great write up. WIll definitely be putting this to the test.

Keep it up

3 of 5

11/07/16 06:53

How to Enable Nested KVM | rdoxenham.com

http://www.rdoxenham.com/?p=275

admin
on June 26, 2012 at 13:19 said:

Thanks for the alternative method, Dom!

Sounds like you have some performance testing to do

Tim Jackson
on June 27, 2012 at 09:38 said:

Thanks for the tip Rhys, looking forward to trying this one out for local test
deployments etc.!

Bhaskar Chowdhury
on September 6, 2012 at 12:06 said:

Wonderful! wasnt aware of it.Thanks.

Rapha
on May 31, 2013 at 18:28 said:

Thanks for the great guide.

Im trying this in my ubuntu 12.02 but after run:
echo options kvm-intel nested=1 | sudo tee /etc/modprobe.d/kvm-intel.conf

and reboot seems like nested dont be able yet.

The cat /sys/module/kvm_intel/parameters/nested returns N (instead of Y).
Anyone can help me and point some direction?
Thanks

4 of 5

11/07/16 06:53

How to Enable Nested KVM | rdoxenham.com

http://www.rdoxenham.com/?p=275

admin
on June 5, 2013 at 19:21 said:

@Rapha, this guide was written for Fedora/Red Hat based distributions.
Ubuntu doesnt have a modprobe.d configuration directory. Youll need to
make the necessary changes to the correct file, see http://askubuntu.com
/questions/51226/how-to-add-kernel-module-parameters for details.

Alex
on October 10, 2013 at 16:16 said:

Just to be clear, this is all done on the bare-metal host, right?

Ive been trying to get KVM guests to work on a rented KVM host, but that
host in itself is of course a KVM guest on the companys bare-metal server.
So from this post, I understand that it is they who have to enable nested
KVM, to expose it to me (for the befenfit of the guests I want to run).
Thanks

5 of 5

11/07/16 06:53