VMs,

Unikernels and Containers:

Experiences on the Performance of
Virtualiza=on Technologies

Felipe Huici, Filipe Manco, Jose Mendes, Simon Kuenzer

NEC Europe Ltd. (Heidelberg)
In the Beginning…

VM
 In the Beginning…

“Tinyfied
VMs”

VM
 In the Beginning…

“Tinyfied
VMs”

unikernels
VM
 In the Beginning…

“Tinyfied
VMs”
containers

unikernels
VM
 In the Beginning…

“Tinyfied
VMs”
containers

unikernels
VM
 Virt. Technology Benchmarking
•  Metrics:
–  VM Image and memory consump=on
–  VM crea=on =me
–  Delay
–  Throughput
 Virt. Technology Benchmarking
•  Metrics:
–  VM Image and memory consump=on
–  VM crea=on =me
–  Delay
–  Throughput

higher lower
overhead overhead
 Virt. Technology Benchmarking
•  Metrics:
–  VM Image and memory consump=on
–  VM crea=on =me
–  Delay
–  Throughput

higher lower
overhead overhead
 Virt. Technology Benchmarking
•  Metrics:
–  VM Image and memory consump=on
–  VM crea=on =me
–  Delay
–  Throughput

higher lower
overhead overhead
 Virt. Technology Benchmarking
•  Metrics:
–  VM Image and memory consump=on
–  VM crea=on =me
–  Delay
–  Throughput

higher lower
overhead overhead
 Virt. Technology Benchmarking
•  Metrics:
–  VM Image and memory consump=on
–  VM crea=on =me
–  Delay
–  Throughput

higher lower
overhead overhead
 Virt. Technology Benchmarking
•  Metrics:
–  VM Image and memory consump=on
–  VM crea=on =me
–  Delay

?
–  Throughput

higher lower
overhead overhead
 Virt. Technology Benchmarking
•  Metrics:
–  VM Image and memory consump=on
–  VM crea=on =me
–  Delay
–  Throughput

higher lower
overhead overhead
 Virt. Technology Benchmarking
•  Metrics:
–  VM Image and memory consump=on
–  VM crea=on =me
–  Delay
–  Throughput

?
higher lower
overhead overhead
 Virtualiza=on Technology
Benchmarking
•  Metrics:
–  VM image and memory consump=on: ls, top, xl
–  VM crea=on =me: SYN flood + RST detec=on
–  Throughput: iperf, guest to host (TCP traffic)
–  RTT: ping flood

•  VM-based tests run on both Xen and KVM

•  Hardware: x86_64 server with an Intel Xeon E5-1630

v3 3.7GHz CPU (4 cores), 32GB RAM.

 Virtualiza=on Technologies
•  “Standard” VM
–  Standard Debian-based Linux VM
•  “Tinyfied” VM
–  Tinyx, based on Linux kernel/busybox
•  Unikernel
–  On Xen: MiniOS + miniperf
–  On KVM: OSv + iperf
•  Containers
–  Docker

 Virtualiza=on Technologies
•  “Standard” VM
–  Standard Debian-based Linux VM
•  “Tinyfied” VM
–  Tinyx, based on Linux kernel/busybox
•  Unikernel
–  On Xen: MiniOS + miniperf
–  On KVM: OSv + iperf
•  Containers
–  Docker

Standard VM: Applica=on on Top of Distro
User Applica5on

3rd Party Applica5ons

Libraries

Services

Kernel
 Most of the VM not Used…
Nginx User Applica5on

memcached bash 3rd Party Applica5ons

libssl Libraries
libc

ssh
init Services

ext4 netfront
blkfront Kernel
 Tinyx: Keep Only What’s Needed
Nginx User Applica5on

memcached bash 3rd Party Applica5ons

libssl Libraries
libc

ssh
init Services

ext4 netfront
blkfront Kernel
 Tinyx: Taylor-made Distro
Nginx User Applica5on

memcached

bash 3rd Party Applica5ons

libssl Libraries
libc

ssh
init Services

netfront
blkfront Kernel
ext4
 Tinyx: Taylor-made Distro
Nginx User Applica5on

memcached

bash 3rd Party Applica5ons

libssl Libraries
libc

ssh
init Services

netfront
blkfront Kernel
ext4
 Tinyx: Taylor-made Distro
Nginx User Applica5on

memcached

bash 3rd Party Applica5ons

libssl Libraries
libc

ssh
init Services

netfront
blkfront Kernel
ext4
 Tinyx: Taylor-made Distro
Nginx User Applica5on

memcached

bash 3rd Party Applica5ons

libssl Libraries
libc

ssh
init Services

netfront
blkfront Kernel
ext4
 Tinyx: Taylor-made Distro
Nginx User Applica5on
▌ Keep only the
necessary bits
memcached and pieces
bash 3rd Party Applica5ons l Specialized
kernel build
containing only
the necessary
modules
libssl Libraries l Root filesystem
libc populated with
only necessary
ssh services, libraries
and 3rd party
init Services
applications

netfront
blkfront Kernel
ext4
 Virtualiza=on Technologies
•  “Standard” VM
–  Standard Debian-based Linux VM
•  “Tinyfied” VM
–  Tinyx, based on Linux kernel/busybox
•  Unikernel
–  On Xen: MiniOS + miniperf
–  On KVM: OSv + iperf
•  Containers
–  Docker

 Virtualiza=on Technologies
•  “Standard” VM
–  Standard Debian-based Linux VM
•  “Tinyfied” VM
–  Tinyx, based on Linux kernel/busybox
•  Unikernel
–  On Xen: MiniOS + miniperf
–  On KVM: OSv + iperf
•  Containers
–  Docker

 What’s a Unikernel?
•  Specialized VM: single
applica=on +
minimalis=c OS
•  Single address space,
co-opera=ve scheduler
so low overheads
 What’s a Unikernel?
•  Specialized VM: single
applica=on +

USER SPACE
minimalis=c OS

app 1

app N
app 2

•  Single address space,

co-opera=ve scheduler
so low overheads

KERNEL SPACE

driverN
driver1

driver2
GENERAL-PURPOSE
OPERATING SYSTEM
(e.g., Linux, FreeBSD)
 What’s a Unikernel?
•  Specialized VM: single
applica=on +

USER SPACE
minimalis=c OS

app 1

app N
app 2

•  Single address space,

co-opera=ve scheduler
so low overheads

KERNEL SPACE
app

SINGLE ADDRESS
Vdriver1

SPACE
vdriver2
driverN
driver1

driver2
GENERAL-PURPOSE MINIMALISTIC
OPERATING SYSTEM OPERATING SYSTEM
(e.g., Linux, FreeBSD) (e.g., MiniOS, OSv)
Unikernels for Benchmarking
apps

On Xen guest
OS

Xen
Unikernels for Benchmarking
apps iperf

On Xen guest mini

OS OS

Xen Xen
Unikernels for Benchmarking
apps iperf

On Xen guest mini

OS OS

Xen Xen

apps
On KVM

guest
OS

KVM
Unikernels for Benchmarking
apps iperf

On Xen guest mini

OS OS

Xen Xen

apps iperf
On KVM

guest OSv
OS

KVM KVM
 Nota Bene…
•  Our unikernel numbers include op=miza=ons
to the underlying virtualiza=on plamorms
(Xen, KVM)
–  Toolstacks
–  Back-end stores
–  Hotplug scripts
–  Network drivers (on Xen Tx)
•  No =me to go over these…
RESULTS
Image Size, Memory Usage (log scale)
1000
913 913 img size
mem usage
100
112
82
61
MB

52
31 30
10
12
8
3.8 3.7 3.5
2
1
 Boot Times (log scale)
10000
6500
2988
1000 1711
Boot Time (ms)

1081
431 330
100

31
10

1
 RTT
40
35
30 34

25
RTT (ms)

20
15 19 18
15
10
5 9
5 4
0
 Throughput
60
Tx
Throughput (Gb/s)

50 Rx
40

30

20

10

0
 Conclusions
•  Common lore: VMs provide good isola=on but
are heavyweight
–  Results with standard VMs confirm this
•  Containers provide lighter-weight
virtualiza=on
–  But =nyfied VMs and especially unikernels yield
comparable performance
 Conclusions
•  Common lore: VMs provide good isola=on but
are heavyweight
–  Results with standard VMs confirm this
•  Containers provide lighter-weight
virtualiza=on
–  But =nyfied VMs and especially unikernels yield
comparable performance
 Poten5al Contribu5ons to
draT-natarajan-nfvrg-containers-for-nfv-01

2.1.1 Challenges
- VNF provisioning =me
- Run=me performance (throughput, scaling up/down)

3. Benefits of Containers
- Service agility vs VMs
- Containers have beoer run=me performance
- Auto-scaling of VNFs
- Cross-VNF compa=bility: container unikernel/minimalis=c distro
- Overall performance: VMs -25% throughput vs containers

5. Conclusion
- Containers have significant advantages vs hypervisor-based solu=ons