Memory Overcommitment and Memory Reclamation Techniques

Before diving into memory overcommitment or memory contention I would like to first
show the hypervisor architecture so that it becomes easy to understand concepts
described below.
------------VM-------------
Guest Virtual memory (VA)

Application
O.S
Virtual Hardware

Guest Physicl Memory (GA)

VM Kernel

Host Physical Memory (HA)

Physical RAM

<--------------Hypervisor-----------------------
VA is mapped directly to GA and GA in turn is mapped to HA.
Shadow Page Table: Shadow page tables are used by the hypervisor to keep track of
the state in which the guest "thinks" its page tables should be. The guest can't be allowed
access to the hardware page tables because then it would essentially have control of the
machine. So, the hypervisor keeps the "real" mappings (guest virtual -> host physical) in
the hardware when the relevant guest is executing, and keeps a representation of the
page tables that the guest thinks it's using "in the shadows,".
vSwap: For every VM there exists a vswap file (created when vm is powered on) and it is
used when Esxi host memory is not sufficient for satisfying the memory needs of the VM.
Pagefile: Pagefile exists inside the guest O.S and is only used when VMs memory is not
sufficient to satisfy Applications need.
What is Memory Overcommitment in Esxi Server?
ESXI is said to be memory overcommitted when VMs are powered on such that their total
configured memory size is greater than ESXI memory
Alex_Hunt (www.alexhunt86.wordpress.com)

Memory OverCommitment in Esxi Server

Introduction
Memory overcommitment in ESXI is very similar to that in traditional operating systems
(OS) such as Linux and Windows. In traditional OSes, a user may execute applications,
the total mapped memory of which may exceed the amount of memory available to the
OS. This situation is known as memory overcommitment. If the applications consume
memory which exceeds the available physical memory, then the OS reclaims memory
from some of the applications and swaps it to a swap space. It then distributes the
available free memory between applications.
Similar to traditional OSes, ESXI allows VMs to power on with a total configured memory
size that may exceed the memory available to ESXI. The memory installed in an ESXI
Server is called ESXi memory. If VMs consume all the ESXI memory, then ESXI will
reclaim memory from VMs. It will then distribute the ESXI memory, in an efficient and
fair manner, to all VMs such that the memory resource is best utilized.
A simple example of memory overcommitment is when two 4GB VMs are powered on in
an ESXI Server with 4GB of installed memory. The total configured memory of poweredon VMs is 2 * 4 = 8GB, while ESXI memory is 4GB.
Memory overcommitment enables a higher consolidation ratio in a hypervisor. Using
memory overcommitment, users can consolidate VMs on a physical machine such that
physical resources are utilized in an optimal manner while delivering good performance.
For example in a virtual desktop infrastructure (VDI) deployment, a user may operate
many Windows VMs, each containing a word processing application. It is possible to
overcommit a hypervisor with such VDI VMs.
Since the VMs contain similar OSes and applications, many of their memory pages may
contain similar content. The hypervisor will find and consolidate memory pages with
identical content from these VMs, thus saving memory. This enables better utilization of
memory and enables higher consolidation ratio.
Memory overcommitment in ESXI is reliable. This implies that VMs will not be
prematurely terminated or suspended owing to memory overcommitment. Memory
overcommitment in ESXI is theoretically limited by the overhead memory of ESXI. ESXI
guarantees reliability of operation under all levels of overcommitment.
Alex_Hunt (www.alexhunt86.wordpress.com)

Memory OverCommitment in Esxi Server

When ESXI is memory overcommitted, it allocates memory to those powered-on VMs

that need it most and will perform better with more memory. At the same time, ESXI
reclaims memory from those VMs which are not actively using it.
Memory reclamation is therefore an integral component of memory overcommitment.
ESXI uses different memory reclamation techniques to reclaim memory from VMs. There
are 4 memory reclamation techniques: Transparent page sharing, Memory
ballooning, Memory compression, and Host swap to SSD or Memory
swapping.
1: Transparent Page Sharing (TPS): is a passive and opportunistic memory
reclamation technique. It operates on a powered-on VM at all memory states, throughout
its lifetime, looking for opportunities to collapse different memory pages with identical
content into one memory page. This method greatly reduces the memory footprint of VMs
with common memory content.
For example, if an ESXI has many VMs executing word processing applications, then
ESXI may transparently apply page sharing to those VMs and collapse the text and data
content of these applications, thereby reducing the footprint of all those VMs.
The collapsed memory is freed by ESXI and made available for powering on more VMs.
This raises the consolidation ratio of ESXI and enables higher overcommitment. In
addition, if the shared pages are not subsequently written into by the VMs, then they
remain shared for a prolonged time, maintaining the reduced footprint of the VM.
TPS runs regularly even when there is no memory pressure on the host. By default, the
host scans each VM every 60 minutes to find redundant page. After is windows VM
power on, because the guest OS touches all of its memory as it boots up. vSphere
runs TPS over these VMs immediately and doesn't wait until the next cycle. TPS only
scans 4KB memory pages and not 2KB pages.
Other process that runs regularly is the calculation of idle memory tax IMT. IMT is
regularly calculated (every 60 seconds by default) despite the level of free memory on
host. IMT runs all the time but is used only when memory usage is above the High
memory level.

Alex_Hunt (www.alexhunt86.wordpress.com)

Memory OverCommitment in Esxi Server

When the memory state hits High, the hypervisor call TPS immediately even if it is not
due for another run.
2: Memory ballooning is an active method for reclaiming idle memory from VMs. It is
used when ESXI is in the soft state. If a VM has consumed memory pages, but is not
subsequently using them in an active manner, ESXi attempts to reclaim them from the
VM using ballooning.
In this method, non-active pages from VM memory will be moved to VMs pagefile and
GA-HA memory mapping will be deleted. Esxi then allocated those free memory space
which was created due to GA-HA mapping deletion and provides that space to other VMs
which are demanding memory. Ballooning first reclaims free memory pages then it goes
for idle pages and in last it reclaims from active memory pages.
*** Max 65% of memory allocated to VM can be reclaimed using Ballooning.
3: Memory Compression: In this technique there is a small area known as memory
compression area resides in the memory which is allocated to a VM and when this
technique is triggered idle pages of VM is kept inside this area.
Mem Comp Area

---------VMs allocated Memory------

**** Memory Compression Area is only the part of allocated memory to VM.
For E.g. Total allocated memory to a VM is 2 GB then 50 MB may be the size of
compression area. This is not any additional memory provided to a VM.
4: Host swap to cache (aka Host swap to SSD): This is swapping of idle pages of
an Esxi host (not of VM) to an external SSD drive. When this memory reclamation
technique is triggered Esxi starts to swap its idle pages to its swap memory to free some
pages.
Refer this link for more info: http://www.yellow-bricks.com/2011/08/18/swap-to-hostcache-aka-swap-to-ssd/

ESX has an associated memory state which is determined by the amount of free ESX
memory at a given time. The states are high, soft, hard, and low. Table 1 shows the
ESX state thresholds. Each threshold is internally split into two sub-thresholds to avoid

Alex_Hunt (www.alexhunt86.wordpress.com)

Memory OverCommitment in Esxi Server

oscillation of ESX memory state near the threshold. At each memory state, ESX utilizes a
combination of memory reclamation techniques to reclaim memory. This is shown in
Table 2.

Table 1. Free memory state transition threshold in ESX. (a) User visible. (b) Internal
threshold to avoid oscillation.
STATE TPS

BALLOON COMPRESS SWAP

high

soft

hard

low

Table 2. Actions performed by ESX in different memory states.

How memory reclamation techniques are triggered?

For reclaiming memory Esxi uses a memory metric high. So first lets understand how
this metric is calculated
Suppose an Esxi server has 50 GB memory and memory contention has occurred on that
Esxi. How ESXi will use memory metric high to get back memory from the VMs is
explained as follows:
From 0-4 GB= 6% of total physical memory in that Esxi server (50 GB in our example)
From 4-12 GB=4% of total physical memory
From 12-28 GB= 2% of total physical memory

Alex_Hunt (www.alexhunt86.wordpress.com)

Memory OverCommitment in Esxi Server

From 28-50 GB= 1% of total physical memory

So in this case value of high will be (4% + 6% +2% + 1%) = 245.76 Mb + 327. 68 Mb +
327.68 Mb + 225.28= 926.4 Mb High Memory
So when the total free memory on the Esxi server falls below 926.4 Mb then memory
reclamation techniques are triggered. There are 3 threshold values for triggering
different techniques.
Soft= 64% of High
When soft threshold is reached only ballooning technique is used
Hard= 32% of High
When hard threshold is reached memory compression and host swap to SSD techniques
are used
Low= 16% of High
When Low threshold is reached all the memory reclamation techniques are used.

Alex_Hunt (www.alexhunt86.wordpress.com)

Memory OverCommitment in Esxi Server