Troubleshooting Performance Issues

in a Citrix Virtualized Environment

Kapil Ramlal
Sr. Software Maintenance Engineer
Daniel Lazar
Lead Escalation Engineer

Agenda
XenServer Performance Overview
Troubleshooting XenServer Performance
Windows Application Architecture Primer
Troubleshooting Virtual Machine Performance
Citrix Performance VM Demo
Q &A

Citrix Confidential - Do Not Distribute

XenServer Performance Overview

XenServer Performance Overview

XenServer is designed to do one thing

All hypervisors have VM

density limitations
Citrix XenServer

Consolidate machine workloads

Citrix Confidential - Do Not Distribute

Performance is
a function of
VM density

XenServer Performance Overview

How do we determine optimal VM density for a host?

XenServer Hardware
Infrastructure, such as network and storage
Workload and sizing demands of the virtual machines
Native XenServer characteristics

Citrix Confidential - Do Not Distribute

App
App

Toolstack
Toolstack

App
App

Guest
Guest OS
OS
netback
netback

netfront
netfront

App
App

App
App

Guest
Guest OS
OS
netfront
netfront

Native
Native
Driver
Driver

Dom0
Dom0

DomU
DomU

Xen
Xen Hypervisor
Hypervisor

Host
Host Machine
Machine (Hardware)
(Hardware)

DomU
DomU

XenServer Performance Overview

External Factors

Network
Storage
VM Workload and Sizing

Citrix Confidential - Do Not Distribute

Troubleshooting XenServer Performance

Domain 0 Memory Management

for Dom0
for DomU

6346
352
400
752
MB
xn
328
6 MB

total Dom0
) = memory
DomU
required
+ ( for
= Total
DomU
footprint
for n VMs
for DomU

Total XenServer
for
DomU
Memory
Pool
Dom0 Memory
(ex. 12GB)
752MB allows for about
Pool

60 VMs-per-host

Troubleshooting XenServer Performance

What happens when we start more VMs than Dom0 has
memory to manage?

Slow VM performance, poor user experience.

Slow response from XenAPItakes longer to process tasks like starting,
shutting down and migrating virtual machines.
It can cause XenServer host instability resulting in unpredictable behavior and
potentially crashing the XenServer host machine!!

Citrix Confidential - Do Not Distribute

Troubleshooting XenServer Performance

There are two common ways to monitor performance in
XenServer
XenCenter Performance Tab

XenServer Command Line Interface

Citrix Confidential - Do Not Distribute

Troubleshooting XenServer Performance

Using XenCenter

Good for at a glance monitoring

Unwieldy for refined or customized performance testing
Difficult to use for historical trending
Data cannot be easily exported
Some types of information not gathered.

Citrix Confidential - Do Not Distribute

Troubleshooting XenServer Performance

Performance monitoring commands

# top

# Provides a dynamic real-time view of a running system.

Tasks:
Cpu(s):
Mem:
Swap:

68 total,
2 running, 65 sleeping,
0 stopped,
1 zombie
13.0%us, 33.6%sy, 0.0%ni, 1.0%id, 52.5%wa, 0.0%hi, 0.0%si, 0.0%st
417792k total,
302832k used,
114960k free,
68384k buffers
524280k total,
104k used,
524176k free,
80928k cached

PID
12857
4679
5993
1264
4641
4650
12722

USER
65550
root
root
root
root
root
root

PR
15
12
15
16
15
15
15

NI VIRT RES SHR S %CPU %MEM

0 27784 3012 1280 D
19 0.6
-3 281m 16m 5308 S
12 3.4
-3 6164 2276 1188 S
2 0.5
-4 2244 664 384 S
0 0.1
0 16348 1936 952 S
0 0.4
0 12304 652 544 S
0 0.1
0 2188 1052 836 R
0 0.2

TIME+
0:00.57
3:47.85
0:24.73
0:24.00
0:01.25
0:00.05
0:00.03

Citrix Confidential - Do Not Distribute

COMMAND
qemu-dm
xapi
stunnel
udevd
xenstored
blktapctrl
top

Troubleshooting XenServer Performance

Performance monitoring commands

# xentop

# Displays real-time information about a Xen system and domains.

xentop - 17:24:33
Xen 3.3.1
4 domains: 1 running, 3 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 12580820k total, 7092880k used, 5487940k free
CPUs: 8 @ 1600MHz
NAME STATE
CPU(sec) CPU(%)
MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) VCPUS NETS ...
Domain-0 -----r
9849
65.5
417792
3.9
no limit
n/a
8
0 ...
Win2K3-01 -----1
1.5
2097020
16.7
2106164
16.7
2
1 ...
Win2K3-02 -----1
4.3
2097020
16.7
2106164
16.7
2
1 ...
Win2K3-03 -----0
9.8
2097020
16.7
2106164
16.7
2
1 ...

Citrix Confidential - Do Not Distribute

Troubleshooting XenServer Performance

What if I need to increase my VM-density-per-host?

We can tune XenServer to increase VM density. In a scalability study conducted

by Citrix to determine the maximum number of Windows XP virtual desktops per
host for XenServer 5.5 running XenDesktop 4 we were able to host 130 VMs per
host.

Citrix Confidential - Do Not Distribute

Troubleshooting XenServer Performance

This was achieved by making two key configuration changes to a
default XenServer 5.5 installation.

Increased the amount of RAM assigned to Dom0 to 2.94GB from the default
752MB; increasing it enabled us to launch more desktop clients.
Increased the Xen-heap setting to take into account the large number of VMs
on this single server host. This was done by adding "xenheap_megabytes=24"
to the Xen command-line in /boot/extlinux.conf which resulted in an increase
from the default of 16MB to 24MB.

Citrix Confidential - Do Not Distribute

Troubleshooting XenServer Performance

Additional Information

Both the scalability study and instructions for increasing

Dom0 memory limits are documented in the Citrix
Knowledge Center here:

http://support.citrix.com/article/CTX124086 - XenServer Single Server

Scalability with XenDesktop
http://support.citrix.com/article/CTX124259 - Adjusting Dom0 and Xenheap
Setting in XenServer
Disclaimer: Your results may vary! This
testing was done on very high-end equipment
using Citrix best practices!
Citrix Confidential - Do Not Distribute

Troubleshooting XenServer Performance

Troubleshooting commands - Storage

# iostat

# Reports basic I/O stats for devices and

partitions

# hdparm

# Performs timed sequential reads

# dd

# Simple, common block device copy

utility

TIP: iSCSI storage throughput can usually be tied directly

to network performance. If there is slow throughput for an
iSCSI storage array, perform network diagnostics first!!

Citrix Confidential - Do Not Distribute

Troubleshooting XenServer Performance

Troubleshooting commands - Network

# tcpdump

# Dumps traffic on a network

# netstat

# Display network interface statistics

# ifconfig

# Display and configure network interfaces

http://support.citrix.com/article/CTX120869 - detailed instructions for using

tcpdump.

TIP: You can always type man followed by

a Linux command name (i.e., man netstat)
to get detailed help for the command.
Citrix Confidential - Do Not Distribute

Troubleshooting XenServer Performance

Running Shell Scripts

Can capture customized data sets

Can be run over defined periods of time
Can be formatted specifically for reporting purposes.
Requires knowledge of Linux and shell scripting languages.

Citrix Confidential - Do Not Distribute

Troubleshooting XenServer Performance

Additional Information

On the Citrix Knowledge Center you can find shell script

examples, procedures and best practices for how to
troubleshoot all aspects of a XenServer environment.
Some useful links to troubleshooting articles:
http://support.citrix.com/article/CTX124157
http://support.citrix.com/article/CTX121634
http://support.citrix.com/article/CTX122806
http://support.citrix.com/article/CTX120737

Citrix Confidential - Do Not Distribute

Windows Application
Architecture Primer

A process, in the simplest terms,

is an executing program.
- Microsoft (2010)

Citrix Confidential - Do Not Distribute

Application Basics
An application consists of one or more processes
Each process provides the resources needed to execute a
program
One or more threads run in the context of the process
Each process is started with a single thread, often called the
primary thread, but can create additional threads

Application Basics
A thread is the basic unit to which the operating system
allocates processor time
Threads carry out the work of a process
All threads of a process share its virtual address space and
system resources
Uses stack-based storage for handling data

Application Basics

Thread Stack
Frame 0

What is the Stack?

Its temporary memory used by threads

Its used to store functions parameters
and Local variables

Frame 1
Frame 2
Frame 3

Local Variables
Frame Pointer

Saved Frame pointer

Return Address
Function Parameters

A closer look

Citrix Confidential - Do Not Distribute

Application Basics
User & Kernel Space
The Windows operating system can be conceptually divided into 2
parts:
User Space (User Mode)
Kernel Space (Kernel Mode)

Applications run in User Mode

System drivers run in Kernel Mode (Privileged Mode)

USER SPACE
USER
USER
USER MODE
APPLICATION
USER
APPLICATION USER
APPLICATION
APPLICATION USER
USER
APPLICATION
USER
APPLICATION USER
APPLICATION
APPLICATION

KERNEL SPACE
rusb2w2k.sys
keyboard.sys
win32k.sys
tcpip.sys
[]

Troubleshooting Virtual Machine

Performance

Troubleshooting Virtual Machine Performance

Common performance related issues inside the VM:
High CPU
Disk/registry contention
High network utilization
Memory

High CPU
Identify offending Thread (s)
Identify the top function call and its module
Capture user memory dump of offending process for
analysis
Engage respective application vendor
ProcessExplorer can be used for live stack-trace viewing!

The Windows Performance Tools

Next generation performance monitoring from Microsoft
Track CPU usage, application start times, boot issues etc.
Identify common performance problems without a debugger
Included with Windows 7 SDK Download

The Windows Performance Tools: Case Study

ISSUE:
High CPU on wfica32.exe
Methodology
Compare a 30-second sample of activity and compare to
non-working ICA, working ICA and working RDP

The Windows Performance Tools: Case Study

ICA test run where problem occurred
Notice that on this dual processor machine 1 processor is frequently at
or very close to 100%.

Looking inside the above testing to see which instructions were being
executed the most during the test was wfica32.exe.

The Windows Performance Tools: Case Study

Drilling into the calls of wfica32.exe, lead to the Windows function
NtUserSetCursor() which results in calls to the igdkmd32.sys driver
and then into the kernel specifically the memcpy() function.

Memory Dump Collection

User dumps contain a snapshot of a process memory
Kernel dumps contain a snapshot of kernel memory space
A complete memory dump contains both the kernel and the
entire user space

User Dump Collection

Configure a default post-mortem debugger:
How to Set the NT Symbolic Debugger as a Default Windows
Postmortem Debugger (CTX105888)
How to Set WinDbg as a Default Windows Postmortem
Debugger (CTX107528)

Use Task Manager for manual dumps

System Dump Collection

Small Memory Dump
Generally we avoid

Kernel Memory Dump

System crash

Complete Memory Dump

System unresponsive

Control Panel -> System->Advanced Tab -> Startup and Recovery

System Dump Collection

Windows 7 introduced the Dedicated Dump Drive setting
Allows a pagefile to be configured on a dedicated drive for
dump capture
Recommended to debug VMs streamed through PVS

How to Recover Windows Kernel Level Dump Files

from Provisioned Target (CTX123642)

Citrix Confidential - Do Not Distribute

Demo: Citrix XenServer

Performance VM

Q&A

TechEdge Survey, Video Postings & PPTs

The TechEdge survey will be emailed out
to end-user customers
If you complete the survey, you will be
entered to win a $250 Amazon gift card.
The winner will be announced June 1st.
View TechEdge videos & PPTs on the
Knowledge Center by Monday, May17th
http://support.citrix.com/techedge2010