XenServer Storage Management and Troubleshooting, Xenserver, Techedge 2010, Presentation, Video

XenServer Storage Management and
Troubleshooting
Daniel Lazar
Lead Escalation Engineer
May 11, 2010
Agenda
XenServer Storage Overview
Management and Monitoring
Troubleshooting and Diagnosing Common Storage Issues
Q &A
Citrix Confidential - Do Not Distribute

XenServer Storage Objects
SRs, VDIs, PBDs and VBDs
Virtual Disk Data Formats
File-based VHD, LVM and StorageLink
What is an SR (Shared Repository)?
Describes a particular storage target in which Virtual Disk

Images (VDIs) are stored.
Flexiblesupports a wide variety of storage types.
Centralizedeasier to manage, more reliable with a
XenServer pool.
Must be accessible to each XenServer host.
VDIs, PBDs, VBDs
Virtual Disk Images are a storage abstraction that is

presented to a VM.
Physical Block Devices represent the interface between a
physical server and an attached SR.
Virtual Block Devices are connector objects that allow
mappings between VDIs and VMs.

SR
XenServer Host
PBD
VDI
VBD
Virtual Machine
XenServer Host
PBD
VDI
VBD
Virtual Machine
XenServer Host
PBD
VDI
VBD
File-based VHD
VM images are stored as thin-provisioned VHD format files

on either a local non-shared file system (EXT type SR) or a
shared NFS target (NFS type SR).
What is VHD?
A Virtual Hard Disk (VHD) is a file formatted to be structurally identical to a

physical Hard Disk Drive.
Image Format Specification was created by Microsoft in June, 2005.
Logical Volume (LVM)-based VHDs
The default XenServer block device-based storage inserts a

Logical Volume manager on a disk. VDIs are represented
as volumes within the Volume manager.
Introduced LVHD in XenServer 5.5
Enhances LVM for SRs

Hosts VHD files directly on LVM volumes
Adds Advanced Storage features like Fast Cloning and Snapshots
Fast and simple upgrade
Backwards compatible
StorageLink (LUN per VDI)
LUNs are directly mapped to VMs as VDIs by SR types that

provide an array-specific plug-in (NetApp, Equallogic or
StorageLink type SRs). The array storage abstraction
therefore matches the VDI storage abstraction for
environments that manage storage provisioning at an array
level.
StorageLink Architecture
XenServer calls direct to Array APIs to

provision and adjust storage on demand.
Fully leverages array hardware capabilities.
Virtual disk drives are individual LUNs.
High performance storage model.
Only the server running a VM connects to
the individual LUN(s) for that VM.
A special master server coordinates which
servers connect to which LUNs
LVM vs. StorageLink

XenServer
XenServer 5.5
5.5
iSCSI
iSCSI // FC
FC
XenServer
XenServer 5.5
5.5
iSCSI
iSCSI // FC
FC
Storage Repository
Storage Repository
LUN
VHD
VHD header
header
VHD
VHD header
header
LVM
Logical
Volume
LVM
Logical
Volume
LUN
LUN
LVM Volume Group

VM Virtual Disk
LUN
LUN
Storage Management and

Monitoring
Management and Monitoring Overview

Understanding how XenServer Perceives the Storage
Monitoring Storage
Protecting Your Data
Understanding the physical disk layout
# fdisk l
# Lists the physical block devices on the host
Disk /dev/cciss/c0d0: 146.7 GB, 146778685440 bytes

255 heads, 32 sectors/track, 35132 cylinders
Units = cylinders of 8160 * 512 = 4177920 bytes
Device
Start
End
Blocks
Id
System
981
4002464
83
Linux
/dev/cciss/c0d0p2
982
1962
4002480
83
Linux
/dev/cciss/c0d0p3
1963
35132
135333600 83
Linux
/dev/cciss/c0d0p1
Boot
Denotes a SCSI block device

locally attached to the system
(HP RAID array in this case)
The first partition

on the disk
contains the boot
information for the
OS.
Understanding the physical disk layout (continued)
# fdisk l
# Continued output
Disk /dev/sda: 107.3 GB, 107374182400 bytes

255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sda doesn't contain a valid partition table
Implies a block device using the SCSI

Generic (sg) driver. It is likely
attached via a separate interface such
as iSCSI or FC HBA
This disk is part of a Storage
Repository using an LVM file
system and therefore does
not require a local partition
table.
# sg_map x
# Displays the mapping between Linux sg and regular SCSI devices
/dev/sg0
0 0 0 0
13
/dev/sg1
0 0 0 1
/dev/sda
/dev/sg2
0 0 0 2
/dev/sdb
/dev/sg3
1 0 0 0
13
/dev/sg4
1 0 0 1
/dev/sdc
/dev/sg5
1 0 0 2
/dev/sdd
Host
Number
Bus
SCSI
ID
LUN
SCSI
Type
# ll /dev/disk/by-id
# List the attached block devices by SCSI ID.
cciss-3600508b1001035373120202020200003 -> ../../cciss/c0d0

cciss-3600508b1001035373120202020200003-part1 -> ../../cciss/c0d0p1
scsi-360a98000503350642f4a553833616b57 -> ../../sda
This SCSI device is

mapped to /dev/sda
Unique ID assigned by
udev. It corresponds to
individual block devices.
To identify a specific SR
based on the SCSI ID,
compare /dev/disk/by-id
with the SR in
XenCenter
LVM-related commands
# pvs
# Lists physical volumes
PV
VG
/dev/sda
VG_XenStorage-40bbf542-b9d9-ffa1-6efe-aa9c56aadd95 lvm2 a-
Linux sg
device
# vgs
Fmt
LVM Volume Group stored on the

physical volume.
Attr
PSize
PFree
99.99G
59.88G
SR UUID
# Lists volume groups
VG
#PV #LV #SN Attr
VSize
VFree
VG_XenStorage-40bbf542-b9d9-ffa1-6efe-aa9c56aadd95
99.99G
59.88G
wz--n-
LVM (continued)
# lvs
# Lists the logical volumes
LV
VG
Attr
LSize
VHD-c67a887f-3a1a-41f4-8d40-1b21f6307c4a VG_XenStor...
-wi---
24.00G
VHD-c9b919a7-b93b-49ea-abe5-00acb8240cf5 VG_XenStor...
-wi-ao
8.00G
VHD-f3d26dde-254f-4d80-a3bb-d993e904bd63 VG_XenStor...
-wi---
24.00G
LV-e056f479-b0f3-49f3-bc5d-6c226657ae6c
-wi-ao
10.00G
VG_XenStor...
LV-ebdcad46-66d9-4020-baa1-0d5b6ac439c7
Represents Logical Volume containersVG_XenStor...
for individual VDIs.

The a and o
attributes indicate
the LV is active
and open implying
it is attached to a
running VM
Tip: Type lvm help for a

-wi-ao
complete24.00G
list of LVM command
options.
Understanding how the physical storage is represented as virtual objects in XenServer using the XenAPI
# xe sr-list type=lvmoiscsi params=name-label,uuid,VDIs,PBDs

# Lists the SRs configured for the pool
name-label ( RW)
: NetApp - iSCSI
uuid ( RO)
: 40bbf542-b9d9-ffa1-6efe-aa9c56aadd95
VDIs (SRO)
: f3d26dde-254f-4d80-a3bb-d993e904bd63; c67a887f-3a1a-41f4...
PBDs (SRO)
: 27d05ffc-07d3-4f02-d265-3594a2179f8f
Note that the VDI UUID is the

same as the logical volume ID.
We will make a note of this UUID
to refer back to.
Using the PBD UUID from this

command output we will query for its
characteristics in the next slide
Understanding how the physical storage is represented as virtual objects in XenServer using the XenAPI (continued)
# xe pbd-list uuid=27d0 params=uuid,sr-uuid,device-config,currently-attached

# List PBD params
uuid ( RO)
: 27d05ffc-07d3-4f02-d265-3594a2179f8f
sr-uuid ( RO): 40bbf542-b9d9-ffa1-6efe-aa9c56aadd95
device-config (MRO): port: 3260; SCSIid: 360a98000503350642f4a553833616b57;

target: 10.12.45.10; targetIQN: iqn.1992-08.com.netapp:sn.135027806
currently-attached ( RO): true
device-config describes all the physical

characteristics of the block device
attached to this PBD. Note the SCSIid as
referenced earlier from /dev/disk/by-id
# xe vdi-list uuid=f3d26dde-254f-4d80-a3bb- params=uuid,sr-uuid,vbd-uuids

# List VDI params
uuid ( RO)
: f3d26dde-254f-4d80-a3bb-d993e904bd63
sr-uuid ( RO): 40bbf542-b9d9-ffa1-6efe-aa9c56aadd95

vbd-uuids (SRO): 69afb055-3b52-57e3-63fa-d26b82a9b01d
This tells us what VBDs are attached to this VDI.

We will use this UUID in the next slide to query for
the VBD characteristics and determine which VM
this disk is attached to.
# xe vbd-list uuid=69afb055-3b52- params=uuid,vm-uuid,vm-name-label,vdiuuid,mode
# List VBD params

uuid ( RO)
: 69afb055-3b52-57e3-63fa-d26b82a9b01d
vm-uuid ( RO): 2c3a0e82-3f96-eab8-4982-db33fdb3bd88
vm-name-label ( RO): Windows 7 Test

vdi-uuid ( RO): f3d26dde-254f-4d80-a3bb-d993e904bd63
This tells us which VM (name

and UUID) this VBD is attached
to, and which VDI it is providing
to the VM.
mode ( RW): RW
Tip: You can issue xe help

<command> to get syntax help for
any xe commands.
Fibre Channel LUN Zoning
Since Enterprise SANs consolidate data from multiple servers and operating systems, many types of traffic and
data are sent through the interface, whether it is fabric or the network.
With Fibre Channel, to ensure security and dedicated resources, an administrator creates zones and zone sets
to restrict access to specified areas. A zone divides the fabric into groups of devices.
Zone sets are groups of zones. Each zone set represents different configurations that optimize the fabric for
certain functions.
WWN - Each HBA has a unique World Wide Name (similar to an Ethernet MAC)
node WWN (WWNN) - can be shared by some or all ports of a device
port WWN (WWPN) - necessarily unique to each port
Fibre Channel LUN Zoning

Pool1
Xen1
Xen1
Pool2
Xen2
Xen2
Xen3
Xen3
FC Switch
Storage
FC Switch example
iSCSI Isolation
With iSCSI type storage a similar concept of isolation as fibre-channel zoning can be achieved by using IP
subnets and, if required, VLANs.
IQN Each storage interface (NIC or iSCSI HBA) has configured a unique iSCSI Qualified Name
Target IQN Typically associated with the storage provider interface
Initiator IQN Configured on the client side, i.e. the device requesting access to the storage.
IQN format is standardized:
iqn.yyyy-mm.{reversed domain name} (e.g. iqn.2001-04.com.acme:storage.tape.sys1.xyz)
iSCSI Isolation
Pool1
Xen1
Xen1
Pool2
Xen2
Xen2
Xen3
Xen3
Network Switch
Storage
iSCSI Example
Monitoring XenServer Storage - Alerts
XenServer will generate alerts for certain storage events:

Missing or duplicate IQNs configured
HA state file lost or inaccessible
PBD plug failure on server startup
XenServer can be configured to send alert notifications via

email too.
See the XenServer Administrators Guide for more
information about configuring alerts.
Monitoring XenServer Storage CLI Commands
# iostat k
avg-cpu:
%user
0.12
Device:
# Reports basic I/O stats for devices and partitions

%nice %system %iowait
0.00
0.05
0.09
%steal
%idle
0.02
99.72
tps
kB_read/s
kB_wrtn/s
kB_read
kB_wrtn
cciss/c0d0
4.05
0.52
32.11
164361
10156264
sda
0.11
1.38
1.79
437259
566151
Note: iostat is not a great performance indicator for shared

storage devices because it is unaware of external bottlenecks, for
example the network in the case of iSCSI.
# hdparm t /dev/<device> # Performs timed sequential reads

/dev/cciss/c0d0:
Timing buffered disk reads:
286 MB in
3.00 seconds =
95.19 MB/sec
Has some limitations:

Does not measure non-sequential disk reads.
Does not measure disk write speed
May not be accurate with non-local storage devices since it is
unaware of underlying bus architecture (iSCSI, FC, etc.)
Must be sampled repeatedly over time to get an accurate picture
of I/O read performance.
# dd if=<infile> of=<outfile> # Simple, common block device copy utility

# dd if=/dev/<device> of=/dev/null
1998929+0 records in
if = infile, the source dd reads from.

of = outfile, the target dd writes to.
1998929+0 records out

1023451648 bytes (1.0 GB) copied, 13.8456 seconds, 73.9 MB/s
WARNING: NEVER run dd specifying an active, running VHD as the outfileit

WILL destroy the VM container making it unreadable!!
Monitoring XenServer Storage Additional Tips
iSCSI storage throughput can usually be tied directly to network

performance. If there is slow throughput for an iSCSI storage array,
perform network diagnostics first!!
Many SAN arrays have native logging and monitoring tools that can
identify bottlenecks affecting storage performance.
Refer to the Citrix Knowledge Base for best practices and known
issues relating to storage performance.
http://support.citrix.com/article/CTX121634
Protecting Your Data Backup VM Metadata
Can use xsconsole or the CLI.
For more information relating to using

a Disaster
Recovery
CanXenServer
be used as part of aas
Disaster
Recovery solution,
or, as part of regular
maintenance of the environment.
solution, refer to the Citrix Knowledge
Can be scheduled within xsconsole.
Center:
Makes the SR portable.
Protecting Your Data Exporting VMs
Virtual machines can be exported directly out of XenServer

into XVA files that contain a complete clone of the VM and all
of its attached VDIs.
Can be initiated via XenCenter or from the XenServer CLI.
VM must be offline (shutdown) during export process.
Since it backs up all the VM data it can take a very long time
depending on the size of the VM!
Protecting Your Data Creating VM Snapshots
Snapshots create VDI clones of a VM that can be used for

backup or quickly provisioned into new VMs or templates.
XenServer supports two types in version 5.5
Regular Supports all guest environments, including Linux

Quiesced Takes advantage of Windows Volume Shadow Copy Service (VSS).
It requires the manual installation of in-guest components to enable.
Protecting Your Data Creating VM Snapshots (continued)

New in XenServer 5.6!
Introduces snapshot Revert, a.k.a. Checkpoint.

Introduces a new snapshot mode: Snapshot with disk and
memory
XenCenter GUI enhanced for easier management of VM
snapshots and to support Checkpoint feature.
Protecting Your Data Third-Party Solutions
There are also Third-Party backup options:
In-guest backups can be performed using any guest-supported solution (backup

agents running in Windows or Linux, for example).
Volume snapshots performed directly on the storage via StorageLink plugins
(for Dell and NetApp).
Backup solutions that plug into the XenAPI to capture VM data, or clone the
LVM data directly.
Troubleshooting and Diagnosing

Common Storage Issues
Troubleshooting XenServer Storage
Native Troubleshooting Tools XenServer Logs
Always check the logs first! XenServer creates several logs

that are useful for diagnosing storage problems
/var/log/messages
/var/log/xensource.log
/var/log/SMlog
# General messages and system related stuff

# Logging specific to XenAPI
# Logging specific to XenServer storage manager
Often errors logged in any of these files can be searched for in the Citrix
Knowledge Center for a solution. See http://support.citrix.com.
Native Troubleshooting Tools XenAPI commands
The XenAPI (xe) can be used to troubleshoot storage issues too

# xe sr-scan
# Force XAPI to sync the database with local VDIs present in

the underlying substrate.
# xe sr-probe # Using device-config parameters you can probe a block device
for its characteristics, such as existing VM metadata and SR
uuid.
# xe pbd-plug/unplug # Manually plug or unplug a PBD for an SR. This can be
useful when repairing an SR in XenCenter fails.
Native Troubleshooting Tools VHD commands
See and verify mount point of VHD SR

# /var/run/sr-mount/<SR UUID>
full provision VHD SR
vhd-util
See http://support.citrix.com/article/CTX118842
Check VHD architecture
# hexdump -vC <VDI-UUID>.vhd | less
Storage Multipathing
Ensure that multipathing is enabled if you have multiple paths zoned to the XenServer
Use sg_map x and check the host and bus IDs
Problems if you do not enable multipath

I/O Errors
Decrease in performance
Introduce errors with SR.create
What is multipath.conf vs multipath-enabled.conf

multipath.conf is symlink to multipath-enable.conf or multipath-disabled.conf
DMP vs. MPP multipathing
http://support.citrix.com/article/ctx121364
SAN Debugging
Always start at the hardware

adapter, use the Qlogic or
Emulex CLI tools to verify the
LUNs known to the adapter
For QLogic, run scli
For Emulex, run hbanywhere
Use xe sr-probe
type=lvmohba to trigger a bus
refresh
Additional Scenarios
Unable to create SRs
Verify that XenServer can see the storage/LUN

Use fdisk and /dev/disk/xxx
Verify that HBA can see the LUN
Use the HBA CLI tools
Verify that iSCSI can login:
# iscsiadm m node L all # Will force iscsid service to log into the storage array.
Clearing the device mappings via CMD line

# echo 1 > /sys/class/scsi_devices/x:x:x:x/device/delete
Be extremely careful what device is being deleted!
Clean up of orphaned VDIs, XC not displaying the right amount of free storage
If a logical volume has no corresponding VDI it can be deleted. Be extremely careful with this
because if you delete a parent disk, then you lost all differentiated disks.
Questions? Comments?
TechEdge Survey, Video Postings & PPTs

The TechEdge survey will be emailed out
to end-user customers
If you complete the survey, you will be
entered to win a $250 Amazon gift card.
The winner will be announced June 1st.
View TechEdge videos & PPTs on the
Knowledge Center by Monday, May17th
http://support.citrix.com/techedge2010

XenServer Storage Management and Troubleshooting, Xenserver, Techedge 2010, Presentation, Video

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

XenServer Storage Management and Troubleshooting, Xenserver, Techedge 2010, Presentation, Video

Hochgeladen von

Copyright:

Verfügbare Formate

XenServer Storage Management and

Citrix Confidential - Do Not Distribute

XenServer Storage Overview

XenServer Storage Overview

Virtual Disk Data Formats

File-based VHD, LVM and StorageLink

Citrix Confidential - Do Not Distribute

XenServer Storage Objects

What is an SR (Shared Repository)?

Describes a particular storage target in which Virtual Disk

Citrix Confidential - Do Not Distribute

XenServer Storage Objects

VDIs, PBDs, VBDs

Virtual Disk Images are a storage abstraction that is

Citrix Confidential - Do Not Distribute

XenServer Storage Objects

Virtual Disk Data Formats

VM images are stored as thin-provisioned VHD format files

A Virtual Hard Disk (VHD) is a file formatted to be structurally identical to a

Citrix Confidential - Do Not Distribute

Virtual Disk Data Formats

Logical Volume (LVM)-based VHDs

The default XenServer block device-based storage inserts a

Enhances LVM for SRs

Virtual Disk Data Formats

StorageLink (LUN per VDI)

LUNs are directly mapped to VMs as VDIs by SR types that

Citrix Confidential - Do Not Distribute

Virtual Disk Data Formats

XenServer calls direct to Array APIs to

Citrix Confidential - Do Not Distribute

LVM vs. StorageLink

LVM Volume Group

Storage Management and

Management and Monitoring Overview

Citrix Confidential - Do Not Distribute

Management and Monitoring

Understanding the physical disk layout

# Lists the physical block devices on the host

Disk /dev/cciss/c0d0: 146.7 GB, 146778685440 bytes

Denotes a SCSI block device

Citrix Confidential - Do Not Distribute

The first partition

Management and Monitoring

Understanding the physical disk layout (continued)

Disk /dev/sda: 107.3 GB, 107374182400 bytes

Disk /dev/sda doesn't contain a valid partition table

Citrix Confidential - Do Not Distribute

Implies a block device using the SCSI

Management and Monitoring

Understanding the physical disk layout (continued)

# Displays the mapping between Linux sg and regular SCSI devices

Management and Monitoring

Understanding the physical disk layout (continued)

# List the attached block devices by SCSI ID.

cciss-3600508b1001035373120202020200003 -> ../../cciss/c0d0

This SCSI device is

Management and Monitoring

Understanding the physical disk layout (continued)

Citrix Confidential - Do Not Distribute

Management and Monitoring

# Lists physical volumes

LVM Volume Group stored on the