1

Optimizing the Hitachi Virtual Storage Platform in vSphere Environments

Best Practices Guide
By Bob Ingram January 2011

Feedback
Hitachi Data Systems welcomes your feedback. Please share your thoughts by sending an email message to SolutionLab@hds.com. Be sure to include the title of this white paper in your email message.

Table of Contents
Solution Components ............................................................................................................... 4 Hitachi Virtual Storage Platform ............................................................................................... 4 Hitachi Adaptable Modular Storage 2300 ................................................................................ 8 Hitachi Dynamic Provisioning Software ................................................................................... 9 Hitachi Dynamic Tiering ........................................................................................................... 10 Hitachi Universal Volume Manager .......................................................................................... 11 VMware vSphere 4 ................................................................................................................... 11 Storage Configuration Best Practices ..................................................................................... 12 Redundancy ............................................................................................................................. 13 Zone Configuration ................................................................................................................... 14 Storage Host Group Configuration ........................................................................................... 15 Hitachi Dynamic Provisioning Software Best Practices ........................................................... 18 Hitachi Dynamic Tiering Software Best Practices .................................................................... 19 Use Cases for Dynamic Tiering on vSphere ............................................................................ 20 Automatic and Manual Mode.................................................................................................... 27 Distributing Computing Resources Using DRS and Dynamic Provisioning Software ...... 28 Universal Volume Manager ....................................................................................................... 32 Disk Alignment and RAID Stripe Size ...................................................................................... 33 ESX Host Configuration ............................................................................................................ 34 Multipathing .............................................................................................................................. 35 Queue Depth ............................................................................................................................ 35 ESX Host Metrics to Monitor .................................................................................................... 38 SCSI Reservations ................................................................................................................... 38 VMkernel Advanced Disk Parameters ..................................................................................... 39 Scalability Best Practices ......................................................................................................... 40 Conclusion ................................................................................................................................. 41

Optimizing the Hitachi Virtual Storage Platform in vSphere 4 Environments

Best Practices Guide
Todays demanding economic and IT environment makes it more important than ever that IT administrators provide high-performance, scalable, highly available and easy-to-manage infrastructures to maximize both business and IT efficiency. With this goal in mind, organizations are expanding their use of virtualization to more business-critical and mission-critical (tier one and tier two) applications. As they do, the type of storage they deploy to support this environment becomes increasingly important. With industry-differentiated 3D scaling architecture, the Hitachi Virtual Storage Platform is ideal for VMware vSphere environments. 3D scaling enables the platform to scale up, scale out and scale deep in a single storage system, which has important benefits for VMware environments. 3D scaling allows for optimal infrastructure growth in all dimensions: Scale up To meet increasing demands by dynamically adding processors, connectivity and capacity in a single unit, providing the highest performance for both open and mainframe environments. Scale out To meet multiple demands by dynamically combining multiple units into a single logical system with shared resources, support increased demand in virtualized server environments, and ensure quality of service through partitioning of cache and ports. Scale deep To extend storage value by dynamically virtualizing new, existing external storage systems and extend Hitachi Virtual Storage Platform advanced functions to multivendor storage and offload less demanding data to external tiers to optimize the availability of the tier one resources. In addition, when combined with vSpheres round-robin multipathing policy, the Virtual Storage Platform can dramatically simplify setup and management of the virtualized environment, allowing workloads to be automatically balanced across host bus adapters (HBAs), the SAN fabric and the storage systems front-end Fibre Channel ports. Also, when you combine vSphere 4.1, VMwares Dynamic Resource Scheduling (DRS) and Hitachi Dynamic Provisioning software, resource utilization rates improve, which has IT and in turn business benefits. This document describes best practices for deploying vSphere 4 with the Virtual Storage Platform. It is written for storage administrators, vSphere administrators and application administrators who are charged with managing large, dynamic environments. It assumes familiarity with SAN-based storage systems, vSphere and general IT storage practices.

Solution Components
This section describes the hardware and software components mentioned in this best practices guide.

Hitachi Virtual Storage Platform

The Hitachi Virtual Storage Platform is the industrys only 3D scaling storage platform. With the unique ability to concurrently scale up, scale out and scale deep in a single storage system, the new Virtual Storage Platform flexibly adapts for performance, capacity, connectivity and virtualization. No other enterprise storage platform can dynamically scale in three dimensions. The Virtual Storage Platform provides virtual storage that meets the growing demands of server virtualization. The trend in server virtualization is to consolidate the I/O workload of many servers onto a single storage system. As more virtual machines are consolidated onto a physical host, storage systems must be able to dynamically add more storage resources to keep up with I/O demand. The 3D scaling capability of the Virtual Storage Platform meets that requirement. Scaling up allows you to increase virtual server consolidation, improve utilization of resources, and reduce costs. With the Hitachi Virtual Storage Platform, you can increase performance, capacity and connectivity by adding cache, processors, connections and disks to the base system. A virtual server that accesses the storage system can use all these resources, which act as one system managed as a common pool of resources. Scaling out allows you to meet increasing demands by combining multiple chassis into a single logical system with shared resources. By scaling out you can support increased resource needs in virtualized server environments. Scaling deep extends the advanced functions of the Virtual Storage Platform to external multivendor storage. By dynamically virtualizing new and existing storage systems, those systems become part of the Virtual Storage Platforms pool of storage resources. Once virtualized, external data can then be migrated, tiered, replicated and managed by the Virtual Storage Platform. In this manner, older data storage systems can gain a longer useful life. You can extend distance replication for business continuity to lower-cost, lower-function storage systems by virtualizing them behind a Virtual Storage Platform. The switch matrix architecture of the Virtual Storage Platform makes all of this possible. It connects the basic components, front-end directors, back-end directors, global cache modules and virtual storage directors. You can add redundant pairs of directors and cache modules as required without disruption to connected host servers. All these resources are tightly coupled through a global cache that creates a common pool of storage resources. These resources can include external storage that is connected through front-end director initiator ports.

Virtual Storage Platform Architecture

The Virtual Storage Platform offers an entirely new level of scalable enterprise storage, capable of handling the most demanding workloads while maintaining great flexibility. The Virtual Storage Platform offers much higher performance, higher performance scalability, higher reliability and greater flexibility than any storage system on the market today. The Virtual Storage Platform offers these features: The HiStar-E PCI Express Switched Grid acts as the interconnection among front-end directors, back-end directors, data cache adapter boards and virtual storage director boards. Data accelerator processors on the front-end directors and back-end directors work with central processor boards called virtual storage directors that manage all I/O by sets of assigned logical devices (LDEVs). Dual SAS controllers on back-end director boards contain eight 6Gbps SAS links per board. The control memory function resides in global cache and each VSD board contains a local copy with information for its LDEVs. Most control memory accesses are lookups to the local copy. Global cache is backed up to solid state drives (SSDs) on the cache boards. Each virtual storage director board controls all I/O operations for a discrete group of LDEVs. LDEVs are assigned round-robin across the installed virtual storage director boards as they are created. If necessary, you can manually reassign LDEV ownership to a different virtual storage director. Each virtual storage director board executes the code for initiator mode (hosts), external mode (virtualization), back-end director mode, or the copy products send and receive modes. Code execution is done on a per-job basis. A Virtual Storage Platform can be scaled from a single-chassis system to a dual-chassis system. Each chassis has a control rack and a logic box. Up to 1,280 3.5-inch large form factor (LFF) drives or 2,048 2.5-inch small form factor (SFF) drives can be installed in a dual-chassis system. If you install both LFF and SFF disk containers and drives in a storage system, the limits change based on the configuration you choose. The Virtual Storage Platform is built as a single-chassis or dual-chassis storage system. Each chassis has one control rack and up to two disk expansion racks. The control rack has the logic box that holds all of the control boards for a chassis and one or two disk containers. The disk expansion racks can hold three disk containers each. Disk containers come in two types: small form factor (up to 128 2.5inch drives) and large form factor (up to 80 3.5-inch drives). When using two chassis as a single integrated storage system, the two units are cross connected at the grid switch level. The storage system behaves as a single unit, not as a pair of units operating as a cluster.

Figure 1 shows the two types of racks available for a Virtual Storage Platform.
Control Rack Disk Expansion Rack

Figure 1

Figure 2 shows the logic boards in a fully populated single-chassis Virtual Storage Platform.

Figure 2

Figure 2 shows the following chassis components (note that a feature is a pair of boards on two separate power domains): GSW Grid Switch PCI Express Switch. One or two features (two or four boards) per control unit with 24 2GB/sec HiStar-E ports each can be installed DCA Data cache adapter cache memory. One to four features (two, four, six or eight boards) per control unit with up to 32GB of RAM each can be installed VSD Virtual storage director processor module. One or two features (two or four boards) per control unit can be installed. FED Front-end director host port module. One to four features (two, four, six or eight boards) per control unit of four or eight 8Gbps Fibre Channel ports can be installed. BED Back-end director disk controller module. One or two features (two or four boards) per control unit with eight 6Gbps SAS links per board can be installed. If the back-end director options are not installed (available for the single-chassis configuration only), two additional front-end director options can be used in those chassis slots.

3D Scaling Architecture
The Hitachi Virtual Storage Platform allows for optimal infrastructure growth in all dimensions by scaling up, scaling out and scaling deep.

Scale Up
Scale up to meet increasing demands by dynamically adding processors, connectivity and capacity in a single unit, providing the highest performance for both open and mainframe environments. In the basic single chassis configuration, the number of logic boards, disk containers, and drives is highly scalable. You can start with the minimum set of logic boards,10, and one disk container, then add more boards (up to a total of 28 boards in a single chassis) and disk containers (up to a total of eight disk containers in a single chassis). Disk container types may be intermixed within a chassis.

Scale Out
Scale out to meet multiple demands by dynamically combining multiple units into a single logical system with shared resources, support increased demand in virtualized server environments, and ensure safe multitenancy that is, the ability to run multiple servers simultaneously without the risk of corruption or modification of data from one server to another and quality of service through partitioning of cache and ports. You can double the scalability of the Virtual Storage Platform with a dual-chassis system with up to six racks. The logic box in each chassis is the same, using the same types and numbers of logic boards. Any front-end port can access any back-end RAID group; no division within the storage system exists between the chassis. A dual-chassis Virtual Storage Platform can manage up to 247PB of total storage capacity.

Table 1 lists the capacity differences between a single-chassis and a dual-chassis Virtual Storage Platform storage system.
Table 1. Virtual Storage Platform Chassis Capacity Comparison

Maximum Capacity Data cache Raw cache bandwidth Solid state drives 2.5 SFF drives 3.5 LFF drives Logical volumes (LDEVs)

Single Chassis 256GB 64GB/s 128 1,024 640 65,280

Dual Chassis 512GB 128GB/s 256 2,048 1,280 130,560

Scale Deep
Scale deep to extend storage value by dynamically virtualizing new, existing external storage systems, extend Hitachi Virtual Storage Platform advanced functions to multivendor storage and offload less demanding data to external tiers to optimize the availability of your tier one resources. The Virtual Storage Platform provides the virtualization mechanisms that allow other storage systems to be attached to some of its front-end director Fibre Channel ports and accessed and managed via hosts that are attached to the host ports on the Virtual Storage Platform. As far as any host is concerned, all virtualized logical units passed through the Virtual storage Platform to the hosts appear to be internal logical units from the Virtual Storage Platform. The front-end ports on the Virtual Storage Platform that attach to the external storage systems front-end ports are operated in external or SCSI initiator mode (attached to servers), rather than the usual SCSI target mode (attached to hosts). For more information about the Hitachi Virtual Storage Platform, see the Hitachi Data Systems web site.

Hitachi Adaptable Modular Storage 2300

The Hitachi Adaptable Modular Storage 2300, part of the Hitachi Adaptable Modular Storage 2000 family, offers the best combination of price and performance in a model that scales to 240 disk drives. Ideal for large businesses and enterprises, the 2300 is a highly reliable, flexible and scalable storage system. Although Hitachi Data Systems used a 2300 to validate its recommendations, the best practices included in this document apply no matter which member of the 2000 family you use. For more information, see the Hitachi Adaptable Modular Storage 2300 datasheet.

Hitachi Dynamic Provisioning Software

On the Virtual Storage Platform, Hitachi Dynamic Provisioning software provides wide striping and thin provisioning functionalities. In the most basic sense, Hitachi Dynamic Provisioning software is similar to the use of a host-based logical volume manager (LVM), but with several additional features available within the Hitachi Virtual Storage Platform and without the need to install software on the host or incur host processing overhead. Hitachi Dynamic Provisioning software provides for one or more pools of wide striping across many RAID groups within a Virtual Storage Platform. One or more Dynamic Provisioning virtual volumes (DP-VOLs) of a user-specified logical size of up to 60TB (with no initial physical space allocated) are created against each pool. Primarily, you deploy Hitachi Dynamic Provisioning software to avoid the routine issue of hot spots that occur on logical devices (LDEVs) from individual RAID groups when the host workload exceeds the IOPS or throughput capacity of that RAID group. By using many RAID groups as members of a striped Dynamic Provisioning pool underneath the virtual or logical volumes seen by the hosts, a host workload is distributed across many RAID groups, which provides a smoothing effect that dramatically reduces hot spots. Hitachi Dynamic Provisioning software also carries the side benefit of thin provisioning, where physical space is only assigned from the pool to the DP-VOL as needed using 42MB pages, up to the logical size specified for each DP-VOL. A pool can also be dynamically expanded by adding more capacity or reduced by withdrawing pool capacity. Either operation is performed without disruption or requiring downtime. Upon expansion, a pool can be rebalanced so that the data and workload are wide striped evenly across the current and newly added RAID groups that make up the pool. Hitachi Dynamic Provisioning softwares thin provisioning and wide striping functionalities provide virtual storage capacity to eliminate application service interruptions, reduce costs and simplify administration, as follows: Optimizes or right-sizes storage performance and capacity based on business or application requirements. Supports deferring storage capacity upgrades to align with actual business usage. Simplifies the storage administration process. Provides performance improvements through automatic optimized wide striping of data across all available disks in a storage pool. Eliminates hot spots across the different RAID groups by smoothing the combined workload. Significantly improves capacity utilization. For more information, see the Hitachi Dynamic Provisioning software datasheet.

Hitachi Dynamic Tiering

Hitachi Dynamic Tiering is a revolutionary solution that eliminates the time-consuming manual processes of data classification and movement between storage tiers, thus optimizing tiered storage usage while improving performance. It is only available on the Hitachi Virtual Storage Platform. Most data is rarely accessed after it is created. As a result, it should not be stored on your most expensive tier of storage, but instead moved to a lower, less-expensive storage tier. Defining where and for how long data should reside at any point in its life cycle can be complex and problematic. Many organizations use a data tiering approach to manage application performance, manually provisioning space from several storage technologies with different performance and cost characteristics. Using this approach, data specialists typically look to past usage patterns to determine how to manually configure tiering, making the storage infrastructure unable to effectively respond to dynamic application and data use. If usage patterns change rapidly, manually tiered storage systems produce less than optimal results. Hitachi Dynamic Tiering takes the automation of tiered storage to a new level. It enables the management of multiple storage tiers as a single entity. By leveraging the existing features of Hitachi Dynamic Provisioning software, Hitachi Dynamic Tiering presents a new kind of virtual volume with embedded smart tiering that monitors access and moves data at the 42MB page level. Hitachi Dynamic Provisioning software breaks the volume into pages and Hitachi Dynamic Tiering automatically moves infrequently referenced pages to lower cost tiers of storage. Moving pages instead of entire data sets or files reduces the time and storage space required to migrate data. It ensures that the right data is in the right place at the right time. Hitachi Dynamic Tiering automatically moves pages of data within virtual volumes configured on a Dynamic Provisioning pool to the most appropriate media according to workload. This maximizes service levels and minimizes total cost of storage ownership. After an initial setup process, Hitachi Dynamic Tiering monitors data access in real time and makes decisions about moving data between the available storage tiers based on actual use. If a page on a lower tier is accessed frequently, Hitachi Dynamic Tiering moves it a higher tier. Using this approach, Hitachi Dynamic Tiering enables you to improve the availability and performance of your storage systems and the applications using that storage. Previously, each Dynamic Provisioning pool had to be created using one RAID level and one disk type. Hitachi Dynamic Tiering on the Virtual Storage Platform allows a single pool to contain tiers made up of multiple types of RAID groups and any type of disk. Hitachi Dynamic Tiering manages the various tiers within a Dynamic Provisioning pool automatically. This eliminates most user management of storage tiers within a storage system and maintains peak performance under dynamic conditions without storage administrator intervention.

10

Hitachi Universal Volume Manager

Universal Volume Manager enables the Virtual Storage Platform to virtualize external multivendor storage into the Virtual Storage Platforms pool of storage resources, a unique capability in the storage industry. A Universal Volume Manager storage domain controller is a Virtual Storage Platform storage system that is connected to external storage devices and integrates their storage into virtual volumes. This allows central management of multiple storage systems. Unlike other storage virtualization products, the Hitachi Virtual Storage Platform with Universal Volume Manager leverages the capabilities of the most reliable SAN component, the storage system, to provide the most hardened and scalable solution. Externally attached storage can be used for the following purposes: Tier 3 and tier 4 storage Applications with low I/O requirements Replicated copies of data volumes using Hitachi ShadowImage software Copies of application data for backup or test and development Archived data Redeployment of legacy storage systems

VMware vSphere 4
vSphere 4 is a highly efficient virtualization platform that provides a robust, scalable and reliable infrastructure for the data center. vSphere 4 features like DRS, High Availability (HA), and Fault Tolerance (FT) provide an easy-to-manage platform. Use of vSphere 4s round robin multipathing policy distributes load across multiple host bus adapters (HBAs) and multiple storage ports. Use of DRS with Hitachi Dynamic Provisioning software automatically distributes loads on the ESX host and across the storage systems back end.

Storage I/O Control

With vSphere 4.1, VMware introduced the Storage I/O Control feature. Use Storage I/O Control to assign I/O priority to critical virtual machines and throttle the I/O on less-critical machines across the DRS cluster. The service level of critical machines can be protected from I/O contention at the storage system by enforcing a share-based or IOPS-based allocation of I/O resources. Without Storage I/O Control, disk shares are enforced only at an ESX host level, and only when local HBAs are saturated. When Storage I/O Control is enabled, disk shares and IOPS controls are enforced across all the ESX hosts in a cluster accessing the Virtual Machine File System datastore (VMFS). vSphere Storage I/O control can be used with Hitachi Dynamic Provisioning software to allocate storage resources and ensure that service levels are met. For more information about vSphere Storage I/O control see the Storage I/O Control Technical Overview and Considerations for Deployment white paper. For more information about vSphere 4, see VMwares vSphere web site.

11

Storage Configuration Best Practices

The following sections describe best practices for optimizing your Virtual Storage Platform storage infrastructure to meet your performance, scalability, availability and ease of management requirements. Figure 3 shows the SAN topology of a vSphere infrastructure that uses a Virtual Storage Platform with external storage.

Figure 3

12

Redundancy
A scalable, highly available and easy-to-manage storage infrastructure requires redundancy at every level. To take advantage of ESXs built-in multipathing support, each ESX host needs redundant HBAs. This provides protection against both HBA hardware failures and Fibre Channel link failures. When ESX 4.1 hosts are connected in this fashion to a Hitachi Virtual Storage Platform system, hosts can use a round robin multipathing algorithm where the I/O load is distributed across all available paths. Hitachi Data Systems recommends using a minimum of two HBAs for redundancy. All LUNs must be accessible to both HBAs. Fabric redundancy is also crucial. If all of your ESX hosts are connected to a single Fibre Channel switch, failures on that switch can lead to failures on all of your virtual machines stored on the storage system. To prevent a single point of failure in the fabric, deploy at least two Fibre Channel switches or a director class switch. If you use multiple Fibre Channel switches, one HBA from each host is connected to one switch while the second HBA is connected to the second switch. If you use a Fibre Channel director, follow the manufacturers recommendation for connecting multiple HBAs. Hitachi Data Systems recommends using at least two Fibre Channel switches or a Fibre Channel director. Without redundancy, a single point of failure within the storage system can expose all of your virtual machines to that single point of failure. The Virtual Storage Platform provides redundancy with two storage clusters where each storage cluster contains front-end directors. Each front-end director can contain either four or eight Fibre Channel ports. Cluster 1 contains all the odd-numbered Fibre Channel ports and cluster 2 contains all the even-numbered Fibre Channel ports. This protects against Fibre Channel link failures and storage controller failures. If one link fails, another link is available. If one storage controller fails, another storage controller is available. All LUNs need to be mapped to at least two storage ports, one on each storage cluster. Figure 4 shows a configuration containing two ESX hosts, a Virtual Storage Platform system and an Adaptable Modular Storage Platform family system connected to a SAN director with full Fibre Channel path redundancy.

13

Figure 4 Key Best Practice Connect at least two HBAs from each ESX host to two Fibre Channel ports on the Virtual Storage Platform, one from each storage cluster. For example, connect HBA 1 to Fibre Channel port CL1-A and HBA 2 to Fibre Channel port CL2-A.

Zone Configuration
Zoning divides the physical fabric into logical subsets for enhanced security and data segregation. Incorrect zoning can lead to LUN presentation issues to ESX hosts. Two types of zones are available, each with advantages and disadvantages: Port Uses a specific physical port on the Fibre Channel switch. Port zones provide better security and can be easier to troubleshoot than WWN zones. This might be advantageous in a smaller static environment. The disadvantage of this is ESX hosts HBA must always be connected to the specified port. Moving an HBA connection results in loss of connectivity and requires rezoning. WWN Uses nameservers to map an HBAs WWN to a target ports WWN. The advantage of this is that the ESX hosts HBA can be connected to any port on the switch, providing greater flexibility. This might also be advantageous in a larger dynamic environment. However, two disadvantages are the reduced security and additional complexity when troubleshooting. Hitachi Data Systems recommends using single initiator zones, which have one initiator (HBA) with single or multiple targets in a single zone.

14

Table 2 lists an example of ESX hosts with single-initiator zones and multiple paths.
Table 2. Sample Zoning with Multiple Paths per ESX Host

Host ESX1 ESX2

Host HBA HBA 1 Port 1 HBA2 Port 1 HBA 1 Port 1 HBA 2 Port 1

Director Zone Name ESX1_HBA1_1_VSP_CL1-A ESX1_HBA2_1_VSP_CL2-A ESX2_HBA1_1_VSP_CL1-A ESX2_HBA2_1_VSP_CL2-A

Storage Port CL1-A CL2-A CL1-A CL2-A

When connecting external storage to the Virtual Storage Platform, zone the fabric so that the external storage has multiple paths to the Virtual Storage Platform. Table 3 lists an example of external storage with redundant paths.
Table 3. Sample Zoning for External Storage with Redundant Paths

External Storage Adaptable Modular Storage 2300

External Storage Port 0C 1C

Director Zone Name AMS2300_0C_VSP_CL3-A AMS2300_1C_VSP_CL4-A

Storage Port CL3-A CL4-A

Storage Host Group Configuration

To grant a host access to an LDEV, assign a logical unit number (LUN) within a host group. This section describes the settings and LUN mapping for host group configurations.

Fibre Channel Port Options

If a Fibre Channel port is connected via a SAN switch or director, you must change the following settings in Hitachi Storage Navigator software: Port security Set to Enable; allows multiple host groups on the Fibre Channel port. Fibre Channel switch Set to Enable; allows connection to a Fibre Channel switch or director. Connection type Set to P-to-P; allows a point-to-point connection to a Fibre Channel switch or director. Figure 5 shows these settings in the Hitachi Storage Navigator software GUI.

15

Figure 5

Hitachi Data Systems recommends that any configuration applied to a port in cluster 1 also is applied to the port in cluster 2 in the same location. For example, if you create a host group for a host on port CL1-A, also create a host group for that host on port CL2-A. Figure 6 shows the front-end director port names for a Virtual Storage Platform system with two eightport front-end director pairs installed.

Figure 6

16

One Host Group per ESX Host, Standalone Host Configuration

If you plan to deploy ESX hosts in a standalone configuration, each hosts WWNs can be in its own host group. This approach provides granular control over LUN presentation to ESX hosts. This is the best practice for SAN boot environments, because ESX hosts do not have access to other ESX hosts boot LUNs. However, in a cluster environment this approach can be an administration challenge because keeping track of which ESX hosts are in a cluster can be difficult. When multiple ESX hosts need to access the same LDEV for clustering purposes, the LDEV must be added to each host group. This operation is error prone and might lead to confusion. If you have numerous ESX hosts, this approach can be tedious.

One Host Group per Cluster, Cluster Host Configuration

vSphere features such as vMotion, DRS, HA, FT and Storage vMotion require shared storage across the ESX hosts. Many of these features require that the same LUNs are presented to all ESX hosts participating in these cluster functions. If you plan to use ESX hosts with these features, create host groups with clustering in mind. Place all of the WWNs for the clustered ESX hosts in a single host group. This ensures that when LDEVs are added to the host group, all ESX hosts see the same LUNs. This creates consistency with LUN presentation across all hosts.

Host Group Options

On the Virtual Storage Platform, host groups are created using Hitachi Storage Navigator software. The host mode is set for the host operating system. Select the host mode of VMware for all host groups. Figure 7 shows the host mode setting for vSphere environments.

Figure 7

17

Hitachi Dynamic Provisioning Software Best Practices

The following sections describe best practices for using Hitachi Dynamic Provisioning software with vSphere 4.

Dynamic Provisioning Space Saving and Virtual Disks

Two of vSpheres virtual disk formats are thin-friendly, meaning they only allocate pages from the Dynamic Provisioning pool as required. Thin and zeroedthick format virtual disks are thin friendly, eagerzeroedthick format virtual disks are not. Thin and zeroedthick virtual disk format do not zero the VMFS filesystem blocks on creation. When the first write to each VMFS filesystem block occurs, the VMkernel writes zeros to the block to assure that any data previously written to this storage cannot be accessed. This additional write produces additional write overhead, commonly called a warm-up anomaly. The eagerzeroedthick format virtual disk zeroes all VMFS filesystem blocks on creation and therefore allocates 100 percent of the Dynamic Provisioning volumes (DP-VOL) space in the Dynamic Provisioning pool. While an eagerzeroedthick format virtual disk does not give the benefit of cost savings by overprovisioning storage, it preallocates the pages in the Dynamic Provisioning pool across all disks. This preallocation of pages can improve initial performance of the virtual disk. When using DP-VOLs to overprovision storage, follow these best practices: Create the VM template on a zeroedthick format virtual disk. When deploying, select the Same format as source radio button in the vCenter GUI. Use the default zeroedthick format virtual disk. Note that you can use the thin virtual disk format and achieve similar space savings; however, this creates an additional data point to monitor as storage overprovisioning must be monitored from vCenter and the storage. vSphere thin provisioning can also add ESX CPU overhead and SCSI reservation issues. Hitachi Data Systems recommends using Hitachi Dynamic Provisioning software with zeroedthick virtual disks when thin provisioning is required. This ultimately lessens management tasks for data center administrators. Do not convert a thin format virtual disk to thick (eagerzeroedthick) when using Dynamic Provisioning software for cost savings. NOTE: Eagerzeroedthick virtual disk format is required for Fault Tolerance.

18

Virtual Disk and Dynamic Provisioning Performance

Dynamic Provisioning software provides performance improvements through automatic optimized wide striping of data across all available disks in a storage pool. To obtain maximum storage performance for vSphere 4.1 when using the Virtual Storage Platform, follow these best practices: Use eagerzeroedthick virtual disk format to prevent warm-up anomalies. Warm-up anomalies occur one time when a block on the virtual disk is written to for the first time. Zeroedthick is fine for use on the guest OS boot volume where maximum write performance is not required. Size the Dynamic Provisioning pools according to the I/O requirements of the virtual disks and applications. Separate sequential and random workloads on different Dynamic Provisioning pools when large Dynamic Provisioning pools are not possible, Separate the logs from the database on different Dynamic Provisioning pools for applications that use log recovery. This separates the sequential and random workloads and can also help protect data. In the rare case where a dual hardware failure causes the corruption or loss of a Dynamic Provisioning pool, the logs are available for recovery.

Virtual Disk Format Best Practices

Zeroedthick and eagerzeroedthick format virtual disks provide similar performance after the zeroedthick warm-up period. Either virtual disk format provides similar throughput after all blocks are written at least one time, however, zeroedthick initially shows some write latency and lower write throughput. When deciding whether to use zeroedthick or eagerzeroedthick format virtual disks, follow these best practices: If you plan to use vSphere 4s Fault Tolerance on a virtual machine, you must use the eagerzeroedthick virtual disk format. If minimizing the time to create the virtual disk is more important than maximizing initial write performance, use the zeroedthick virtual disk format. If maximizing initial write performance is more important than minimizing the time required to create the virtual disk, use the eagerzeroedthick format.

Hitachi Dynamic Tiering Software Best Practices

In Hitachi Dynamic Tiering software, new page allocation and page reallocation are performed automatically based on data collected by the Virtual Storage Platform. The collection of performance data and reallocation of pages can be triggered automatically or manually. When using Hitachi Dynamic Tiering software, you must keep the following in mind: Tier levels are defined by storage media throughput. Storage media with higher throughput are positioned as higher tiers. Storage media with lower throughput are positioned as lower tiers. The tier with the highest throughput in a Dynamic Tiering pool is identified as tier 1. A maximum of three tiers of storage can be defined in each Dynamic Tiering pool. A tier cannot contain multiple RAID levels.

19

Figure 8 shows the Dynamic Tiering workflow.

Figure 8

Figure 8 illustrates the following Dynamic Tiering steps: Allocate new page New 42MB pages are allocated from the highest tier with space available. By default, SAS and SATA tiers reserve 8 percent for new page allocation. This can be adjusted from 0 to 50 percent through the command line. Monitor Access to the disks is counted and the I/O per hour (IOPH) per page average is entered into a table Determine Optimal Tier Data is analyzed and a page relocation plan is calculated. Relocate Relocate (called reallocate in some commands) is executed one DP-VOL at a time starting with the lowest number DP-VOL. The next relocation cycle continues from the last DP-VOL completed, as follows: If the target tier has insufficient capacity to relocate all the pages requested, these pages are not relocated. Relocation completes when one of the following conditions occurs: All pages that are scheduled to be relocated, and can be, are relocated. Auto cycle time is reached. Pool configuration or parameters are modified. Relocation is canceled by a user. Repeat Several cycles might be needed to complete the relocation.

Use Cases for Dynamic Tiering on vSphere

Virtual machines with very different workloads are often combined in a single DRS cluster in VMware environments. Segmenting these virtual machines on storage tiers is a labor-intensive administrative task. Dynamic Tiering can save you time and money by automatically placing less frequently accessed pages on lower-cost storage, and can improve performance by placing more frequently accessed pages on high-speed storage.

Relocating To Lower Tiers

When a new application or virtual machine is implemented and the I/O workload is unknown, its common for storage administrators to assign a high tier of storage and monitor after the implementation is completed. Dynamic Tiering can automate for this implementation methodology. In this use case, eight virtual machines were tested using Vdbench and a mix of high, medium and low IOPS workloads. The Dynamic Tiering pool contained LDEVs on SAS 10K RPM disks in a RAID-5 (3D+1P) configuration and SATA 7200RPM disks in a RAID-6 (6D+2P) configuration. The initial distribution of pages was fairly equal across the two tiers of disks.

20

Figure 9 shows the initial distribution of pages on tier 1 and tier 2 before Dynamic Provisioning relocated the pages.

Figure 9

Next, Dynamic Tiering manual mode monitoring was started using the command line; it ran for two hours to collect IOPH information. After the monitoring cycle, manual relocation was started and allowed to run to completion, approximately 27 hours. Figure 10 shows the distribution of pages after the relocation process was complete.

Figure 10

21

The IOPS throughput and goals are shown in Figure 11. The highlighted bars show the VM with improved IOPS.

Figure 11

The response times of the virtual machines are shown in Figure 12. The highlighted bars show the VMs with improved response times.

Figure 12

22

Promoting Pages For Performance Enhancement

In this use case, a Dynamic Tiering pool made up of LDEVs from SATA 7200RPM disks in a RAID-6 (6D+2P) configuration was created. Four virtual machines were tested using Vdbench and a mix of high, medium and low IOPS workloads. After IOPS data was collected, SAS 10K RPM disks in a RAID5 (3D+1P) configuration were added to the pool. In addition, Dynamic Tiering in automatic mode was started and was set to run for four hours. Figure 13 shows the throughput achieved before and after the reallocation.

Figure 13

23

Figure 14 shows the response times of the virtual machine I/O before and after reallocation.

Figure 14

Figure 15 shows the distribution of allocated space on SATA and SAS after the reallocation was complete. Note that before relocation, no pages were allocated to SAS disks.

Figure 15

Using a single tier of storage made up of LDEVs on SATA RAID-6 (6D+2P) disks only, virtual machine throughput and response times were unacceptable. Response times on all virtual machines were affected by disk contention. Relocation after adding LDEVs on SAS RAID-5 (3D+1P) disks took approximately 14 hours, after which response times and throughput improved.

24

vSphere Virtual Desktop Infrastructure

Hitachi Dynamic Tiering software benefits a virtual desktop infrastructure by promoting pages used by the linked clone delta file and allowing common files to remain on low-cost disks. In the Hitachi Data Systems lab, tests were performed using 100 clones linked to a single virtual machine. The initial implementation was done on a Dynamic Tiering pool consisting of LDEVs from SATA disks, then SAS disks were added to the pool and automatic mode with a four-hour cycle time was set. Figure 16 shows the distribution of the tiers before and after relocation.

Figure 16

25

Figure 17 and Figure 18 show how performance improved with the addition of SAS disks. Figure 17 shows the total IOPS for the ESX hosts before and after the relocation. Note that some ESX I/O overhead was caused by virtual machine log file I/Os, delta file I/O and other ESX host functions.

Figure 17

Figure 18 shows the total ESX guest millisecond per command. This metric includes all I/O latencies combined.

Figure 18

26

When using multiple media types, follow these best practices: Size the SAS tier for the storage and performance of the most critical applications. For applications with lower performance requirements, use SATA disks, which can provide the following benefits: Relief for the SAS tier by offloading the I/O Additional capacity for applications with lower I/O requirements. You can also add SSD media to the Dynamic Tiering pool. SSD is utilized as tier 1, SAS is utilized as tier 2 and SATA is utilized as tier 3. The addition of SSD to the Dynamic Tiering pool can improve application performance.

Automatic and Manual Mode

Dynamic Tiering can be run in automatic or manual mode. Automatic mode can be set to cycle times of 1, 2, 4, 8 and 24 hours. Manual mode allows for a monitoring cycle of up to seven days and, once started, relocation continues until complete. Advantages of automatic mode: Once set, no user intervention is required. If a 24-hour cycle is used, monitoring can be stopped for a specific period of time. Disadvantages of automatic mode: Cycle times might not be long enough for a complete relocation. The maximum cycle time of 24 hours can be too short for some environments. Advantages of manual mode: Performance collection start and stop can be easily controlled. Reallocation can be started at a specified time instead of by the cycle time. Reallocation runs until either complete or stopped by a command. Disadvantages of manual mode: Requires the use of the command line. Requires user intervention or scripting. Follow these best practices for cycle times: Set the cycle time long enough to collect IOPH on typical workloads. Use manual mode to monitor and relocate for periods of time greater than 24 hours. Note: A short relocation cycle time might not relocate the pages quickly enough for immediate needs. The length of time required for monitoring and relocation indicates that Dynamic Tiering is best suited for long-term location of pages. Set the monitoring cycle to a long enough interval to collect data on the I/O patterns throughout the critical part of the business cycle.

27

For example, if VM1 and VM2 run during business hours in the US, and VM3 and VM4 run during business hours in APAC, set monitoring to 24 hours so that pages used by all VMs are monitored and placed on the correct tier during the subsequent relocation cycle. For another example, if an application performs OLTP during business hours and batch processing during off hours, set the monitoring cycle to business hours when OLTP is running. This maximizes the page allocations for OLTP where response time is more critical.

Best Practice Recommendation for Automatic and Manual Modes

Use manual mode for initial page distribution or when new virtual disks are created and relocation is expected to take many hours. Use automatic mode when small changes in page allocation are expected.

General Dynamic Tiering Best Practices

Use a monitoring cycle long enough to collect IOPH on typical workload. For example, if a workload is expected to change throughout a 24-hour cycle, monitor for the full 24 hours to collect IOPH on every page access throughout the day. If a workload is expected to remain consistent throughout a 24-hour timeframe, you can use a shorter monitoring cycle. Use Dynamic Tiering to place frequently accessed pages on the correct tier for the long-term needs. For short term needs, migrate data using Hitachi Tiered Storage Manager or vSphere Storage vMotion.

Distributing Computing Resources Using DRS and Dynamic Provisioning Software

DRS can balance computing capacity by implementing CPU and memory pools. When you use Hitachi Dynamic Provisioning software with DRS, CPU, memory and storage I/O loads are distributed by combining them within resource pools. DRS combines CPU and memory from all ESX hosts into a DRS resource pool and Hitachi Dynamic Provisioning software combines multiple RAID groups into a Dynamic Provisioning pool. Figure 19 shows standalone ESX hosts with imbalanced CPU utilization.

Figure 19

28

In Figure 19, each standalone ESX host contains an amount of CPU and memory resources. When ESX hosts are configured in a DRS cluster, the resources are aggregated into a pool. This allows the resources to be used as a single entity. A virtual machine can run on any ESX host in the DRS cluster, rather than being tied to a single host. DRS manages these resources as a pool and automatically places virtual machines on a host at power-on and continues to monitor resource allocation. DRS uses vMotion to move virtual machines from one host to another when it detects a performance benefit or based on other optimization decisions. Figure 20 shows how Hitachi Dynamic Provisioning software aggregates disks in to a Dynamic Provisioning pool.

Figure 20

Hitachi Dynamic Provisioning software aggregates all the allocated disks into a Dynamic Provisioning pool. All DP-VOLs are striped across all disks in the Dynamic Provisioning pool. This allows you to treat all the disks in the Dynamic Provisioning pool as a single entity. A single standard RAID group is bound by the IOPS available from the disks in that RAID group. When LDEVs are placed in a Dynamic Provisioning pool, the DP-VOLs are not tied to a single RAID group; instead, DP-VOLs span all the RAID groups and disks in the Dynamic Provisioning pool.

29

Figure 21 shows how standalone ESX configurations with standard RAID groups can have unbalanced loads.

Figure 21

With standalone ESX hosts, computing resources might be heavily used by certain virtual machines. These virtual machines might also be underperforming because the host can no longer provide any more resources; the host becomes a limiting factor. Other ESX hosts in the same farm might be moderately used or might be sitting idle. Meanwhile, the disk utilization in a RAID group might be at its performance limit handling I/Os from the virtual machines. Other RAID groups might be moderately used or lightly used. In this scenario, heavy imbalance of resource utilization exists on both the host side and the storage side. Utilization of resources changes throughout the day and only careful monitoring and manual administration can balance these loads. Figure 22 shows how DRS can distribute loads on the ESX hosts.

Figure 22

30

With ESX hosts configured in DRS cluster, computing resources are managed as a pool. DRS automatically uses vMotion to move virtual machines to other hosts to evenly balance utilization or for performance benefits. Monitoring and placement of the virtual machines is done automatically by DRS; no manual administration is required. However, when used with standard RAID groups, a heavy imbalance still exists on the storage system. To balance the loads on the storage system, manual monitoring is required and Storage vMotion can be used to migrate virtual disks to other RAID groups when necessary. Figure 23 shows how DRS with Hitachi Dynamic Provisioning software can distribute loads on ESX hosts and the RAID groups.

Figure 23

When RAID groups are configured in a Dynamic Provisioning pool, all the disks in the Dynamic Provisioning pool are treated as a single entity. DP-VOLs span multiple RAID groups in the Dynamic Provisioning pool. This distributes the I/O across all the disks in the pool. By using vSphere DRS, Hitachi Dynamic Provisioning software and round robin multipathing together, computing resources and I/O load are automatically distributed for better performance and scalability.

31

Universal Volume Manager

External storage systems can be connected to the Virtual Storage Platform in one of two ways: Direct attached Switched SAN In both cases, the Fibre Channel ports on the Virtual Storage Platform are configured as external. Load balancing and failover are automatically configured by the Virtual Storage Platform when multiple Fibre Channel ports are connected to the same external storage system LUNs. After the external storage systems LUNs are available on the Virtual Storage Platform, you can use them as standard LUNs or you can add them to a Dynamic Provisioning pool. Figure 24 shows an Adaptable Modular Storage 2000 family storage system connected to a Virtual Storage Platform using Universal Volume Manager. The LUNs from the external storage are configured in a Dynamic Provisioning pool on the Virtual Storage Platform.

Figure 24

Universal Volume Manager offers two cache mode settings: Cache Mode = Enable Processes I/O to external LDEVs exactly the same as internal LDEVs. When a host write occurs the data is duplexed in cache and an immediate I/O complete response is sent back to the host. Cache Mode = Disable Default; tells the Virtual Storage Platform to hold off on sending an I/O complete response to the host until the I/O has been committed to the external storage system.

32

The cache mode setting does not change the cache handling for read I/Os. On a read request the Virtual Storage Platform will examine the cache to see if the data is available in cache, if it is it will return the data from cache, if it is not it will retrieve the data from the external storage system. Slower external storage systems can cause cache write pending to rise and affect the throughput of other hosts or LDEVs. Do not use Cache Mode = Enable when high IOPS are expected to the external storage system. Universal Volume Manager offers the ability to set the queue depth on the external storage connected to the Universal Storage Platform VM. Keep the following in mind when adjusting the queue depth setting: The range of queue depth values is 2 to 128 with a default of 8. Increasing the queue depth from the default setting of 8 to 32, 64 or 128 can have a positive effect on the response time of OLTP type applications. The maximum queue depth on an external port is 256, so if multiple external storage systems are attached, be sure to set the queue depth to not exceed 256 total for all external LUNs. In a Universal Volume Manager configuration that uses a Hitachi Adaptable Modular Storage 2000 family storage system connected as external storage, Dynamic Provisioning pools can be created on the Virtual Storage Platform system, the Adaptable Modular Storage 2000 family or both. Hitachi Data System recommends placing the Dynamic Provisioning pool on the Virtual Storage Platform only. Placing a Dynamic Provisioning pool on the Adaptable Modular Storage 2000 family system can result in administrative overhead, poor performance and poor utilization of the thin provisioning feature of Dynamic Provisioning.

Disk Alignment and RAID Stripe Size

Properly aligned storage is important for optimized I/O. Improper alignment can lead to incurring additional I/O when accessing data. A properly aligned system must be aligned at the guest operating systems file system level, ESXs VMFS and on the storage system. The Virtual Storage Platform has a strip size of 512k per data disk. Multiply this by the number of data disks in the RAID group for the total stripe size. VMFS3 has a default starting block of 128. If the VMFS datastore is upgraded from ESX 2.x, the VMFS volume might not be aligned. Previous versions have a starting block of 63, which is not aligned along the 512k stripe boundary. Hitachi Data Systems lab testing shows that using a starting block of 128 yields optimal performance. A starting block of 63 or 256 yields slightly lower performance. Hitachi recommends using the default VMFS3 starting block or 128 and converting any VMFS datastores created under ESX 2.x to VMFS3 with a starting block of 128. To see if the VMFS is properly aligned, issue the following ESX command: fdisk lu The output might be similar to this: Device Boot /dev/sdn1 Start 128 End 2247091874 Blocks Id 1123545873+ System fb VMware VMFS

33

The Start value of 128 indicates an aligned partition. A Start value of 63 indicates that the partition is not aligned. If the VMFS is not properly aligned, consider migrating the VMs to another LU and recreating the volume. If this is not an option, see VMwares Performance Best Practices for VMware vSphere 4 white paper. With Windows 2008, newly created partitions are properly aligned. New partitions created with previous versions of Windows operating system are not aligned by default. When a partition that was created on earlier versions of Windows is attached to Windows 2008, it carries the same partition properties as when it was created. For more information, see Microsofts Disk Partition Alignment Best Practices for SQL Server article

ESX Host Configuration

Configuring ESX 4.1 requires some understanding of how I/O flows from a virtual machine through the VMkernel then to the physical disks. Figure 25 shows the I/O stack of ESX 4.1 with a Virtual Storage Platform.

Figure 25

34

When an application issues an I/O, the guest operating systems virtual adapter driver passes the I/O to the virtual SCSI adapter in virtual machine monitor (VMM). The I/O is then passed to the VMkernel. At this point, the I/O can take different routes based on the kind of virtual disk the virtual machine uses: If virtual disks on VMFS are used, the I/O is passed through the virtual SCSI layer, then through the VMFS layer. If NFS is used, the I/O passes through the NFS layer. If raw device mapping (RDM) is used, it can be virtual or physical. Virtual RDM passes through the virtual SCSI layer where the physical RDM uses the virtual machine guest operating systems SCSI layer and bypasses the virtual SCSI layer. The I/O is then issued to the pluggable storage architecture (PSA) to determine what path the I/O to be sent. I/O is then passed to the HBA driver queue, then to the Virtual Storage Platform. The testing done to support this best practices guide used VMFS.

Multipathing
ESX 4.1 uses PSA, which allows the use of VMware Native Multipathing (NMP) and Path Selection Plugin (PSP). As shown in Figure 22, when I/O is issued through the PSA, the NMP calls the PSP assigned to the storage device. PSP determines to which path the I/O is to be sent. If the I/O is complete, NMP reports its completion. If the I/O is incomplete, the Storage Array Type Plug-in (SATP) is called to interpret the error codes and activate paths if necessary. PSP then resets the path selection. When the Virtual Storage Platform is used with ESX 4.1, NMP is automatically configured with the following plug-ins: SATP Default active-active storage system type (VMW_SATP_DEFAULT_AA) PSP Fixed path policy (VMW_PSP_FIXED) However, the Virtual Storage Platform can also support the round robin multipathing policy (VMW_PSP_RR). Round robin rotates through all available paths distributing the I/O load across the paths.
Key Best Practice To maximize the capabilities of the Hitachi Virtual Storage Platform, use the round robin path policy.

Queue Depth
Queue depth settings for ESX hosts can be complex and difficult to calculate. If you see a QFULL condition in the VMkernel logs, you can enable the VMkernel adaptive queue depth algorithm. Hitachi recommends using this only as a temporary measure until you can calculate and adjust the queue depth at the HBA. For more information, see VMwares Controlling LUN queue depth throttling in VMware ESX/ESXi Knowledge Base article. Unless you see problems in the VMkernel logs as described in the Controlling LUN queue depth throttling in VMware ESX/ESXi Knowledge Base article, Hitachi Data Systems recommends using a maximum queue depth setting of 32 in most environments. Monitoring of the ESX hosts is required to verify that this setting is optimal. For more information on monitoring the ESX host, see Table 4.

35

Calculating Queue Depth

If adjustments are needed to avoid a QFULL condition, follow these best practices: Queue depth per disk drive Tagged command queuing (TCQ) on SAS drives or native command queuing (NCQ) on SATA drives are the queuing mechanisms for disk drives. Typical values used for this calculation are 16 TCQs for SAS drives and four NCQs for SATA. Queue depth per active LUN The LUN is a virtual SCSI disk drive and has a queue depth. The host HBA assigns tag numbers and populates the queue. The Virtual Storage Platform has no perLUN queue depth limit; divide the Fibre Channel port queue limit of 2,048 by the number of LUNs on a Fibre Channel port. Queue depth per Fibre Channel port on the storage system The maximum queue depth per port on the Virtual Storage Platform Fibre Channel ports is 2,048. Queue depth per active HBA Calculate the total queue depth for all HBAs communicating with a Fibre Channel port on the Virtual Storage Platform. The default LUN queue depth for Emulex drivers is 30. The default LUN queue depth for Qlogic drivers is 32. Use the following formula to calculate a starting point for adjusting the queue depth at the ESX host HBA: (TCQ or NCQ x Number of disks / LUNs) / Active HBAs In some cases in a vSphere cluster, not all ESX hosts perform active I/O to a LUN. Take into account only the ESX hosts performing active I/O to the LUN. For example, to calculate the HBA queue depth for a Dynamic Provisioning pool made up of 16 SAS disk drives with four LUNs connected to four ESX hosts with two active HBAs each, use this formula: (16 disk drives x 16 TCQs / 4 LUNs) / 8 HBAs = 4 If you do not expect all four LUNs to be active on all four ESX hosts at the same time, adjust this number upward. For example, in a vSphere DRS cluster, a LUN might be accessed by all or only a few of the ESX hosts in the cluster at a time, so the formula can be modified. In the following example, two ESX hosts access the LUNs: (16 disk drives x 16 TCQs / 4 LUNs) / 4 HBAs = 16

36

Figure 26 shows 16 SAS disk drives in a Dynamic Provisioning pool with four LUNs associated with the disks connected to two ESX hosts with two HBAs each.

Figure 26

In very large Dynamic Provisioning pools, the wide striping advantage can be beneficial; however, you must take care when assigning LUNs. A few large LUNs in the Dynamic Provisioning pool mean that this calculation results in a very high queue depth and poor performance. .
Key Best Practice Do not exceed the queue depth of 2,048 for the Fibre Channel port on the Virtual Storage Platform.

The procedure to change queue depths differs depending on the HBA vendor or type of HBA driver. To change queue depths for Emulex and Qlogic drivers, see the VMware Knowledge Base article Changing the Queue Depth for QLogic and Emulex HBAs. When changing LU queue depths, you also typically change the Disk.SchedNumReqOutstanding ESX advanced parameter. However, Hitachi Data Systems recommends setting this parameter to a value no higher than the default of 32 on ESX 4.1. This parameter affects the number of outstanding commands to a target when competing virtual machines exist. Monitoring queue depth is an important part of monitoring your environment and troubleshooting performance problems. When the queue depth is exceeded, I/O is queued in the VMkernel. This can increase I/O latency for the virtual machines. The esxtop or resxtop utilities can be used to monitor queue depth on ESX 4 at the storage disks adapter, device and virtual machine levels.

37

ESX Host Metrics to Monitor

Table 4 lists important metrics to monitor from the ESX hosts.
Table 4. esxtop Storage I/O Metrics

Metric AQLEN LQLEN ACTV QUED %USD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd

Description Maximum number of ESX VMkernel active commands that the adapter driver is configured to support (storage adapter queue depth) Maximum number of ESX VMkernel active commands that the LU is allowed to have (LUN queue depth) Number of commands in the ESX VMkernel that are currently active for a LUN Number of commands in the ESX VMkernel that are currently queued for a LUN Percentage of queue depth used by ESX VMkernel active commands for a LUN Number of commands issued per second for a device Number of read commands issued per second for a device Number of write commands issued per second for a device Megabytes read per second for a device Megabytes written per second for a device Average device latency per command in milliseconds Average ESX VMkernel latency per command in milliseconds Average guest OS latency per command in milliseconds Average queue latency per command in milliseconds

SCSI Reservations
SCSI reservation conflicts can cause I/O performance problems and limit access to storage resources. This can occur when multiple ESX hosts access a shared VMFS volume simultaneously during certain operations. The following operations use SCSI reservations: Creating templates Creating virtual machines either new or from template Running vMotion Powering on virtual machines Growing files for virtual machine snapshots Allocating space for Thin virtual disks Adding extents to VMFS volumes Changing the VMFS signature

38

Many of these operations require VMFS metadata locks. Experiencing a few SCSI reservations conflicts is generally acceptable; however, best practice is to minimize these conflicts. The following conditions can affect the number of reservation conflicts: Number of virtual machines per VMFS volume Number of ESX hosts accessing a VMFS volume Use of virtual machine snapshots In addition, follow these best practices to minimize SCSI reservation conflicts: Do not run VMware Consolidated Backups (VCBs) on multiple virtual machines in parallel to the same VMFS volume. Run operations that require SCSI reservations to the shared VMFS volume serially.

VMkernel Advanced Disk Parameters

You must fully understand an applications I/O workloads and test all changes in a controlled environment before changing these parameters on ESX hosts. Tuning for better performance for one type of workload might decrease performance for other types of workload. ESX hosts generally experience a wide variety of workloads and tuning must accommodate those. In general, the default parameters are sufficient for wide variety of workloads. Table 5 lists the parameters that are most likely to affect performance.
Table 5. Advanced Disk Parameters

Parameter Disk.BandwidthCap Disk.ThroughputCap Disk.SectorMaxDiff

Default Value 4294967294 4294967294 2000

Description Limit on disk bandwidth (KB/s) usage. Limit on disk throughput (IO/s) usage. Distance in sectors in which I/O of a VM is considered sequential. Sequential I/O is given higher priority to get the next I/O slot. Number of consecutive requests from one world (VMs). Number of outstanding commands to a target with competing worlds (VMs). Number of consecutive requests from VM required to raise the outstanding commands to the maximum. Number of switches between commands issued by different VMs required to reduce outstanding commands to SchedNumReqOutstanding. Maximum disk read/write I/O size before splitting (in KB).

Disk.SchedQuantum Disk.SchedNumReqOutstanding Disk.SchedQControlSeqRegs

8 32 128

Disk.SchedQControlVMSwitches

Disk.DiskMaxIOSize

32767

39

Scalability Best Practices

No one size fits all storage design exists. When designing a scalable environment, you must understand the following design parameters for your environment: Level of acceptable I/O performance LUN size Size of virtual disks Number of virtual machines per LUN Number of virtual disks per virtual machine Type of workload Type of physical disks Size of physical disks RAID group type RAID group configuration Changes to one parameter can lead to changes in other parameters. For example, changing the LUN size you choose may affect the number of virtual machines per LUN or size of virtual disks. Follow these best practices when sizing your vSphere 4 environment: Configure for performance first, then capacity. The number of disks required to meet performance requirements might be greater than the number of disks required to meet capacity requirements. When you use Hitachi Dynamic Provisioning software, adding more capacity will yield more performance at the same time. Capacity can be added to Dynamic Provisioning pools without disruption. Aggregate application I/O requirements, but take care not to exceed the capability of the underlying RAID groups. Make configuration choices based on I/O workload. You can determine I/O workload by monitoring application I/O loads for a period of time that contains a full cycle of business demands. Distribute workloads to other RAID groups. An application with high I/O load can affect performance and can create hot spots. Consider moving these virtual machines to another RAID group or LUN with Storage vMotion. This is less of a concern with Dynamic Provisioning software because loads are distributed across multiple RAID groups.

40

Conclusion
This white paper describes best practices for deploying vSphere 4 on the Virtual Storage Platform. Following these best practices helps to ensure that your infrastructure is robust, offering high performance, scalability, high availability, ease of management, better resource utilization and increased uptime. Table 6 lists best practices for optimizing the Hitachi Virtual Storage Platform for vSphere 4 environments.
Table 6. Best Practices for Hitachi Virtual Storage Platform

Description Distributing loads Redundancy

Best Practice Use Hitachi Dynamic Provisioning software with DRS to distribute loads on the storage system and ESX hosts. Use minimum of two host HBA ports. Use at least two Fibre Channel switches or a director class switch. Use at least two Fibre Channel Ports, one port for each storage cluster, on the Virtual Storage Platform.

Zone Host groups

Use single-initiator zones. For a standalone ESX host, configure one host group per ESX host on each storage cluster. For clustered ESX hosts, configure one host group per ESX cluster on each storage cluster.

Dynamic provisioning space saving and virtual disk

Create the VM template on a zeroedthick format virtual disk. Use the default zeroedthick format virtual disk. Use zeroedthick virtual disks when thin provisioning is required When using Dynamic Provisioning software for cost savings, do not convert a thin format virtual disk to thick using the vCenter GUIs Inflate option.

Dynamic Tiering

Use a monitoring cycle long enough to collect IOPH on typical workload. Use manual mode to monitor and relocate for periods of time greater than 24 hours Use manual mode for initial page distribution or when new virtual disks are created and relocation is expected to take many hours. Use automatic mode when small changes in page allocation are expected.

Virtual disk and Dynamic Provisioning performance

Use eagerzeroedthick virtual disk format to prevent warm-up anomalies. Size the Dynamic Provisioning pools according to the I/O requirements of the virtual disk and application. When larger Dynamic Provisioning pools are not possible, separate sequential and random workloads on different Dynamic Provisioning pools. For applications that use log recovery, separate the logs from the database on different Dynamic Provisioning pools.

Virtual disk format

If minimizing the time to create the virtual disk is more important than maximizing initial write performance, use the zeroedthick virtual disk format. If maximizing initial write performance is more important than minimizing the time required to create the virtual disk, use the eagerzeroedthick format.

41

Description Scalability

Best Practice Configure for performance first, then capacity. Aggregate application I/O requirements, but take care not to exceed the capability of the RAID group. Make configuration choices based on I/O workload. Distribute workloads to other RAID groups.

Table 7 lists best practices for using external storage with Universal Volume Manager on the Virtual Storage Platform.
Table 7. Best Practices for External Storage with the Hitachi Virtual Storage Platform

Description Dynamic Provisioning software Redundancy

Best Practices Use Hitachi Dynamic Provisioning software on the Virtual Storage Platform for best performance and space utilization. Use minimum of two Fiber Channel connections from the external storage system to the Virtual Storage Platform. Use at least two Fibre Channel switches or a director class switch. Set to Disable. For random workloads, increase from the default of 8 to 32 or 64.

Cache mode WWN QDepth

Table 8 lists best practices for optimizing vSphere 4.1 for the Hitachi Virtual Storage Platform
Table 8. Best Practices for vSphere 4.1

Item Multipathing policy Queue depth

Best Practice Use round robin (VMW_PSP_RR). Set LU queue depth to no more than 32 for SAS drives. Set LU queue depth to no more than 16 for SATA drives. Use the default value of 32 for the ESX VMkernel advanced parameter Disk.SchedNumReqOutstanding.

Minimize SCSI reservation conflicts

Reduce the number of virtual machines per VMFS volume. Reduce the number of ESX hosts accessing a VMFS volume. Minimize the use of virtual machine snapshots. Avoid running VMware Consolidated Backups (VCBs) on multiple virtual machines in parallel to the same VMFS volume. Run operations that require SCSI reservations to the shared VMFS volume serially.

Scalability

Configure for performance first, then capacity. Aggregate application I/O requirements, but take care not to exceed the capability of the RAID group. Make configuration choices based on I/O workload.

42

Item Storage I/O Control

Best Practice Do not use different shares or IOPS settings on VMFS datastores that share the same underlying storage resources because this can produce uneven I/O results.

Hitachi Data Systems Global Services offers experienced storage consultants, proven methodologies and a comprehensive services portfolio to assist you in implementing Hitachi products and solutions in your environment. For more information, see the Hitachi Data Systems Global Services web site. Live and recorded product demonstrations are available for many Hitachi products. To schedule a live demonstration, contact a sales representative. To view a recorded demonstration, see the Hitachi Data Systems Corporate Resources web site. Click the Product Demos tab for a list of available recorded demonstrations. For more information about Hitachi products and services, contact your sales representative or channel partner or visit the Hitachi Data Systems web site.

43

Hitachi is a registered trademark of Hitachi, Ltd., in the United States and other countries. Hitachi Data Systems is a registered trademark and service mark of Hitachi, Ltd., in the United States and other countries. All other trademarks, service marks and company names mentioned in this document are properties of their respective owners. Notice: This document is for informational purposes only, and does not set forth any warranty, expressed or implied, concerning any equipment or service offered or to be offered by Hitachi Data Systems Corporation

Hitachi Data Systems Corporation 2011. All Rights Reserved. AS-059-01 January 2011 Corporate Headquarters 750 Central Expressway, Santa Clara, California 95050-2627 USA www.hds.com Regional Contact Information Americas: +1 408 970 1000 or info@hds.com Europe, Middle East and Africa: +44 (0) 1753 618000 or info.emea@hds.com Asia Pacific: +852 3189 7900 or hds.marketing.apac@hds.com

44