IBM DS4000 Auto Volume Transfer Usage with RDAC

Version 1 Date: 9th Sep 2010 Juan Li (ljuan@cn.ibm.com) IBM China Systems & Technology Lab

Reviewed By Frank Xiao (xiaofeng@cn.ibm.com)

Copyright IBM Corporation 2010. All rights reserved.

Page 1 of 15

Notices and Disclaimer

Copyright 2010 by International Business Machines Corporation. No part of this document may be reproduced or transmitted in any form without written permission from IBM Corporation. Product data has been reviewed for accuracy as of the date of initial publication. Product data is subject to change without notice. This information may include technical inaccuracies or typographical errors. IBM may make improvements and/or changes in the product(s) and/or programs(s) at any time without notice. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. THE INFORMATION PROVIDED IN THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IBM EXPRESSLY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT. IBM shall have no responsibility to update this information. IBM products are warranted according to the terms and conditions of the agreements (e.g., IBM Customer Agreement, Statement of Limited Warranty, International Program License Agreement, etc.) under which they are provided. IBM is not responsible for the performance or interoperability of any nonIBM products discussed herein. Any performance data contained herein was obtained in a controlled, isolated environment. Actual results that may be obtained in other operating environments may vary significantly. While IBM has reviewed each item for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents or copyrights. Inquiries regarding patent or copyright licenses should be made, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A.

Copyright IBM Corporation 2010. All rights reserved.

Page 2 of 15

Trademarks
IBMDS4000, AVT, ADT, and RDAC are trademarks of International Business Machines Corporation in the United Stated, other countries, or both. Other company, product or service names may be trademarks or service marks of others.

Acknowledgements
My thanks to the following colleagues for their significant contribution to this White Paper: Jing Zhou, Manager, IBM Open Systems Lab, China Feng Xiao, IBM Open Systems Lab, China

Copyright IBM Corporation 2010. All rights reserved.

Page 3 of 15

Table of Contents
1. 2. 3. 4. Introduction ........................................................................................................................ 5 Logical drives and controller ownership............................................................................. 5 Overview of AVT and RDAC .............................................................................................. 7 AVT or non-AVT mode works with RDAC .......................................................................... 8 4.1. 4.2. 4.3. 5. AVT enabled failover with RDAC................................................................................ 8 AVT disabled failover with RDAC ............................................................................... 9 How to enable and disable AVT ................................................................................11

Possible performance degradation with or without AVT .................................................. 12 5.1. 5.2. 5.3. 5.4. Multi-path driver and AVT enabled ........................................................................... 12 Multi-path driver and AVT disabled........................................................................... 13 No multi-path driver with AVT enabled ..................................................................... 14 SAN boot limitations between RDAC and AVT ........................................................ 14

Reference ................................................................................................................................ 15

Copyright IBM Corporation 2010. All rights reserved.

Page 4 of 15

1. Introduction
The IBM System Storage DS4000 Series are IBMs solution for mid-range or departmental storage requirements. In a DS4000 storage subsystem equipped with two controllers, you can provide redundant I/O paths between the host systems and the storage. There are two different components that provide redundancy in the I/O data paths: the Auto Volume Transfer (AVT) feature of the storage subsystem and a host multi-path driver, for example, Redundant Disk Array Controller (RDAC). This paper is to provide an overview of AVT and RDAC, and introduce how AVT works with RDAC depending on your configuration. It also intends to illustrate several possible path thrashing situations with the AVT mode or not.

2. Logical drives and controller ownership

About logical drives
Logical drives, sometimes simply referred to as volumes or LUNs (LUN stands for Logical Unit Number and represents the number a host uses to access the logical drive), are the logical segmentation of arrays. A logical drive is a logical structure you create on a storage subsystem for data storage. A logical drive is defined over a set of drives called an array and has a defined RAID level and capacity. The drive boundaries of the array are hidden from the host computer. IBM System Storage DS4000 provides great flexibility in terms of configuring arrays and logical drives.

About controller ownership

For DS4000 products with dual array controllers, each logical drive has a preferred controller of ownership, which is considered the owner, or the active (primary) controller, of a particular LUN. The other controller is considered as an alternative or passive (secondary) controller. This means that every logical drive is owned by only one controller. When assigning logical volumes to the systems, it is very important to remember that the DS4000 storage uses a preferred controller ownership approach for communicating with logical drives. The alternate controller only takes over and handles the I/O requests in the case of a failure along the I/O path, for example, a failure of the host bus adapter or switch. When defining logical drives, the system normally alternates ownership between the two controllers. All heavily stressed logical drives can reside on only one controller while the other one handles only a small amount of all I/O requests. To balance the workload between the controllers, you can change the preferred ownership of a logical drive to the other controller, and normally the storage system is balanced better regarding the workload. It is, therefore, important at the

Copyright IBM Corporation 2010. All rights reserved.

Page 5 of 15

system level to make sure that traffic is correctly balanced among controllers. Balancing traffic is unfortunately not always a trivial task. For example, if an application requires large disk space to be located and accessed in one chunk, it becomes harder to balance traffic by spreading the smaller volumes among controllers. In addition, typically, the load across controllers and logical drives are constantly changing. The logical drives and data accessed at any given time depend on which applications and users are active during that time period, hence the importance of monitoring the system.

Assign the preferred ownership

Ownership is assigned to an array or a logical drive. Figure 2-1 shows how to change the preferred controller ownership for a selected array.

Figure 2-1 Change preferred controller ownership for an array Figure 2-2 shows how to change the preferred controller ownership for a selected logical drive.

Figure 2-2 Change preferred controller ownership for a logical drive

Copyright IBM Corporation 2010. All rights reserved.

Page 6 of 15

Considerations
If the preferred controller is undergoing a firmware download, ownership of the logical drives is automatically shifted to the other controller, and that controller becomes the current owner of the logical drives. If the preferred controller must be replaced, disable the controller first. This intentionally causes a failover of LUNs to the other controller and allows the removal and replacement of the preferred controller. This is considered a routine ownership change and is reported with an informational entry in the event log. Besides, a secondary logical drive in a Remote Mirror does not have a preferred owner. Instead, the ownership of the secondary logical drive is determined by the controller owner of the associated primary logical drive. For example, if controller A owns the primary logical drive in the primary storage system, then controller A owns the associated secondary logical drive in the secondary storage system. Controller ownership changes of the primary logical drive cause a corresponding controller ownership change of the secondary logical drive.

3. Overview of AVT and RDAC

About Auto Volume Transfer (AVT)
AVT, also referred to as the Auto Disk Transfer (ADT), is a built-in feature of controller firmware that allows logical drive-level failover rather than controller-level failover (as is the case with RDAC). It provides redundant I/O paths in conjunction with a multi-path driver installed on the host system. The AVT feature is automatically disabled or enabled depending on the type of host ports on the host partition to which you mapped the logical drives. It is disabled by default for Microsoft Windows, IBM AIX, and Sun Solaris operating systems. It is enabled by default for Linux, Novell Netware, and HP-UN operating system.

About multi-path driver and RDAC

The multi-path driver manages the I/O data connection for storage subsystems with redundant controllers. If a component (cable, controller, host adapter, and so on) fails along the I/O data path, the multi-path driver automatically reroutes all I/O operations to the other controller. RDAC stands for Redundant Disk Array Controller. The RDAC component, a part of the Storage Manager software, contains a multi-path driver and host-add support. It must be installed on the host system and it will provide redundant path to the storage subsystem when both RAID controllers are installed. If a RAID controller fails or becomes inaccessible due to

Copyright IBM Corporation 2010. All rights reserved.

Page 7 of 15

path failure, RDAC will provide redundant I/O paths and reroute the I/O requests through another RAID controller. The host-add part of RDAC allows you to register new logical drives to the operating system dynamically. Some operating systems do not use RDAC as they have their own multi-path drivers.

4. AVT or non-AVT mode works with RDAC

4.1. AVT enabled failover with RDAC
There are two different components that provide redundancy in the I/O data paths: AVT and host multi-path driver, for example, RDAC. If there is a problem on the preferred path, the multi-path driver, like RDAC, will change the logic drive ownership to an alternate controller, switch the path that was passive to active, select the active path for I/O, and let the AVT react. AVT will guarantee that the logical drive is accessible through the alternate controller. After the I/O path recover from the failure, the preferred controller automatically reestablishes ownership of the logical drive and controller the I/O along this path as soon as RDAC detects that the path is normal again. Figure 4-1, 4-2, 4-3 and 4-4 show the phases of failover in the AVT-enabled case. In AVT mode, RDAC automatically redistributes the logical drives to their preferred path after the failed path is operational again.

Application

P0 X

Controller A

DS4000
P1 Controller B 0 1 2 3

Host/Server
Figure 4-1 Application running I/Os against LUN 0 through preferred controller (A), and failover driver hides the two physical paths to LUNs.

Copyright IBM Corporation 2010. All rights reserved.

Page 8 of 15

Application

P0 X

Controller A

DS4000
P1 Controller B 0 1 2 3

Host/Server
Figure 4-2 Port 0 receives a path failure, and error is passed on to the failover driver

Application

P0 X

Controller A

DS4000
P1 Controller B 0 1 2 3

Host/Server
Figure 4-3 Failover driver transfers the LUN 0 ownership to the alternate controller (B), and use the controller (B) for any retired or new I/Os

Application

P0 X

Controller A

DS4000
P1 Controller B 0 1 2 3

Host/Server
Figure 4-4 Failback LUN 0 to the preferred controller (A) after the path failure corrected

4.2. AVT disabled failover with RDAC

Even if AVT is disabled, the RDAC will still provide redundancy. Each logical drive still has a preferred controller which controls the I/O along the path. If a component fails, RDAC will send a SCSI Mode Select command to cause a change in volume ownership before using the alternate path. All logical drives on the preferred controller are transferred to the alternate controller, as opposed to transfer only a single logical drive with AVT.

Copyright IBM Corporation 2010. All rights reserved.

Page 9 of 15

Ensure that all attached hosts have a multi-path driver installed in this situation, because all logical drives will be moved to the other controller. This also includes logical drives from other host groups with different mappings in the storage partitioning. After the I/O data path problem is corrected, the preferred controller does not automatically reestablish ownership of the logical drive. In AVT-disabled mode, you are required to issue a redistribution command manually to balance the LUNs across the controllers. Figure 4-5, 4-6, 4-7 and 4-8 show the AVT-disabled failover mode phases with RDAC.

Application

P0 X

Controller A

DS4000
P1 Controller B 0 1 2 3

Host/Server
Figure 4-5 Application running I/Os against LUN 0 through preferred controller (A), and failover driver hides the two physical paths to LUNs

Application

P0 X

Controller A

DS4000
P1 Controller B 0 1 2 3

Host/Server
Figure 4-6 Port 0 receives a path failure, and error is passed on to the RDAC

Application

P0 X

Controller A

DS4000
P1 Controller B 0 1 2 3

Host/Server
Figure 4-7 RDAC transfers all logical drives to the alternate controller (B) to continue I/Os

Copyright IBM Corporation 2010. All rights reserved.

Page 10 of 15

Application

P0 X

Controller A

DS4000
P1 Controller B 0 1 2 3

Host/Server
Figure 4-8 The preferred controller does not automatically reestablish ownership of the logical drive after path failure corrected

4.3. How to enable and disable AVT

The AVT feature can be enabled and disabled with the following script. If you set the HostNVSRAMBYTE 0x24 to 1, AVT will be enabled after the script executed; if you set the HostNVSRAMBYTE 0x24 to 0, AVT will be disabled after the script executed. It is important that the script must be executed with the correct number/Index of the host type. Please refer to the table in the script for the cross reference between host type and host number. Be sure to reset two controllers to take these settings effect after the update. // Name: Enable or Disable AVT Script // Host number/index // // // // // // // // // // // // // // // // Replace the value z in the following script commands with the appropriate host // index/number. // 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Host Type Windows Non-Clustered (SP5 or higher) Windows Clustered (SP5 or higher) Windows 2000 Non-Clustered Windows 2000 Clustered NetWare-IBMSAN Linux AIX HP-UX Solaris (Sparc) PTX Irix Netware Failover IBM TS SAN VCE LNXCL enable by default enable by default enable by default enable by default enable by default enable by default

Copyright IBM Corporation 2010. All rights reserved.

Page 11 of 15

// For Controller A show " "; show " Displayed old setting and then enable AVT in controller A"; show controller [a] HostNVSRAMBYTE [z,0x24]; set controller [a] HostNVSRAMBYTE [z,0x24]=0x01; // 0x01 is to enable AVT; 0x00 is to disable AVT // Now for Controller B show " "; show " Displayed old setting and then enable AVT in controller B"; show controller [b] HostNVSRAMBYTE [z,0x24]; set controller [b] HostNVSRAMBYTE [z,0x24]=0x01; // 0x01 is to enable AVT; 0x00 is to disable AVT // Verify the settings show " "; show " Displayed new setting for controllers A and B"; show controller [a] HostNVSRAMBYTE [z,0x24]; show controller [b] HostNVSRAMBYTE [z,0x24]; // // For fw 5.3x.xx.xx and later, you can reset the // controller using these two script commands. Otherwise, you // must manually reset the controller. // show " "; show "reset controllers A and B"; reset Controller [a]; reset Controller [b];

5. Possible performance degradation with or without AVT

If your server is unable to access a LUN, or access is very slow, you might have a problem with path thrashing (also called LUN thrashing). When this phenomenon happens, logical drives can not be accessed by the hosts without coordination of path access between the hosts. Failure to provide path coordination between the hosts may result in performance degradation. The following illustrates several possible path thrashing situations.

5.1. Multi-path driver and AVT enabled

For DS4000 storage arrays, if AVT is enabled, path thrashing might occur when hosts access the same LUN through different preferred controllers, they all continuously cause the

Copyright IBM Corporation 2010. All rights reserved.

Page 12 of 15

ownership of the LUN to move between controllers, effectively ping-pong the ownership of the LUN. Because the system moves the ownership quickly, the storage array cannot process any I/O. As a result, the LUN is never actually available. The performance will decrease significantly, you may want to disable AVT. Figure 5-1 illustrates one example, host A and host B, both with redundant paths, concurrently access to the same logical drive. If only host B has a path failure in the blue path, AVT will move the logical drive from the preferred to the alternate controller (red path). Host A continues to send I/O requests to the logical drive through the preferred path (blue) unaware of the path problems of host B. The ownership of the LUN will transfer to the preferred controller then to alternate controller repeatedly, which will degrade the performance of the storage subsystem.

Figure 5-1 LUN thrashing with MPIO and AVT In a cluster environment you often have disks that are accessible by different hosts at the same time. It now depends on the architecture of the cluster if this situation can be accepted or not. In a cluster environment you may disable the AVT feature to ensure proper handling of the logical drives even in the case of a path failure depending on the cluster in use.

5.2. Multi-path driver and AVT disabled

If a host system running RDAC has a path failure, RDAC will move all logical drives to the other controller. Other host systems with a multi-path I/O driver different to RDAC will survive this move of their logical drives to the other controller.

Copyright IBM Corporation 2010. All rights reserved.

Page 13 of 15

But if a host system without RDAC but with its own multi-path I/O driver has a path failure, the DS4000 storage would not move the logical drives to the other controller as AVT is disabled. And the host cannot issue a failover of the logical drives as no RDAC is installed. The consequence is that the host loses connection to its logical drives. Be sure that the operating systems are using RDAC when you disable AVT. Otherwise, it loses access to the logical drive, since other systems will not failover properly as their multi-path I/O driver is based on AVT.

5.3. No multi-path driver with AVT enabled

If the hosts do not support multi-path driver you can still attach them to the DS4000 storage with two active controllers. Each logical volume will also have a preferred controller. However, as there is no multi-path driver there will not be a failover to the other controller in the case of a path failure. Again there is one situation were you may experience problems with the AVT. If two host systems without multi-path drivers are each connected to one of the controllers, you should use storage partitioning to divide the logical drives between the host systems. Otherwise, it may happen that a logical drive of host A is moved to the alternate controller by AVT, because host B tries to access this logical drive. Host B is sending I/O to the alternate controller of the logical drive, AVT will move the drive, and host A loses connection to his logical drive. Host A needs to gain access to his logical drive again by sending I/O to the preferred controller. The logical drive will move back again. As in the case of the concurrent access, the performance will be degraded significantly. But opposed to the situation above you do not have to disable AVT but use storage partitioning.

5.4. SAN boot limitations between RDAC and AVT

The possible LUN thrashing is specific to a Windows or Linux System booted from a SAN attached DS4000 storage array. When using RDAC, AVT must be disabled to avoid a condition where a LUN may thrash between controllers due to receiving commands from RDAC while being managed by AVT. In this event, if a failover to a non-preferred path has occurred, an attempt to reboot the system may result in the system failing to reboot. The following actions may need to be performed to recover the storage array: Verify that the Boot LUN is on the preferred controller; and if not, resolve any issues that caused the LUN to failover to the alternate controller. Verify that all paths to both controllers are available and active. Verify that the HBA BIOS successfully configures the boot LUN.

Copyright IBM Corporation 2010. All rights reserved.

Page 14 of 15

Reference
Fibre Array Storage Technology A FAStT Introduction http://www.redbooks.ibm.com/redbooks/pdfs/sg246246.pdf IBM System Storage DS4000 Storage Manager - Installation and support Guide for Windows 2000/Server 2003, NetWare, Esx Server, and Linux. http://ps-2.kev009.com:8081/DS4xxx/Docs/gc26784703.pdf

Copyright IBM Corporation 2010. All rights reserved.

Page 15 of 15