You are on page 1of 64

Stefan Gocke

Consultant

Gocke IT Solutions

PowerHA SystemMirror for AIX V7.1.x


Whats New
Based on work with Bernd Bhler (IBM STG Lab Service)

Copyright IBM Corporation 2015


Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.
9.0
IBM Technical University Prague 2015

Whats new in PowerHA SystemMirror V7.1.3

Unicast-based heartbeat
Dynamic host name change
Offers two types for dynamically changing the host name: Temporary or permanent.
Cluster split and merge handling policies
Operator-managed manual failover policy for multisite linked clusters.
Cluster Aware AIX (CAA) enhancements
Scalability (CAA now supports up to 32 nodes)
Dynamic host name and IP address support
Unicast support (supports IP unicast and IP multicast)
IBM HyperSwap enhancements
Active-active sites
One node HyperSwap
Auto resynchronization of mirroring
Node level unmanage mode support
Enhanced repository disk swap management
Dynamic policy management support
Enhanced verification and RAS

2 2015 IBM Corporation


IBM Technical University Prague 2015

Whats new in PowerHA SystemMirror V7.1.3

clmgr enhancements
Enhancements for PowerHA plug-in for Systems Director
Restore snapshot wizard
Cluster split/merge support
Cluster simulator
Smart Assist Enhancements for SAP
Support for SAP instance installation variations supported by SAP
Support for local configuration installation for SAP instances
Pure Java stack support
Multiple SIDs support
SAP configuration tunables customization
Support for internal/external NFS
Manual configuration enhancements
Updated support for IBM Systems Director
Resource group configuration enhancements (miscellaneous data, dependencies, etc.)

3 2015 IBM Corporation


IBM Technical University Prague 2015

Whats new in PowerHA SystemMirror V7.1.2

Version 7.1.2 offers both a Standard and an Enterprise Edition.


The Enterprise Edition provides for Disaster Recovery solutions with both host based
mirroring and storage based mirroring

IPv6 support is enabled with this version for v7 product


HyperSwap capability is introduced.
HyperSwap with DS8800 storage subsystems provides for continuous availability against
storage failures.

Support for multi sites Disaster Recovery management


Cross Site Mirroring using LVM mirror pools
Enhancements to the Director plugin to facilitate the use of these new
features
Software Levels Required:
OS AIX 6.1 TL8 SP1
OS AIX 7.1 TL2 SP1
PowerHA SystemMirror 7.1.2 SP1
Add. SW req. for EE and HyperSwap PowerHA SystemMirror 7.1.2 APAR IV27586
4 2015 IBM Corporation
IBM Technical University Prague 2015

Whats new in PowerHA SystemMirror Standard Edition V7.1.1

Federated Security
for single-point-of-control, cluster-wide security management.
SAP LiveCache Hot Standby
for fast failover with IBM DS8000 and SAN Volume Controller (SVC).
Smart Assist enhancements
Addition of MQSeries Smart Assist to the Smart Assist portfolio for out-of-the-box
middleware high availability management. Some of the Smart Assists also have been
qualified for later versions of middleware products.
Support for Mirror pools in Cluster Single Point of Control (C-SPOC)
Disk renaming across the cluster
Get the same name on each node
Display the UUID
Third party disk support
The ability to use EMC Powerpath and Hitachi HDLM multi-pathing based disks as
repository disks.

5 2015 IBM Corporation


IBM Technical University Prague 2015

Whats new in PowerHA SystemMirror Standard Edition V7.1.1

Foreground Application Start


Respond immediately to start failure
Mount Guard
A new JFS2 facility to help prevent accidental double mounts
Private Networks
Reserve a network for Oracle
DARE Progress Indicators
Whats going on, and when is it done
Repository Resiliency
CAA backup/restore
Heart Beat Tuning

6 2015 IBM Corporation


IBM Technical University Prague 2015

PowerHA 6.1

RSCT

Resource Monitoring
and Control
Resource Manager Group Services

Topology Services

AIX 6.1/7.1
7 2015 IBM Corporation
IBM Technical University Prague 2015

PowerHA 7.1

RSCT

Resource Monitoring
and Control
Resource Manager Group Services

CAA
AIX 6.1/7.1
8 2015 IBM Corporation
IBM Technical University Prague 2015

Default Multi Channel Health Management


Minimal Setup
Multiple channels of
communication
Network
SAN
Central Repository
Host 1 Host 2

Reliable Heartbeats Reliable


Heartbeats
Messaging Messaging

First line of Defense Network

Second line of Defense SAN

Third line of Defense Cluster Repository


Heartbeats

Triple redundant communication pipe

9 2015 IBM Corporation


IBM Technical University Prague 2015

Cluster Aware AIX: Topology Management

V 6.1 V 7.1.0 V 7.1.3 V 7.1.3

Host 2 Host 2 Host 2

Host 1

Unicast
(Ring)
Host 3
Host 1
MULTICAST
Host 3

+ Host 1

Unicast
(Any to any)
Host 3

Host 4
Host 4 Host 4

PowerHA 6.1 PowerHA 7.1


Heartbeat Rings: detailed protocol Multicast or Unicast based protocol
Leader, Successor, Mayor etc Discover and use as many adapters as
Difficult to add/delete nodes possible
Use network and SAN as needed
Adapt to the environment: delay, subnet etc
Requires IP aliases management in the
Kernel based cluster message handling
subnet
10 2015 IBM Corporation
IBM Technical University Prague 2015

CAA Unicast details

Host 2
Uses one network for Heartbeat
Network with fastest roundtrip time used
TCP/IP Keep Alive messages sent on
remaining networks
All IP networks use a any to any Host 3
Host 1
connection

Unicast
(Any to any)

Host 4
Example: lscluster -m
node1:/ # lscluster -m
Calling node query for all nodes...
Node query number of nodes examined: 2
. . .
Interface State Protocol Status
SRC_IP->DST_IP
-----------------------------------------
tcpsock->03 UP IPv4 none
192.168.1.1->192.168.1.2

11 2015 IBM Corporation


IBM Technical University Prague 2015

PowerHA and CAA Cluster Components


In HA The first step is to build the topology Initial Cluster Setup
SystemMirror Topology: Cluster, nodes, networks, adapters/interfaces
AIX Cluster: CAA repository disk constructs, cluster IP address

pha_cluster

CAA cluster IP address

net_ether_02

net_ether_01

Network interfaces CAA repository


disk
node1 node2
hdisk1 hdisk1

12 2015 IBM Corporation


IBM Technical University Prague 2015

Configure SAN heart beating in virtual environment

2015 IBM Corporation


IBM Technical University Prague 2015

PowerHA SystemMirror 7.1 TL03 (7.1.3)

GA: DEC 2013

14 2015 IBM Corporation


IBM Technical University Prague 2015

Dynamic hostname change

Permanent hostname change:


smitty hostname
Persists across reboot
Sets COMMUNICATION_PATH in HACMPnode
Resets handle field in HACMPcluster (on the node where hostname changed)
sync required

Temporary hostname change:


hostname command
Does not persist across reboot

clmgr has a new option to override such that temp changes behaves the same as
permanent change
[ TEMP_HOSTNAME={disallow|allow} ]

15 2015 IBM Corporation


IBM Technical University Prague 2015

PowerHA Split/Merge Handling

X
Two site cluster Two site cluster
Split Cluster 1 Cluster Merge
2

Policy Setting Split Merge Approach

Manual Manual steps needed for recovery to continue

Tie Breaker Tie break Holder side wins

Majority Rule Greater of N/2 side wins


Else, side that includes node with the smallest node id
wins
Priority Operator chooses a numerical value such as largest
serial number

16 2015 IBM Corporation


IBM Technical University Prague 2015

Manual (operator controlled failover)

Split/Merge Policies
Administrator prompts
Cluster will wait for Admin inputs
Optional Policy: After N prompts
allow auto-recovery
Custom action scripts can invoked at
the time of split or merge as well
site down

Defaults X
Number of prompts (N)=infinite
Interval between notifications: once in
30 seconds and then increasing in
frequency
Auto-Recovery after N prompts cluster split

17 2015 IBM Corporation


IBM Technical University Prague 2015

clmgr improvements

Embedded Hyphen and Leading Digit Support in Node Labels


For example: 2ndnode or first-node.
Native HTML Report
This is part of the base product. The main benefits are:
Contains more cluster configuration information than any other report
Can be scheduled to run automatically via AIX core functionality like cron
Portable, so it can send by email without loss of information
Fully translated
Allows for inclusion of a company name or logo into the report header
Cluster Copying
Allows the administrator to take a snapshot from a fully configured and tested cluster which
can then be restored on a new hardware, or LPAR

Syntactical Built-In Help


Lists all possible inputs for an operation
Shows valid groupings
Provides complete required versus optional input information
Provides standard versus verbose modes

18 2015 IBM Corporation


IBM Technical University Prague 2015

Native HTML Report


A sample clmgr HTML cluster report:

2015 IBM Corporation


IBM Technical University Prague 2015

PowerHA SystemMirror 7.1 TL02 (7.1.2)

GA: DEC 2012

20 2015 IBM Corporation


IBM Technical University Prague 2015

PowerHA SystemMirror 7.1.2 Enterprise Edition

High Availability and Disaster Recovery across multi site


Compute infrastructure deployment New York London

PowerHA SystemMirror for AIX Enterprise Edition Network


Adds long distance failover for Disaster Recovery
Low cost host based mirroring support
Host Mirroring
Extensive support for storage array replication Site 1 Site 2
Short distance (~100KMs) deployment: Synchronous
Fiber
Long distance (1000s of KM) deployment: Asynchronous

Storage Mirroring

Supported Mirroring Technologies


Enterprise Edition
Replication Technology Sync Async
Host Replication Geo LVM
IBM DS8K Series Storage - PPRC
SVC, Storevize,
XIV
EMC SRDF *
Storage Array Hitachi Universal Replicator,Truecopy *
Replication
HP Continuous Access *

21
* Support is expected later in 1H2013 2015 IBM Corporation
IBM Technical University Prague 2015

PowerHA 7.1 Multi Site Solutions (2012: Two Site Support)

Multi Sites Stretched Linked


Site 1 Site 2
Cluster Clusters
Inter Site Communication Multicast Unicast

Repository Disk Shared Separate

Cluster Communication Networks Networks


SAN SAN *
Repository Disk
Disk
Cross Site LVM Mirroring

Fig 1: Multi Sites with Stretched Cluster HyperSwap

Multi Site Concurrent RG w/ HyperSwap

Standard Enterprise

Site 1 Site 2 Multi Site Definition


Site Service IP
Site Policies
Stretched Cluster
Links Linked Clusters
Repository Repository
Disk 1 Disk 2 HADR with Storage
Replication Management
HyperSwap
Fig 2: Multi Sites with Linked Clusters
* Future Support

22 2015 IBM Corporation


IBM Technical University Prague 2015

Tie Breaker Support

Site 1 Site 2
PowerHA 7.1.2 Tie Breaker Support Cluster
Separate Site Split and Merge policies
Split/Merge: Tie Breaker policy
FC/iSCSI Tie Breaker
SCSI 3 reservation disk
Losing side is queisced. SCSI or iSCSI

Shared Disk
Tie Breaker
More suited
Suited for Linked
for Linked Clusters
Clusters

Site 3

Policy Setting Split Merge Comments

Tie Breaker Tie break Holder side wins

Majority Rule >N/2 side wins


In case of N/2, side that includes node with
the smallest node id
Manual Manual steps needed for recovery to
continue

23 2015 IBM Corporation


IBM Technical University Prague 2015

HyperSwap Technology
HA/DR
Continuous Availability against Storage
Hyperswap
failures
Technology
Application
Substitutes storage secondary to take the
place of failed primary device
Cluster
Non-disruptive - applications keep running
Key value add to HA/DR deployments Hyperswap

Sync
Customer Benefits
Mirror
Unplanned HyperSwap:
Continuous Availability against
storage failures Primary DS8K Secondary DS8K
Planned HyperSwap: Site 1 Site 2
Storage Maintenance without
Legend:
downtime Active Path
Storage migration without Passive Path
downtime
24 2015 IBM Corporation
IBM Technical University Prague 2015

HyperSwap Metro: AIX Host to DS8800 Communication

AIX AIX
PowerHA PowerHA
Device Driver

Data reads & Data reads &


Configuration, Date Writes Configuration, Date Writes
Events FiberChannel Events FiberChannel
(Network) (SCSI) (SCSI)

Management Storage Management Storage


\
Interface Controller \
Interface Controller

Fig 1: Out of band control of DS8K Series Storage Fig 2: Inband control of DS8800 Storage

Inband FC-SCSI advantages


Better performance (host to storage interactions
perform better)
Reliable and consistent communication

25 2015 IBM Corporation


IBM Technical University Prague 2015

HyperSwap Support by AIX-PowerHA


HyperSwap device configuration transparent to application
Application can continue to use the device as before

Application/LVM/Middleware Application/LVM/Middleware

/dev/hdiskX
HyperSwap Pair

/dev/hdiskX /dev/hdiskY Configure /dev/hdiskX /dev/hdiskY


HyperSwap

SYNC
SYNC

Primary DS8K Secondary DS8K Primary DS8K Secondary DS8K

26 2015 IBM Corporation


IBM Technical University Prague 2015

HyperSwap Support by AIX-PowerHA: contd..

Site 1 Site 2
HyperSwap coordination across hosts and sites N1 N2 N3 N4
Planned or unplanned HyperSwap based on multi host
synchronization

Consistency group management across DS8K systems


Goal: Swap times less than 60 seconds
Results so far: 5 to 10 seconds

HyperSwap Support for critical system disks


S2
Rootvg S1 S3
Paging device
Dump Devices
Repository disk

Disk Grouping Support


Groups disks and establish consistency groups

Support for both AIX LVM and Raw disks


Disk or VG preparation Site 3
Disk Error handling
Oracle can be deployed with LVM or ASM disks
Shared Disk SCSI or iSCSI
Tie Breaker

27 2015 IBM Corporation


IBM Technical University Prague 2015

HyperSwap Multi Site Deployments: Oracle RAC Example


PowerHA Cluster

Compute Node outages: Site 1


Site 2
Active-Active workload provides continuous availability Oracle RAC
(Active) (Active) (Passive) (Passive)
Storage outages:
HyperSwap provides continuous availability N1-1 N1-2 N2-1 N2-2

SYNC
S1 S2
Active-Passive Sites
< 100 KM
Active-Active workload within a site
Active-Passive across sites Fig 1: Active-Passive HyperSwap
Continuous availability for site storage outages

Site 1 Site 2
Active-Active Sites (Future) Oracle RAC
Active-Active workload across sites (Active) (Active) (Active) (Active)
Continuous availability of site compute N1-1 N1-2 N2-1 N2-2
infrastructure and storage outages
Oracle RAC long distance deployment
SYNC
S1 S2
< 100 KM

Fig 2: Active-Active HyperSwap

28 2015 IBM Corporation


IBM Technical University Prague 2015

PowerHA 7.1.2 Director Plugin Enhancements

Wizards
Cluster Create Wizard
Single Site and Multi Site deployment
Resource Group Creation Wizard
Custom and Smart Assist based RG
deployment
SAP liveCache HotStandby solution Wizard
Federated Security Setup Wizard
Volume Group Create Wizard
Support for LVM Mirror Pools
Replication (Mirror) Group Wizard
HyperSwap Setup

Management Enhancements
Repository Disk/s Management
Resource Groups management
Snapshots, networks, log files etc
Reports Management
Notifications management
Event driven callouts
Capacity upgrade based fallovers
HyperSwap Management
File collections

29 2015 IBM Corporation


IBM Technical University Prague 2015

Discussion - Questions

Now or send your question and/or remarks per email

Stefan.Gocke@t-online.de

Thank You and enjoy the rest of the conference

2015 IBM Corporation


IBM Technical University Prague 2015

31 2015 IBM Corporation


IBM Technical University Prague 2015

BACKUP CHARTS

32 2015 IBM Corporation


IBM Technical University Prague 2015

PowerHA 7.1.2 Director Plugin: Multi Site Management

33 2015 IBM Corporation


IBM Technical University Prague 2015

PowerHA SystemMirror 7.1 TL01 (7.1.1)

GA: Dec 2011

34 2015 IBM Corporation


IBM Technical University Prague 2015

Introduction - PowerHA SystemMirror Federated Security

Centralized Multi Cluster Security Administration

Enable easy setup of Centralized Security


Administration System Security administration
Automated LDAP Configuration Role Based Access Controls
1. Auto Setup to create LDAP server/s (redundant
LDAP servers) Encrypted File System
2. Adapt to any existing LDAP infrastructure
LDAP based policy centralization
Supports Windows Active Directory
Roles can be used to administer PowerHA LDAP
SystemMirror (RBAC Support) (Policy
Cluster 1 Tables)
Easy Encrypted File System management
in the cluster

Customer Benefits:
Centralized Security Administration Cluster 2

Simplified Encrypted file system mgmt
Cluster 3
Role based PowerHA administration

35 2015 IBM Corporation


IBM Technical University Prague 2015

Federated Security

The Federated Security feature integrates support for :


Lightweight Directory Access Protocol (LDAP)
Role Based Access Control (RBAC)
Encrypted Files System (EFS)
System requirements
PowerHA SystemMirror Version 7.1.1, or later
IBM LDAP 6.2 (Correct version of Gskit and DB2 packaged with LDAP), or later (Applicable
for server and client both. )
Microsoft Windows Server:
Microsoft Windows Server 2003/R2 Active Directory
Microsoft Windows Server 2008/R2 Active Directory
Services for UNIX (SFU) 3.5, or later, or the Subsystem for UNIX-based Applications (SUA)
(Needed to configure rsh between AIX client and Windows server)
expect.base

Note:
If youre going to use LDAP, the listed software requirements must be present.
The LDAP server can be either IBM TDS, or Microsoft Active Directory; both are supported.
It is necessary to configure rsh between the cluster nodes and the LDAP server. Also, note
that the expect facility must be present.

36 2015 IBM Corporation


IBM Technical University Prague 2015

Introduction - SystemMirror Federated Security: Role Base


Administration (RBAC)
Setup LDAP based security administration
PowerHA SystemMirror Administration Roles
Role Description
ha_admin HA Administrator
Has the most Privileges
Can perform tasks:
Configure & Administer all of SystemMirror functions
Assign roles

ha_op HA Operator
Can perform tasks:
Start and Stop Workload Resource Groups
Monitor the Workload health
Target Delegation: Workload management within the cluster
ha_mon HA Monitor
Can perform tasks:
Monitor the current status of certain Smart Assists and workloads
Helpful to assign to Database administrator and such person
ha_view HA View
Has the least amount of Privileges
Can perform tasks:
View SystemMirror log files
Target delegation: Support Personnel
37 2015 IBM Corporation
IBM Technical University Prague 2015

Introduction - SystemMirror Support For Encrypted File System

AIX Encrypted File System(EFS)


OS Integrated Encryption Capabilities SystemMIrror Cluster
Support for User and System wide encryption
Support for Middleware exploitation
SystemMirror EFS Support
Encrypted data and key Setup across the cluster
Multiple choices of Key Store setup
Shared Filesystem based Key Stores
Centralized LDAP Key Stores

Encrypted Keys
Data (Store)

Could be LDAP

38 2015 IBM Corporation


IBM Technical University Prague 2015

SAP liveCache Hot Standby Configuration

Master Standby
SAP SAP
PowerHA liveCache PowerHA
liveCache

Log
Volume

Data Data
Volumes FlashCopy Volumes

DS8K / SVC

39 2015 IBM Corporation


IBM Technical University Prague 2015

Mirror Pools and LVM Split Site Mirroring

No equivalent for LVM split site mirroring in PowerHA 7.1


Sites not supported

Mirror Pools and Repository Resiliency can provide equivalent


C-SPOC support for mirror pools
Back ported to PowerHA 6.1

Mirror Pool
Collection of disks in a volume group containing logical volume copies
Available only for scalable volume groups
Logical volume does not cross mirror pool boundary
Mirror pools do not cross volume groups
Names are arbitrary, and need be unique only to a volume group
Super Strict mirror pools
Each mirror pool contains a complete mirror copy of each logical volume

40 2015 IBM Corporation


IBM Technical University Prague 2015

Physical Volume Rename

A given physical volume may have different names on different nodes


Almost guaranteed if nodes access a different number of disks
Only volumes not already part of a volume group can be renamed
Pick list only gives disks not in a volume group
Can change all instances by PVID

Rename a Physical Volume

Type or select values in entry fields.


Press Enter AFTER making all desired changes.

[Entry Fields]

Physical Volume Name hdisk2


Physical Volume Identifier 00f638bc49c50b8a
Physical Volume is Known on these Nodes r3r6m21,r3r6m22

New Physical Volume Name []


Change all Physical Volumes with this PVID? no

41 2015 IBM Corporation


IBM Technical University Prague 2015

Configuring application startup mode

Application controllers started in the background by default


Add/Change controller menu has a new option for foreground startup

Add Application Controller Scripts

Type or select values in entry fields.


Press Enter AFTER making all desired changes.

[Entry Fields]
* Application Controller Name []
* Start Script []
* Stop Script []
Application Monitor Name(s) +
Application startup mode [background] +

Foreground start causes cluster event processing to wait for completion of


the application controller start script
Simplifies design of start scripts
Allows sequencing of resource groups with dependencies
Poorly designed scripts may cause hangs (config_too_long)
42 2015 IBM Corporation
IBM Technical University Prague 2015

Application startup mode - debug

Original design started all user supplied scripts in the background (ksh &)
such that cluster events would not hang or fail because of poorly written
user supplied scripts

New option applies to individual controllers:


Option stored in a new field in HACMPserver
Option exercised in start_server at the point where the script is called

Exit code of the script is not currently checked !


SP1 will change this non-zero exit will cause event error
Temp error is a future possibility, based on customer demand

hacmp.out will have tracing of the startup


SP1 will add timestamps

43 2015 IBM Corporation


IBM Technical University Prague 2015

Mount Guard

Preventing Double Mounts


Mounting a file system on two nodes at once corrupts it
LVM Active/Passive mode and CAA Storage Framework fencing help,
but can be defeated

AIX Mount Guard


A second mount without intervening unmount will be rejected
Mount state maintained on disk, does not require node interaction

Set by chfs option, resettable by logredo or chfs


PowerHA always sets if right AIX level, runs logredo

Available in all AIX levels required for PowerHA SystemMirror 7.1.1


bos.rte.filesystems 7.1.1 or 6.1.7

PowerHA support back ported to PowerHA 6.1 and 5.5

44 2015 IBM Corporation


IBM Technical University Prague 2015

Private Networks

Oracle requires that a network can be reserved to it


No heart beating or protocol traffic

PowerHA 6.1 and prior supported this


PowerHA 7.1.0 did not

PowerHA 7.1.1 restores ability to declare a network as private

Interfaces restricted from use by CAA


PowerHA lists local private network interfaces in /etc/cluster/ifrestrict
Do not restrict the interface that has the host name

Improved configuration
Can change network attribute without redefinition, provided cluster is down
Restriction to be removed in the service stream

45 2015 IBM Corporation


IBM Technical University Prague 2015

DARE Progress Indicators

Completion of a configuration change is not apparent


Users could start up overlapping or contradictory changes

With PowerHA 7.1.1, users terminal remains locked during DARE


processing

Progress indicators displayed


Show Cluster Manager state, on-going events
The progress indicator tracks cluster manager state via lssrc.
When it has been stable for 3 seconds, the event is declared complete.

Plan to fine-tune display


Remove confusing and redundant information

Plan to back port to PowerHA 6.1

46 2015 IBM Corporation


IBM Technical University Prague 2015

Example of DARE Progress Indicators

Cluster Manager Current state: ST_RP_RUNNING


Cluster Manager Current state: ST_RP_RUNNING
Cluster Manager Current state: ST_RP_RUNNING
Cluster Manager Current state: ST_RP_RUNNING
Cluster Manager Current state: ST_BARRIER
Cluster Manager Current state: ST_RP_RUNNING
Cluster Manager Current state: ST_RP_RUNNING
Cluster Manager Current state: ST_RP_RUNNING
Cluster Manager Current state: ST_RP_RUNNING
Cluster Manager Current state: ST_BARRIER
Cluster Manager Current state: ST_RP_RUNNING
Cluster Manager Current state: ST_RP_RUNNING
Cluster Manager Current state: ST_CBARRIER
Cluster Manager Current state: ST_UNSTABLE
Cluster Manager Current state: ST_UNSTABLE
Cluster Manager Current state: ST_BARRIER
Cluster Manager Current state: ST_BARRIER
Cluster Manager Current state: ST_RP_RUNNING
Cluster Manager Current state: ST_UNSTABLE
Cluster Manager Current state: ST_UNSTABLE
Cluster Manager Current state: ST_STABLE
Cluster Manager Current state: ST_STABLE
Cluster Manager Current state: ST_STABLE
...Completed
47 2015 IBM Corporation
IBM Technical University Prague 2015

Repository Resilience

In PowerHA 7.1.0, the node shuts down on when the repository disk fails
Disk failure or lost connection

CAA will provide Repository Resiliency


Requires AIX 6.1.7 SP4 or AIX 7.1.0 SP3, PowerHA 7.1.1 SP1
Node continues running even on repository disk failure, using locally cached information
User can provide a new disk on which to rebuild the repository
No changes allowed while repository is out of service

On repository failure
Message posted to hacmp.out
Repeated on config_too_long pattern
DARE and sync continue to function, but any CAA topology changes are rejected

User must recognize repository failure, and allocate a new disk


SMIT path under Manage the Cluster -> Select a new Repository Disk

48 2015 IBM Corporation


IBM Technical University Prague 2015

Heart Beat Tuning

New Heartbeat Tuning Parameters


Grace Period: The amount of time (seconds) the node will wait before marking a node as
DOWN. Accepted values are between 5 and 30 Seconds.
Failure Cycle: The frequency of the heartbeat. Accepted values are between 1 and 20
seconds

Settings apply to all networks across the cluster.

To change these settings from Smitty sysmirror Cluster Nodes and


Networks Manage the Cluster Cluster Heartbeat Settings

These settings can be modified from command line using clmgr command
Clmgr modify cluster HEARTBEAT_FREQUENCY= 10000 GRACE_PERIOD=5000

The settings will take effect only after the next sync

49 2015 IBM Corporation


IBM Technical University Prague 2015

Comparison of Heart Beat Parameters

RSCT (topsvcs) CAA

Heartbeat settings are same for all networks in the


Heartbeat settings can be defined for each network
cluster. However, PowerHA 7.1.1 supports only
type(nim).
ethernet adapters

The settings for heartbeat are The settings for heartbeat are
Grace period Grace period
Failure Cycle Failure cycle
Interval between Heartbeats

Failure cycle is the time that another node may


The combination of heartbeat rate and failure cycle
consider the adapter to be DOWN if it receives no
determines how quickly a failure can be detected and
incoming heartbeats.
may be calculated using this formula:
Actual heartbeat rate is calculated depending on the
(heartbeat rate) * (failure cycle) * 2 seconds
Failure cycle.

Grace period is the waiting time period after detecting Grace period is the waiting time period after detecting
the Failure before it is reported. the Failure before it is reported.

50 2015 IBM Corporation


IBM Technical University Prague 2015

Supported Migrations

Migrating from versions 6.1.0 and prior

An AIX command (/usr/sbin/clmigcheck) must be run before version 7.1.1 is installed.


This is the same procedure as migration from version 6.1 and prior to version 7.1.0, and
is documented in the installation guide.

Rolling, snapshot, and offline migrations are supported from the following versions:
Version 5.4.1, Version 5.5.0, Version 6.1.0

Migrating from version 7.1.0

Since both are CAA based, you do not use the AIX clmigcheck command.

Snapshot and offline are the only supported migration techniques

51 2015 IBM Corporation


IBM Technical University Prague 2015

Limitations Migrating from Versions 6.1 and prior

Not all configurations can be migrated


Configurations with FDDI, ATM, X.25, and Token Ring, can not be migrated and must be
removed from the configuration
Configurations with IPAT via Replacement or Hardware Address Takeover can not be
migrated and must be removed from the configuration
Configurations with Heartbeat via Aliasing can not be migrated and must be removed from
the configuration

Non-IP Networking is accomplished differently


RS232, TMSCSI, TMSSA, Disk Heartbeat are not supported , configuration data will not be
in the migrated cluster

PowerHA/XD configurations can not be migrated to version 7.1.1

Due to the radically different communication infrastructure and AIX


migration, active Rolling Migration will not be outage free.

52 2015 IBM Corporation


IBM Technical University Prague 2015

Limitations Migrating from Version 7.1.0

When migrating from version 7.1.0 a cluster outage is required

The only types of migration supported when migrating from version 7.1.0 are
snapshot
offline migration

53 2015 IBM Corporation


IBM Technical University Prague 2015

Additional Resources
PowerHA Website
www.ibm.com/systems/power/software/availability/
PowerHA SystemMirror 7.1 Beta Program.June time frame contact me if your interested
Availability Factory
Contact your IBM representative or an IBM Business Partner and they will contact us via e-mail (hacoc@us.ibm.com) to learn more.

IBM Technology Service Offering for PowerHA SystemMirror XD deployment


http://www-935.ibm.com/services/us/index.wss/offering/its/a1000032
Redbooks
SG24-7739 : PowerHA for AIX Cookbook
SG24-7841 : Exploiting IBM PowerHA SystemMirror Enterprise Edition
SG24-7845 : IBM PowerHA SystemMirror 7.1 for AIX
Education: PowerHA for AIX Implementation, Configuration and Administration AN610
Go to www.ibm.com/services/learning, search for AN610 or PowerHA coming soon
Education: Lab Services AN44 Extended Distance and Disaster Recovery
Go to www.ibm.com/services/learning, search for AN44
GLVM white paper
www.ibm.com/systems/resources/systems_p_os_aix_whitepapers_pdf_aix_glvm.pdf
clmgr white paper
www.ibm.com/systems/resources/systems_power_software_availability_clmgr_tech_guide.pdf
IBM storage virtualization offerings
www.ibm.com/systems/storage/virtualization
SAP consulting services for POWERHA and POWERVM
gehenni@us.ibm.com
sbranden@us.ibm.com
Wiki
http://www.ibm.com/developerworks/wikis/display/WikiPtype/High%20Availability

54 2015 IBM Corporation


IBM Technical University Prague 2015

55 2015 IBM Corporation


IBM Technical University Prague 2015

Offering highlights V7.1

HACMP has a new name PowerHA SystemMirror


rebranding now complete publications, binaries, etc

Key dates:
General Availability: September 10, 2010
Lifecycle information:
http://www-01.ibm.com/software/support/lifecycle/index_h.html

Offerings:
Standard Edition has base function plus Smart Assists
New features added to Enterprise Edition 6.1 (only) no 7.1 EE

RSCT and AIX


AIX 6.1 TL 6 (SP3 recommended), 7.1 (with SP1)
RSCT 3.1 works with all versions of AIX

56 2015 IBM Corporation


IBM Technical University Prague 2015

New features Standard Edition 7.1

Cluster Aware AIX (CAA)


Functions built into base operating system

Simplified user interface


SMIT changes, new terminology

Disk Handling
ECMVGs now required
Disk fencing to protect against improper usage of disks

New Smart Assists


Expanded middleware support

IBM Systems Director plug-in


Integrates SystemMirror management with Director
57 2015 IBM Corporation
IBM Technical University Prague 2015

Disk Handling Changes ECMVG required


All PowerHA shared volume groups are Enhanced Concurrent Mode
Existing volume groups automatically converted
No user action required, no override allowed
Done by call to cl_makecm out of node_up
C-SPOC creates all volume groups as ECM
Either Fast Disk Takeover or Concurrent Access
Active/Passive mode used for non-concurrent resource groups

No SCSI-2 disk reserves set or broken


Most disk differences now irrelevant
Disk reserve handling code cl_disk_available retained for migration
Fast path through code if ECM and no reserves

58 2015 IBM Corporation


IBM Technical University Prague 2015

System Director Plug-in: Basic Architecture

Three-tier architecture provides scalability:


User Interface
User Interface Management Server Director Agent Web-based interface
Command-line interface

Director Agent
Automatically installed on AIX 7.1 & AIX V6.1 TL06 AIX

PowerHA Director
P D
Agent

P D

P D
Secure communication
P D

Director Server
Central point of control
P D
Supported on AIX,
Linux, and Windows
P D Agent manager
Discovery of clusters
P D and resources

59 2015 IBM Corporation


IBM Technical University Prague 2015

System Director Plug-in Getting Started

60 2015 IBM Corporation


IBM Technical University Prague 2015

Monitoring Services
All communication interfaces are monitored
Cluster Aware AIX tells you what interfaces have been discovered on a node and
information on those interfaces including state

All cluster disks are monitored


Cluster Aware AIX tells you what disks are in the cluster and information on
those disks including state

All monitors implemented at a low-level of the AIX kernel, therefore they are largely
insensitive to system load

All nodes are monitored


Cluster Aware AIX tells you what nodes are in the cluster and information on
those nodes including state. A special gossip protocol is used over the
multicast address to determine node information and implement scalable reliable
multicast. No traditional heartbeat mechanism is employed. Gossip packets
travel over all interfaces including storage.

61 2015 IBM Corporation


IBM Technical University Prague 2015

LVM Split Site Equivalent

Assumes SAN connected disks and nodes at two locations

Define shared volume group with super strict mirror pools


Mirror pool for each location
Disks must be manually assigned to each mirror pool
Knowing which disks are where is a user responsibility
LVM mirrors logical volume between two locations
Resource group definition should allow forced varyon

In the event of node and disk loss at one location


Volume group forced on line at other location by PowerHA
Mirror pool set up guarantees a local copy of the data
Manual recovery of repository using Repository Resiliency

62 2015 IBM Corporation


IBM Technical University Prague 2015

Configure SAN heart beating in virtual environment

63 2015 IBM Corporation


IBM Technical University Prague 2015

Continue growing your IBM skills

ibm.com/training provides a
comprehensive portfolio of skills and career
accelerators that are designed to meet all
your training needs.

Training in cities local to you - where and


when you need it, and in the format you want
Use IBM Training Search to locate public training classes
near to you with our five Global Training Providers
Private training is also available with our Global Training Providers

Demanding a high standard of quality


view the paths to success
Browse Training Paths and Certifications to find the
course that is right for you

If you cant find the training that is right for you with
our Global Training Providers, we can help.
Contact IBM Training at dpmc@us.ibm.com

Global Skills Initiative

2015
Copyright IBM Corporation IBM Corporation
2015