You are on page 1of 252

Veritas Cluster

Server 6.0 for UNIX:


Cluster Management

100-002685-E

COURSE DEVELOPERS

Bilge Gerrits
Steve Hoffer
Siobhan Seeger
Pete Toemmes

LEAD SUBJECT MATTER


EXPERTS

Graeme Gofton
Sean Nockles
Brad Willer

TECHNICAL
CONTRIBUTORS AND
REVIEWERS

Copyright 2012 Symantec Corporation. All rights reserved.

Geoff Bergren
Kelli Cameron
Tomer Gurantz
Anthony Herr
James Kenney
Gene Henriksen
Bob Lucas
Paul Johnston
Rod Pixley
Clifford Barcliff
Danny Yonkers
Antonio Antonucci
Satoko Saito
Feng Liu

Copyright 2012 Symantec Corporation. All rights reserved.


Symantec, the Symantec Logo, and VERITAS are trademarks or
registered trademarks of Symantec Corporation or its affiliates in
the U.S. and other countries. Other names may be trademarks of
their respective owners.
THIS PUBLICATION IS PROVIDED AS IS AND ALL
EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS
AND WARRANTIES, INCLUDING ANY IMPLIED
WARRANTY OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE
DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH
DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.
SYMANTEC CORPORATION SHALL NOT BE LIABLE FOR
INCIDENTAL OR CONSEQUENTIAL DAMAGES IN
CONNECTION WITH THE FURNISHING, PERFORMANCE,
OR USE OF THIS PUBLICATION. THE INFORMATION
CONTAINED HEREIN IS SUBJECT TO CHANGE WITHOUT
NOTICE.
No part of the contents of this book may be reproduced or
transmitted in any form or by any means without the written
permission of the publisher.
Veritas Cluster Server 6.0 for UNIX: Cluster Management
Symantec Corporation
World Headquarters
350 Ellis Street
Mountain View, CA 94043
United States
http://www.symantec.com

Table of Contents
Course Introduction
Lesson 1: Service Group Dependencies
Common application relationships ............................................................... 1-3
Service group dependencies........................................................................ 1-7
Service group dependency examples .......................................................... 1-9
Configuring service group dependencies ................................................... 1-12
Alternative methods of controlling interactions........................................... 1-16
Lesson 2: Reconfiguring Cluster Membership
Adding a system to a cluster ........................................................................ 2-3
Merging clusters ........................................................................................... 2-6
Additional reconfiguration tasks ................................................................. 2-10
Lesson 3: Startup and Failover Policies
Startup rules and policies ............................................................................. 3-3
Failover rules and policies.......................................................................... 3-10
Limits and Prerequisites ............................................................................. 3-18
Modeling startup and failover policies ........................................................ 3-22
Lesson 4: Alternate Network Configurations
Multiple service groups with NIC resources ................................................. 4-3
Multiple public interfaces .............................................................................. 4-8

Copyright 2012 Symantec Corporation. All rights reserved.

Lesson 5: High Availability in the Enterprise


Veritas Operations Manager ........................................................................ 5-3
Disaster recovery enhancements................................................................. 5-8
Virtualization support.................................................................................. 5-15

Appendix A: Labs
Lab 1: Service group dependencies............................................................ A-3
Lab 2: Merging clusters .............................................................................. A-13
Lab 3: Failover policies............................................................................... A-25
Lab 4: Creating a parallel network service group ....................................... A-37
Appendix B: Lab Solutions
Lab 1: Service group dependencies............................................................ B-3
Lab 2: Merging clusters .............................................................................. B-37
Lab 3: Failover policies............................................................................... B-65
Lab 4: Creating a parallel network service group ....................................... B-89
Appendix C: Supplemental Content
Service group dependenciesFailover process......................................... C-2

Table of Contents

i
Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

ii

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Course Introduction

Veritas Cluster Server curriculum path

Copyright 2012 Symantec Corporation. All rights reserved.

The Veritas Cluster Server for UNIX curriculum is a series of courses that are
designed to provide a full range of expertise with Veritas Cluster Server (VCS)
high availability solutionsfrom design through implementation.
Veritas Cluster Server for UNIX: Install and Configure
This course covers installation and configuration of common VCS
environments, focusing on two-node clusters running application and database
services.
Veritas Cluster Server for UNIX: Manage and Administer
This course focuses on multinode VCS clusters and advanced topics related to
managing more complex cluster configurations.
The course includes two participant guides:
Veritas Cluster Server for UNIX: Example Application Configurations
This guide provides examples of common service group configurations.
The instructor may opt to present some or all of this material in class,
depending on time constraints and student interest.
Veritas Cluster Server for UNIX: Cluster Management
This guide provides detailed information about configuring and managing
more complex clusters.
Veritas Cluster Server eLearning Library
The eLearning Library is available with bundled training options and includes
content on advanced high availability and disaster recovery features.

12

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Lab design for the course


The diagram shows a conceptual view of the cluster design used as an example
throughout this course and implemented in hands-on lab exercises.
Each aspect of the cluster configuration is described in greater detail where
applicable in course lessons.

Copyright 2012 Symantec Corporation. All rights reserved.

The cluster consists of:


Four nodes
Several high availability services
Fibre connections to SAN shared storage from each node through a switch
Two Ethernet interfaces for the cluster interconnect
Ethernet connections to the public network

Additional complexity is added to the design to illustrate certain aspects of cluster


configuration in later lessons.

Course Introduction

13
Copyright 2012 Symantec Corporation. All rights reserved.

Course overview

Copyright 2012 Symantec Corporation. All rights reserved.

This training provides comprehensive instruction on the deployment of advanced


features of Veritas Cluster Server (VCS). The course focuses on multinode VCS
clusters and advanced topics related to more complex cluster configurations, such
as service group dependencies and workload management.

14

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Lesson 1

Service Group Dependencies

Copyright 2012 Symantec Corporation. All rights reserved.

10

12

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Common application relationships


Multitier services
In most high availability environments, a collection of applications work together
to provide a service. The relationships between these application service groups
must be managed by VCS to ensure that startup, shutdown, and failover
procedures are coordinated properly.

Copyright 2012 Symantec Corporation. All rights reserved.

The example in the slide shows a typical three-tier model where:


A Web server provides an interface to network clients.
The Web server relies on an application that processes client requests.
The application processes requests from the Web server and relies on the
database to manage information.

11

The inherent relationships require that the database is started and running first,
then the application, and finally the Web server.
In a cluster environment, you must also consider which systems should run which
applications, as shown in the following examples.

Lesson 1 Service Group Dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

13

Example 1: Online on the same system


In this example of a two-tier service, both the Web server and the application must
be online on the same system. For example, the services may need to use
interprocess communication (IPC) mechanisms to exchange data.

Copyright 2012 Symantec Corporation. All rights reserved.

Furthermore, the application must come online first, followed by the Web server. If
the application faults and fails over to another cluster node, the Web server must
then be brought online on that same failover target system.

12

14

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Example 2: Online on different systems


In this example of a two-tier service, both the database and the application must be
online, but they cannot run on the same system. For example, the combined
resource requirements of each application may exceed the capacity of the systems,
and you want to ensure that they run on separate systems.

Copyright 2012 Symantec Corporation. All rights reserved.

In this scenario, the database must come online first, and then the application is
started only after the database is running. If the application faults and fails over to
another cluster node, the database can stay online on the original system, as long as
the application is not restarted on that same system.

13

Lesson 1 Service Group Dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

15

Example 3: Offline on the same system


One example relationship is where you have a test version of an application and
want to ensure that it does not interfere with the production version. You want to
give the production application precedence over the test version for all operations,
including manual offline, online, switch, and failover.

Copyright 2012 Symantec Corporation. All rights reserved.

The difference between this example and the previous example, is that in this case,
neither application requires the other to be online. The only requirement is that the
other application cannot be online on a system when one application is brought
online.

14

16

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Service group dependencies


VCS provides service group dependencies to manage application relationship
requirements in multi-application environments.

Copyright 2012 Symantec Corporation. All rights reserved.

Service group dependency definitions

15

There are four basic criteria for defining how services interact when using service
group dependencies.
A service group can require another group to be online or offline in order to
start and run.
You can determine the startup order for service groups by designating one
group the child and another a parent. Parent groups depend on child groups. If
service group B requires service group A to be online in order to start, then B is
the parent and A is the child.
You can specify where the groups must be online or offline.
For all online dependencies, the child group must be online in order for the
parent to start. A location of local, global, or remote determines where the
parent can come online relative to where the child is online.
For offline local, there is no requirement for either service group to start
first. The child group must be offline for the parent to come online.
Failover behavior of linked service groups is specified by designating the
relationship soft, firm, or hard.
Detailed information about how these dependencies impact service group
operations is provided in Veritas Cluster Server Administrators Guide.

Lesson 1 Service Group Dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

17

Failover types
The dependency type determines failover behavior for linked service groups.
Soft dependency
A soft dependency means that VCS does not immediately take the parent offline if
the child group faults.
When both groups are online, the child group can be taken offline while the parent
is online. Likewise, the parent group can be taken offline while the child is online.
The parent remains online if the child group faults and cannot fail over. The child
remains online if the parent group faults.
Firm dependency
Copyright 2012 Symantec Corporation. All rights reserved.

A firm dependency means that VCS imposes additional constraints on the parent
and child service groups. Specifically:
The parent group must be taken offline when the child group faults.
When the child is brought online on another system, the parent group is
brought online on a system determined by the location type (local, global,
remote).
If the parent group faults, the child continues to run.

16

Hard dependency
A hard dependency means that VCS takes the other service group offline when
either the child or parent group faults.

18

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Service group dependency examples


Online local example
In an online local dependency, a child service group must be online on the same
system before the parent service group can come online on that system.
Online local soft
An online local soft dependency designates that the parent service group remains
online when the child service group faults. If the child service group fails over and
is brought online on another system, the parent service group is then migrated to
that same system.

Copyright 2012 Symantec Corporation. All rights reserved.

Online local firm

17

An online local firm is similar to soft, except the parent service group is taken
offline when the child faults. If the child fails over and is restarted on another
system, the parent is then also started on that system
Online local hard
In an online local hard dependency, the child service group is taken offline if the
parent service group faults. Online local is the only dependency supporting the
hard type.

Lesson 1 Service Group Dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

19

Online remote example


In an online remote dependency, a child service group must be online on a remote
system before the parent service group can come online on the local system.
Online remote soft
An online remote soft dependency designates that the parent service group remains
online when the child service group faults, as long as the child service group
chooses another system on which to fail over. If the child service group chooses to
fail over to the system where the parent was online, the parent service group is
migrated to any other available system.
Online remote firm
Copyright 2012 Symantec Corporation. All rights reserved.

An online remote firm is similar to soft, except the parent service group is taken
offline when the child faults.

18

110

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Offline local example

Copyright 2012 Symantec Corporation. All rights reserved.

In an offline local dependency, the parent service group can be started only if the
child service group is offline on the local system. Similarly, the child can only be
started if the parent is offline on the local system. This prevents conflicting
applications from running on the same system. This is commonly used when you
have a test version of an application running on the failover target system for the
production version of the application.

19

Lesson 1 Service Group Dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

111

Configuring service group dependencies


Service group dependency rules

Copyright 2012 Symantec Corporation. All rights reserved.

You can use service group dependencies to implement parent/child relationships


between applications. Before using service group dependencies to implement the
relationships between multiple application services, review the rules governing
these dependencies:
The child service group always has priority over the parent group.
Service groups can have multiple parent service groups.
This means that an application service can have multiple other application
services depending on it.
A service group can have multiple child service groups.
This means that an application service can be dependent on one or more
application services.
A group dependency tree can be no more than five levels deep.
Service groups cannot have cyclical dependencies.

20

112

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Creating service group dependencies


You can create service group dependencies from the command-line interface using
the hagrp command, using Veritas Operations Manager Web GUI, or using the
Cluster Manager Java GUI. To create a dependency, link the groups and specify
the relationship (dependency) type, indicating whether it is soft, firm, or hard.
If not specified, service group dependencies are firm by default.
To configure service group dependencies using the Cluster Manager Java GUI,
you can either right-click the parent service group and select Link to display the
Link Service Groups view that is shown on the slide, or you can use the Service
Group View.

Copyright 2012 Symantec Corporation. All rights reserved.

Linking constraints

21

To link a parent and child group with soft dependency, it is not required that the
child group be online if the parent is online. However, if the child group is also
online, the parent and child may not be linked in such a way that their online states
conflict with the type of link between the parent and child.
To link a parent and child group with firm dependency, the parent group must be
offline, or the parent and child group must be online in such a way that their online
states do not conflict with the type of link between the parent and child.
Removing service group dependencies
When removing a dependency, you do not need to specify the type of dependency,
because only one dependency is allowed between two service groups.

Lesson 1 Service Group Dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

113

Modeling service group dependencies

Copyright 2012 Symantec Corporation. All rights reserved.

You can use the Simulator on Windows to model how different types of links
affect service group behavior. This enables you to fully understand the
implications and the effects of different dependency configurations before you
configure links in your production environment.

22

114

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Online and offline of linked service groups


You can use the -propagate option to the hagrp command to simplify
bringing linked service groups online and offline. You can also combine
-propagate with the -any and -sys options for specifying where the service
groups are brought online.

Copyright 2012 Symantec Corporation. All rights reserved.

Note that propagation is supported only with local service group dependencies
type.

23

Lesson 1 Service Group Dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

115

Alternative methods of controlling interactions


Using triggers to control service group interactions

Copyright 2012 Symantec Corporation. All rights reserved.

VCS provides several event triggers that can be used to enforce service group
relationships where dependencies cannot be configured, as described in the
previous example. These triggers are useful for managing relationships between
service groups:
VCS runs the preonline script before bringing a service group online.
The PreOnline trigger must be enabled for each applicable service group by
setting the PreOnline service group attribute. For example, to enable the
preonline trigger for groupa, type:
hagrp -modify GroupB PreOnline 1
The preonline script must start the service group with the -nopre option
to hagrp.
The postonline script is run after a service group is brought online.
The postoffline script is run after a service group is taken offline.

24

The postonline and postoffline triggers are enabled automatically if the


script is present in the $VCS_HOME/bin/triggers directory. Be sure to copy
triggers to all systems in the cluster. When present, these triggers apply to all
service groups.
Consider implementing triggers only after investigating whether VCS native
facilities can be used to configure the desired behavior. Triggers add complexity,
requiring programming skills as opposed to simply configuring VCS objects and
attributes.
116

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Managing multitier applications

Copyright 2012 Symantec Corporation. All rights reserved.

One of the key tenets of a multitier architecture is the operational independence of


each layereach tier can make its own decisions without relying on an external
brain. This applies to high availability, as well. Each tier needs to have HA of its
own it should be able to independently handle the resilience of the application
component running in that tier.

25

Lesson 1 Service Group Dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

117

Virtual Business Services


Virtual Business Services provide continuous high availability and reduce
frequency and duration of service disruptions for multi-tier business applications
running on heterogeneous operating systems and virtualization technologies.
A Virtual Business Service represents the multi-tier application as a single
consolidated entity and builds on the high availability and disaster recovery
provided for the individual tiers by Symantec products such as Veritas Cluster
Server and Symantec ApplicationHA.
VBS is configured and managed using Veritas Operations Manager (VOM)
version 4.1 and later.

Copyright 2012 Symantec Corporation. All rights reserved.

In the example shown in the slide, the database cluster is the lowest level tier and is
the most critical component in this service. Typically, you would find the database
to be shared across multiple services as well. If something happens in the database
tier, for example, an active server fails or the database runs into software issues on
one of the servers, VCS ensures the database is failed over to another node in the
cluster. Additionally, disaster recovery may be configured so that if the production
site goes down, VCS fails the database over to the DR site.

26

The middle tier is an application cluster running on several servers in parallel in


order to meet user demands. VCS provide local high availability in this tier and
manages the dependencies on the database.
Finally, the web server top tier is running on Vmware virtual machines. Symantec
ApplicationHA is managing high availability for the Apache Web servers running
on Windows.

118

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Fault propagation between clusters


When a service group that is running on a cluster that is a member of a Virtual
Business Service faults, the fault is propagated to other clusters similarly to service
group dependencies.
A restart type dependency is specific to a VBS configuration. In the example
configuration shown in the slide, if oracle_sg faults, oracle_apps_sg ignores the
fault and continues to run. If or when the oracle_sg restarts and is in a running
state, oracle_apps_sg restarts.

Copyright 2012 Symantec Corporation. All rights reserved.

In other words, the parent service group with a restart dependency on a child
service group takes no action when a child service group faults until the child is
restarted.

27

This type of dependency is used when only start or stop ordering is required, and
no fault policy action is needed.

Lesson 1 Service Group Dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

119

Copyright 2012 Symantec Corporation. All rights reserved.

28

Labs and solutions for this lesson are located on the following pages.
Lab 1: Service group dependencies, page A-3.
Lab 1: Service group dependencies, page B-3.
120

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Lesson 2

Reconfiguring Cluster Membership

29

Copyright 2012 Symantec Corporation. All rights reserved.

30

22

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Adding a system to a cluster


Adding a system to a running VCS cluster
The objective of this task is to add a new system to a running VCS cluster with no
or minimal impact on application services. Ensure that the cluster configuration is
modified so that the application services can make use of the new system in the
cluster.

Copyright 2012 Symantec Corporation. All rights reserved.

Assumptions

31

For illustration purposes, these assumptions are used in describing how to perform
this task:
The VCS cluster consists of two or more systems, all of which are up and
running.
There are multiple service groups configured in the cluster. All of the service
groups are online somewhere in the cluster.
The new system to be added to the cluster does not have any VCS software.
The new system has the same version of operating system and VERITAS
Storage Foundation as the systems in the cluster.
The new system may not have all the required application software.
The storage devices can be connected to all systems.

Lesson 2 Reconfiguring Cluster Membership


Copyright 2012 Symantec Corporation. All rights reserved.

23

Copyright 2012 Symantec Corporation. All rights reserved.

Procedure for adding a system to a running cluster

32

24

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

33

Lesson 2 Reconfiguring Cluster Membership


Copyright 2012 Symantec Corporation. All rights reserved.

25

Merging clusters
Merging two running VCS clusters
The objective of this task is to merge two running VCS clusters with no or minimal
impact on application services. Also, ensure that the cluster configuration is
modified so that the application services can make use of the systems from both
clusters.
Assumptions

Copyright 2012 Symantec Corporation. All rights reserved.

The following is a list of assumptions that you need to take into account while
planning a procedure for this task:
All the systems in both clusters are up and running.
There are multiple service groups configured in both clusters. All service
groups are online somewhere in the cluster.
All the systems have the same version of operating system and Veritas Storage
Foundation.
The clusters do not necessarily have the same application services software.
New application software can be installed on the systems to support
application services of the other cluster.
The storage devices can be connected to all systems.
The cluster interconnects of both clusters are isolated before the merger.

34

For this example, assume that two two-node clusters are merged into a single fournode cluster.

26

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Procedure for merging two running clusters

35

Lesson 2 Reconfiguring Cluster Membership


Copyright 2012 Symantec Corporation. All rights reserved.

27

Copyright 2012 Symantec Corporation. All rights reserved.

36

28

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

37

Lesson 2 Reconfiguring Cluster Membership


Copyright 2012 Symantec Corporation. All rights reserved.

29

Additional reconfiguration tasks


Reconfiguring triggers and service groups
The objective of this task is to update the cluster configuration so that the
application services can make use of the systems from both clusters.
Assumptions

Copyright 2012 Symantec Corporation. All rights reserved.

The following is a list of assumptions that you need to take into account while
planning a procedure for this task:
Application resources have been prepared
Service group definitions have been merged
Agents have been installed

38

210

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Procedure for updating triggers and service groups

39

Lesson 2 Reconfiguring Cluster Membership


Copyright 2012 Symantec Corporation. All rights reserved.

211

Copyright 2012 Symantec Corporation. All rights reserved.

40

212

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

41

Labs and solutions for this lesson are located on the following pages.
Lab 2: Merging clusters, page A-13.
Lab 2: Merging clusters, page B-37.

Lesson 2 Reconfiguring Cluster Membership


Copyright 2012 Symantec Corporation. All rights reserved.

213

Copyright 2012 Symantec Corporation. All rights reserved.

42

214

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Lesson 3

Startup and Failover Policies

43

Copyright 2012 Symantec Corporation. All rights reserved.

44

32

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Startup rules and policies

Copyright 2012 Symantec Corporation. All rights reserved.

Rules for automatic service group startup

45

The following conditions must be satisfied for a service group to be automatically


started:
The service group AutoStart attribute must be set to the default value of 1. If
this attribute is changed to 0, VCS leaves the service group offline and waits
for an administrative command to be issued to bring the service group online.
In addition, the service group definition must have at least one system in its
AutoStartList attribute. Also, the service group cannot be frozen.
All of the systems in the service groups SystemList must be in the running
state so that the service group can be probed on all systems in SystemList.
If there are systems on which the service group can run that have not joined the
cluster yet, VCS autodisables the service group until it is probed on all the
systems.
And dependencies on child service groups must have been met before a service
group can be automatically started.

Lesson 3 Startup and Failover Policies


Copyright 2012 Symantec Corporation. All rights reserved.

33

Startup system selection

Copyright 2012 Symantec Corporation. All rights reserved.

The startup system for a service group is chosen as follows:


1 All systems included in the AutoStartList attribute are initial candidates for
service group startup.
2 Systems are culled from this initial list according to these criteria:
Frozen systems are eliminated.
Systems where the service group has a faulted status are eliminated.
Systems that do not meet the service group requirements are eliminated, as
described in detail later in the lesson.
3 The target system is then chosen from this list based on the startup policy
defined for the service group.

46

34

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Automatic startup policies


You can set the AutoStartPolicy attribute of a service group to one of these three
values:
Order: Systems are chosen in the order in which they are defined in the
AutoStartList attribute. This is the default policy for every service group.
Priority: Of the systems listed in the AutoStartList attribute, the system with
the lowest priority number in SystemList is selected.
Load: The system with the highest available capacity is selected.
The autostart policies are described in more detail in the following pages.
To configure the AutoStartPolicy attribute of a service group, execute:
Copyright 2012 Symantec Corporation. All rights reserved.

hagrp -modify group AutoStartPolicy policy

47

where possible values for policy are Order, Priority, and Load. You can
also set this attribute using Veritas Operations Manager or the Cluster Manager
Java GUI.
Note: The configuration must be open to change service group attributes.

Lesson 3 Startup and Failover Policies


Copyright 2012 Symantec Corporation. All rights reserved.

35

AutoStartPolicy=Order
When the AutoStartPolicy attribute of a service group is set to the default value of
Order, the first system available in AutoStartList is selected to bring the service
group online. The priority numbers in SystemList are ignored.
In the example shown on the slide, the A service group is brought online on s1,
although it is the system with the highest priority number in SystemList. Similarly,
the B service group is brought online on s2, and the C service group is brought
online on s3 because these are the first systems listed in the AutoStartList
attributes of the corresponding service groups.

Copyright 2012 Symantec Corporation. All rights reserved.

Note: Because Order is the default value for the AutoStartPolicy attribute, it is
not listed in the service group definitions in the main.cf file.

48

36

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

AutoStartPolicy=Priority
When the AutoStartPolicy attribute of a service group is set to Priority, the system
with the lowest priority number in the SystemList that also appears in the
AutoStartList is selected as the target system during start-up. In this case, the order
of systems in the AutoStartList is ignored.

Copyright 2012 Symantec Corporation. All rights reserved.

The same example service groups are now modified to use the Priority
AutoStartPolicy, as shown on the slide. In this example, the A service group is
brought online on s3, which has the lowest priority number in SystemList even
though it is listed as the last system in AutoStartList. Similarly, the B service group
is brought online on s1 (with priority number 0), and the C service group is
brought online on s1 (with priority number 1).

49

Notice how the startup systems have changed for the service groups by changing
the AutoStartPolicy attribute, although the SystemList and AutoStartList attributes
are the same for these two examples.

Lesson 3 Startup and Failover Policies


Copyright 2012 Symantec Corporation. All rights reserved.

37

AutoStartPolicy=Load
When AutoStartPolicy is set to Load, VCS determines the target system based on
the existing workload of each system listed in the AutoStartList attribute and the
load that is added by the service group.
These attributes control load-based startup:
Capacity is a user-defined system attribute that contains a value representing
the total amount of load that the system can handle.
Load is a user-defined service group attribute that defines the amount of
capacity required to run the service group.
AvailableCapacity is a system attribute maintained by VCS that quantifies the
remaining available system load.
Copyright 2012 Symantec Corporation. All rights reserved.

In the example displayed on the slide, three servers have Capacity set to 300, 200,
and 100, respectively. The s2 system is selected as the target system for starting C
because it has the highest AvailableCapacity value of 200 after A and B are started
on s1.

50

Determining Load and Capacity


You must determine a value for Load for each service group. This value is based
on how much of the system capacity is required to run the application service that
is managed by the service group.

38

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

When a service group is brought online, the value of its Load attribute is subtracted
from the system Capacity value, and AvailableCapacity is updated to reflect the
difference. Both the Capacity attribute of a system and the Load attribute of a
service group are static user-defined attributes based on your design criteria.

51

Lesson 3 Startup and Failover Policies


Copyright 2012 Symantec Corporation. All rights reserved.

39

Failover rules and policies


Rules for automatic service group failover

Copyright 2012 Symantec Corporation. All rights reserved.

The following conditions must be satisfied for a service group to be automatically


failed over after a fault:
The service group must contain a critical resource, and that resource must fault
or be a parent of a faulted resource.
The service group AutoFailOver attribute must be set to the default value of 1.
If this attribute is changed to 0, VCS leaves the service group offline and waits
for an administrative command to be issued to bring the service group online.
The ManageFaults attribute must be set to All, the default setting.
The service group cannot be frozen.
At least one of the systems in the service groups SystemList attribute must be
in a running state.

52

310

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Failover system selection

Copyright 2012 Symantec Corporation. All rights reserved.

The failover system for the service group is chosen as follows:


1 A subset of systems listed in the SystemList attribute is created first.
2 Systems that do not meet the service group requirements are eliminated,
including:
Frozen systems
Systems where the service group has a faulted status are eliminated from
the list.
Systems that do not meet the service group requirements are eliminated, as
described in detail later in the lesson.
3 The target system is chosen from this list based on the failover policy defined
for the service group.

53

Lesson 3 Startup and Failover Policies


Copyright 2012 Symantec Corporation. All rights reserved.

311

Failover policies
VCS supports a variety of policies that determine how a system is selected when
service groups must migrate due to faults. The policy is configured by setting the
FailOverPolicy attribute to one of these values:
Priority: The system with the lowest priority number is preferred for failover
(default).
RoundRobin: The system with the least number of active service groups is
selected for failover.
Load: The system with the highest value of the AvailableCapacity system
attribute is selected for failover.

Copyright 2012 Symantec Corporation. All rights reserved.

The policies are described in more detail in the following pages.

54

312

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

FailOverPolicy=Priority
When FailOverPolicy is set to Priority, VCS selects the system with the lowest
assigned value from the SystemList attribute.
For example, the C service group has three systems configured in the SystemList
attribute and the same order for AutoStartList values:
SystemList = {s3=0, s1=1, s2=2}
AutoStartList = {s3, s1, s2}

Copyright 2012 Symantec Corporation. All rights reserved.

The C service group is initially started on s3 because it is the first system in


AutoStartList. If C faults on s3, VCS selects s1 as the failover target because it has
the lowest priority value for the remaining available systems.

55

Priority policy is the default behavior and is ideal for simple two-node clusters or
small clusters with few service groups.

Lesson 3 Startup and Failover Policies


Copyright 2012 Symantec Corporation. All rights reserved.

313

FailOverPolicy=RoundRobin
The RoundRobin policy selects the system running the fewest service groups as
the failover target.
The RoundRobin policy is ideal for large clusters running many service groups
with essentially the same server load characteristics (for example, similar
databases or applications).

Copyright 2012 Symantec Corporation. All rights reserved.

Take into account these properties of the RoundRobin policy:


Only systems listed in the SystemList attribute for the service group are
considered when VCS selects a failover target for all failover policies,
including RoundRobin.
A service group that is in the process of being brought online is not considered
an active service group until it is completely online.

56

Ties are determined by the order of systems in the SystemList attribute. For
example, if two failover target systems have the same number of service groups
running, the system listed first in the SystemList attribute is selected for failover.

314

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

FailOverPolicy=Load
When FailOverPolicy is set to Load, VCS determines the target system based on
the existing workload of each system listed in the SystemList attribute and the load
that is added by the service group.

Copyright 2012 Symantec Corporation. All rights reserved.

These attributes control load-based failover:


Capacity is a system attribute that contains a value representing the total
amount of load that the system can handle.
Load is a service group attribute that defines the amount of capacity required to
run the service group.
AvailableCapacity is a system attribute maintained by VCS that quantifies the
remaining available system load.

57

In the example displayed in the slide, the three servers that remain running after
the s3 system fails have Capacity set to 300, 200, and 100. Each service group has
a fixed load defined by the user, which is subtracted from the system capacity to
find the AvailableCapacity value of a system.
When failover occurs, VCS checks the value of AvailableCapacity on each
potential targeteach system in the SystemList attribute for the service group
and starts the service group on the system with the highest value.
Note: In the event that no system has a high enough AvailableCapacity value for a
service group load, the service group still fails over to the system with the highest
value for AvailableCapacity, even if the resulting AvailableCapacity value is zero
or a negative number.

Lesson 3 Startup and Failover Policies


Copyright 2012 Symantec Corporation. All rights reserved.

315

Configuring Load and Capacity


You can use VOM, the Cluster Manager Java GUI, or the command-line interface
to set the Capacity system attribute and the Load service group attribute.
To set Capacity from the command-line interface, use the hasys -modify
command as shown in this example:
hasys -modify s1 Capacity 300

To set Load from the CLI, use the hagrp -modify command as shown in this
example:

Copyright 2012 Symantec Corporation. All rights reserved.

hagrp -modify dbsg Load 75

58

316

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

The loadwarning trigger

Copyright 2012 Symantec Corporation. All rights reserved.

You can configure the loadwarning trigger to provide notification that a system
has sustained a predetermined load level for a specified period of time.

59

To configure the loadwarning trigger:


1 Create a loadwarning script in the /opt/VRTSvcs/bin/triggers
directory. You can copy the sample trigger script from /opt/VRTSvcs/
bin/sample_triggers as a starting point, and then modify it according
to your requirements.
2 Set the loadwarning attributes for the system:
Capacity: Load capacity for the system
LoadWarningLevel: The level at which load has reached a critical limit;
expressed as a percentage of the Capacity attribute
Default is 80 percent
LoadTimeThreshold: Length of time, in seconds, that a system must
remain at, or above, LoadWarningLevel before the trigger is run
Default is 600 seconds

Lesson 3 Startup and Failover Policies


Copyright 2012 Symantec Corporation. All rights reserved.

317

Limits and Prerequisites


Startup example
VCS enables you to define the available resources on each system and the
corresponding requirements for these resources for each service group. Shared
memory, semaphores, and the number of processors or application instances are all
examples of resources that can be defined on a system.
Note: The resources that you define are arbitrarythey do not need to
correspond to physical or software resources. You then define the
corresponding prerequisites for a service group to come online on a system.

Copyright 2012 Symantec Corporation. All rights reserved.

In a multinode, multiapplication services environment, VCS keeps track of the


available resources on a system by subtracting the resources already in use by
service groups online on each system from the maximum capacity for that
resource. When a new service group is brought online, VCS checks these available
resources against service group prerequisites; the service group cannot be brought
online on a system that does not have enough available resources to support the
application services.

60

Limits
The Limits system attribute is used to define the resources and the corresponding
capacity of each system for that resource. You can use any keyword for a resource
as long as you use the same keyword on all systems and service groups.

318

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

The example values displayed in the slide are set as follows:


On the first system, the Limits attribute setting in main.cf is:
Limits = { DBs=2 }
On the second two systems, the Limits attribute setting in main.cf is:
Limits = { DBs=1 }
On the fourth system, the Limits attribute setting in main.cf is:
Limits = { DBs=0 }
This value of DBs in this example is used to control how many Oracle service
groups can run on a system.
Service group Prerequisites
Prerequisites is a service group attribute that defines the set of resources needed to
run the service group. These values correspond to the Limits system attribute and
are set by the Prerequisites service group attribute. This main.cf configuration
corresponds to the E service group in the diagram:
Prerequisites = { DBs=1 }

CurrentLimits
CurrentLimits is an attribute maintained by VCS that contains the value of the
remaining available resources for a system. For example, if the limit for DBs is 2
and the A service group is online with a DBs prerequisite of 1, the CurrentLimits
setting for DBs is 1:
CurrentLimits = { DBs=1 }

Selecting a target system

Copyright 2012 Symantec Corporation. All rights reserved.

Prerequisites are used to determine a subset of eligible systems on which a service


group can be started during failover or startup. When a list of eligible systems is
created, HAD then follows the configured policy for autostart or failover.

61

Note: A value of 0 is assumed for systems that do not have some or all of the
resources defined in their Limits attribute. Similarly, a value of 0 is
assumed for service groups that do not have some or all of the resources
defined in their Prerequisites attribute.

Lesson 3 Startup and Failover Policies


Copyright 2012 Symantec Corporation. All rights reserved.

319

Failover example
The configuration in the slide shows how a failover target system is selected for
the C service group when the s3 system faults. Because C has a prerequisite of 1
for DBs and there are no systems with a high enough CurrentLimits value for DBs
to support running the group, C stays offline.
System Limits are hard values, meaning that if a system does not meet the
requirements specified in the Prerequisites attribute for a service group, the service
group cannot be started on that system.
Contrast this with the Load and Capacity, which are soft limits. If the
AvailableCapacity value is not high enough to satisfy the Load requirement for a
service group, VCS still fails over the service group, even if the AvailableCapacity
becomes a negative value.
Copyright 2012 Symantec Corporation. All rights reserved.

Therefore, using Limits and Prerequisites may be a better method for controlling
service group startup and failover in cases where hard limits must be enforced. An
example case is an environment where licensing restrictions prevent two instances
of an application or database from running on a system.

62

320

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Configuring Limits and Prerequisites


You can use the VCS GUI or command-line interface to set the Limits system
attribute and the Prerequisites service group attribute.
To set Limits from the command-line interface, use the hasys -modify
command as shown in the following example:
hasys -modify s2 Limits DBs 2

To set Prerequisites from the CLI, use the hagrp -modify command as shown
in this example:

Copyright 2012 Symantec Corporation. All rights reserved.

hagrp -modify DBSG Prerequisites DBs 1

63

Notes:
To be able to set these attributes, open the VCS configuration to enable read/
write mode and ensure that the service groups that are already online on a
system do not violate the restrictions.
The order that the resources are defined within the Limits or Prerequisites
attributes is not important.

Lesson 3 Startup and Failover Policies


Copyright 2012 Symantec Corporation. All rights reserved.

321

Modeling startup and failover policies


Using the Simulator

Copyright 2012 Symantec Corporation. All rights reserved.

The VCS Simulator for Windows is a good tool for modeling the behavior that you
require before making changes to the running configuration. This enables you to
fully understand the implications and the effects of different workload
management configurations.

64

322

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

65

Labs and solutions for this lesson are located on the following pages.
Lab 3: Failover policies, page A-25.
Lab 3: Failover policies, page B-65.
Lesson 3 Startup and Failover Policies
Copyright 2012 Symantec Corporation. All rights reserved.

323

Copyright 2012 Symantec Corporation. All rights reserved.

66

324

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Lesson 4

Alternate Network Configurations

67

Copyright 2012 Symantec Corporation. All rights reserved.

68

42

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Multiple service groups with NIC resources


NIC resources in multiple service groups
Many clusters have multiple service groups, each containing NIC resources that
monitor the same physical network interface. In this case, VCS monitors the same
network interfacesay e1000g0 on Solarismany times, creating unnecessary
overhead and network traffic.

Copyright 2012 Symantec Corporation. All rights reserved.

In addition to the overhead of many monitor cycles for the same resource, a
disadvantage of this configuration is the effect of changes in NIC hardware. If you
must change the network interface (for example, in the event the interface fails),
you must change the Device attribute for each NIC resource monitoring that
interface.

69

Lesson 4 Alternate Network Configurations


Copyright 2012 Symantec Corporation. All rights reserved.

43

Using Proxy resources


You can use a Proxy resource to allow multiple service groups to monitor the same
network interfaces. This reduces the network traffic that results from having
multiple NIC resources in different service groups monitor the same interface.
The Proxy resource mirrors the status of another resource in a different service
group. The required attribute, TargetResName, is the name of the resource whose
status is reflected by the Proxy resource.

Copyright 2012 Symantec Corporation. All rights reserved.

TargetSysName is an optional attribute specifies the name of the system on which


the target resource status is monitored. If no system is specified, the local system is
used as the target system.

70

44

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Parallel network service groups


Parallel service groups run on multiple systems simultaneously. A network service
group can be configured as a parallel service group because the NIC resource is
persistent and can be online on multiple systems. With this configuration, all the
other service groups are configured with a Proxy to the NIC resource on their local
system.

Copyright 2012 Symantec Corporation. All rights reserved.

A parallel service group is managed like any other service group in VCS, except
that it cannot be switched. The group is only started on a system listed in the
AutoStartList and the SystemList attributes. A parallel service group can also fail
over, in effect, if the service group faults on a system and there is an available
system (listed in the SystemList attribute) that is not already running the service
group. This is accomplished by VCS starting up the parallel group on the target
system.

71

To create a new parallel service group in a running cluster, set the Parallel attribute
to 1 (true) and then add resources.
You cannot change an existing failover service group that contains resources to a
parallel service group except by using the offline configuration procedure to edit
the main.cf file and add Parallel = 1 to the service group definition.

Lesson 4 Alternate Network Configurations


Copyright 2012 Symantec Corporation. All rights reserved.

45

Using Phantom resources

Copyright 2012 Symantec Corporation. All rights reserved.

The example network service group also requires a Phantom resource in order to
ensure that the service group status is displayed properly. A service group shows
an online status only when all of its nonpersistent resources are online. Therefore,
if a service group has only persistent resources, VCS considers the group offline,
even if the persistent resources are running properly. When a Phantom resource is
added, the status of the service group is shown as online.

72

46

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Localizing a NIC resource


An attribute whose value applies to all systems is global in scope. An attribute
whose value applies on a per-system basis is local in scope. By default, all
attributes are global. Some attributes can be localized to enable you to specify
different values for different systems.
In the example displayed in the slide, the Device attribute for the NIC resource is
localized to enable you to specify a different interface for each system.
After creating the resource, you can localize attribute values using the hares
command, a GUI, or an offline configuration method. For example, when using the
CLI, type:
hares -local netnic Device
Copyright 2012 Symantec Corporation. All rights reserved.

hares -modify netnic Device eth0 -sys s1

73

hares -modify netnic Device eth4 -sys s2

Any attribute can be localized. Network-related resources are common examples


for local attributes.

Lesson 4 Alternate Network Configurations


Copyright 2012 Symantec Corporation. All rights reserved.

47

Multiple public interfaces


Trunked or bonded interfaces
Most UNIX operating systems have the capability to create a logical network
interface that represents a collection of physical network interfaces. This is often
referred to as port trunking or interface bonding.

Copyright 2012 Symantec Corporation. All rights reserved.

For example, on Linux, you can use the bond driver to treat multiple network
interfaces as one logical interface. In this case, you can use the VCS NIC agent to
monitor a bond-type interface and an IP resource to bring up a virtual IP address
on the virtual bond interface.

74

48

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

The following table shows an example configuration where eth0 and eth1 are
controlled by the bond0 interface.
ifcfg-eth0

ifcfg-eth1

ifcfg-bond0

DEVICE=eth0
ONBOOT=yes
BOOTPROTO=no
ne
USERCTL=no
PEERDNS=no
TYPE=Ethernet
MASTER=bond0
SLAVE=yes

DEVICE=eth1
ONBOOT=yes
BOOTPROTO=no
ne
USERCTL=no
PEERDNS=no
TYPE=Ethernet
MASTER=bond0
SLAVE=yes

DEVICE=bond0
ONBOOT=yes
USERCTL=no
TYPE=Ethernet
MTU=""
NETMASK=255.255.0.0
BOOTPROTO=none
IPADDR=10.10.2.20
BROADCAST=10.10.255.255
GATEWAY=10.10.2.1
NETWORK=10.10.2.0

The corresponding sample /etc/modules.conf file contains:


alias eth0 e1000
alias eth1 e1000
alias bond0 bonding
options bond0 miimom=100 mode=2

The following VCS resource definitions are contained in a service group to


monitor the bond0 interface and manage the virtual IP address.
IP webip (
Device = bond0
Address = "10.10.10.21"
)

Copyright 2012 Symantec Corporation. All rights reserved.

NIC bond0nic (

75

Device = bond0
)

An advantage of using the Linux bond interface is that you can aggregate
interfaces to increase bandwidth.

Lesson 4 Alternate Network Configurations


Copyright 2012 Symantec Corporation. All rights reserved.

49

Local network interface failover


With the availability of inexpensive network adapters, it is common to have many
network interfaces on each system. By allocating more than one network interface
to a service group, you can potentially avoid failover of the entire service group if
the interface fails. By moving the IP address on the failed interface to another
interface on the local system, you can eliminate or minimize downtime.
VCS provides this type of local failover with the MultiNICB and IPMultiNICB
resources for the Solaris, AIX, and HP-UX platforms. All platforms also support
legacy resource types, MultiNICA and IPMultiNIC, for local interface failover.
Advantages of local interface failover

Copyright 2012 Symantec Corporation. All rights reserved.

Local interface failover can drastically reduce service interruptions to the clients.
Some applications have time-consuming shutdown and startup processes that
result in substantial downtime when the application fails over from one system to
another.

76

Failover between local interfaces can be completely transparent to users for some
applications.
Using multiple networks also makes it possible to eliminate any switch or hub
failures causing service group failover as long as the multiple interfaces on the
system are connected to separate hubs or switches.

410

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Network resources overview


The MultiNICA agent is capable of monitoring multiple network interfaces, and if
one of these interfaces faults, VCS fails over the IP address defined by the
IPMultiNIC resource to the next available public network adapter.

Copyright 2012 Symantec Corporation. All rights reserved.

The IPMultiNIC and MultiNICA resources provide essentially the same service as
the IP and NIC resources, but these resources monitor multiple interfaces instead
of a single interface. The dependency between these resources is the same as the
dependency between IP and NIC resources.

77

The MultiNICB and IPMultiNICB agents provide similar functionality to the


MultiNICA and IPMultiNIC agents with many additional features, such as:
Support for faster failover
Support for active/active interfaces
Support for failback
On Solaris only, these additional features are provided:
Support for the Solaris IPMP daemon
Support for trunked network interfaces on Solaris
See the Veritas Cluster Server Bundled Agents Reference Guide for your platform
for complete details about configuring these resource types.

Lesson 4 Alternate Network Configurations


Copyright 2012 Symantec Corporation. All rights reserved.

411

Comparing MultiNICA and MultiNICB


Advantages of using MultiNICA and IPMultiNIC
Physical interfaces can be plumbed as needed by the agent, supporting an
active/passive configuration.
MultiNICA requires only one base IP address for the set of interfaces under its
control. This address can also be used as the administrative IP address for the
system.
MultiNICA does not require all interfaces to be part of a single IP subnet.

Copyright 2012 Symantec Corporation. All rights reserved.

Advantages of using MultiNICB and IPMultiNICB


All interfaces under a particular MultiNICB resource are always configured
and have test IP addresses to speed failover.
MultiNICB failover is many times faster than that of MultiNICA.
Support for single and multiple interfaces eliminates the need for separate pairs
of NIC and IP, or MultiNICA and IPMultiNIC, for these interfaces.
MultiNICB and IPMultiNICB support failback of IP addresses.
MultiNICB and IPMultiNICB support manual movement of IP addresses
between working interfaces under the same MultiNICB resource without
changing the VCS configuration or disabling resources.

78

On Solaris, MultiNICB and IPMultiNICB support IPMP, interface groups, and


trunked ge and qfe interfaces.

412

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

79

Labs and solutions for this lesson are located on the following pages.
Lab 4: Creating a parallel network service group, page A-37.
Lab 4: Creating a parallel network service group, page B-89.

Lesson 4 Alternate Network Configurations


Copyright 2012 Symantec Corporation. All rights reserved.

413

Copyright 2012 Symantec Corporation. All rights reserved.

80

414

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Lesson 5

High Availability in the Enterprise

81

Copyright 2012 Symantec Corporation. All rights reserved.

82

52

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Operations Manager


Veritas Operations Manager provides a single, centralized management console for
the Veritas Storage Foundation and High Availability products. You can use it to
monitor, visualize, and manage storage and cluster resources, and generate reports.
Veritas Operations Manager enables administrators to centrally manage diverse
data center environments.
You can also use Veritas Operations Manager to manage hosts that do not have
Storage Foundation and High Availability products installed.
Simplified cluster management

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Operations Manager (VOM) enables you to perform administrative tasks


and analysis for all clusters in an environment, across physical locations, virtual
environments, and operating system platforms.

83

You can also use Veritas Operations Manager to visualize the state of multi-tier
applications and all subcomponents components and to start or stop the entire
logical application in an ordered fashion. Virtual Business Services provides this
capability of associating service groups across clusters and managing the
relationships in a high availability and disaster recovery environment.

Lesson 5 High Availability in the Enterprise


Copyright 2012 Symantec Corporation. All rights reserved.

53

VOM functional overview


VOM collects, stores, consolidates, and presents state information for all Storage
Foundation HA resources in a variety of views.
VOM uses a Web browser-based user interface to present information in a layered
form, starting with the concise dashboard view shown in the slide. The dashboard
view, presented upon login, contains highly summarized information about all
managed storage, server, and application resources at a glance.

Copyright 2012 Symantec Corporation. All rights reserved.

Graphics and color coding are both used to draw attention to resources that require
attention because they are either faulted (not functioning) or are at risk (configured
to be fault-tolerant, but having one or more failed components that could result in
faulting if a further component failure occurs).

84

54

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Distributed commands in a VOM domain


VOM provides centralized cluster visualization, monitoring, and control of the
entire distributed environment. The components of a VOM domainthe scope of
a Storage Foundation High Availability management environmentinclude:
Management serversstand-alone peer servers or clusters
One or more Web consoles
Managed hosts: VCS 4.x or 5.x nodes running a connector agent
VOM increases administrator efficiencies and reduces configuration errors by
enabling centralized deployment of configuration changes, such as by applying the
same change to many clusters with one command.

Copyright 2012 Symantec Corporation. All rights reserved.

In the example shown in the slide, a new user account, OraOper, is given service
group operator privileges for all clusters containing Oracle service groups.

85

Lesson 5 High Availability in the Enterprise


Copyright 2012 Symantec Corporation. All rights reserved.

55

License deployment policies and reporting


When using keyless licensing, all cluster nodes must become managed hosts in a
VOM environment. The licenses can then be tracked using the license deployment
reporting feature in VOM.
A 60 day grace period is allowed to set up the VOM environment and configure
SFHA systems as managed hosts. After 60 days, warning messages are issued
every four hours until the system is configured as a managed host.
Note: Although you can install VCS without Storage Foundation and select
keyless licensing, the warning mechanism is only disabled when you install
Storage Foundation and add the host to the VOM domain.
Copyright 2012 Symantec Corporation. All rights reserved.

You can use VOM license deployment policies and reports to ensure all systems in
the data center meet licensing requirements.

86

56

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

VOM solutions

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Operations Manager solutions are independent and optional feature packs
that you can download and use to enhance the functionality of Veritas Operations
Manager. These solutions are grouped into the following categories:
Add-on
Package
Patch
Hotfix

87

A core group of add-ons are bundled in VOM solutions repository. Other add-ons
must be downloaded from sort.symantec.com and then uploaded to your
VOM repository. Any solutions available in the repository can be installed on the
Management Server, managed hosts, or on both. To deploy solutions you must
have domain administrative privileges.
VOM Deployment Management enables you to install a solution on selected hosts,
or a selected platform (for example AIX). When you run the installation process, a
deployment request is sent. You can view that deployment request in the
Deployment Requests page.
After a solution is installed, you must enable the solution to use the management
tools within the add-on.

Lesson 5 High Availability in the Enterprise


Copyright 2012 Symantec Corporation. All rights reserved.

57

Disaster recovery enhancements


Testing disaster recovery
Disaster recovery fire drill is a feature provided with the Global Cluster Option for
VCS that enables you to anticipate and address configuration problems at the DR
site prior to encountering an actual disaster.
A fire drill is implemented by configuring a clone service group with specialpurpose resources. These fire drill service groups are modified to use copies of the
primary sites data so that the applications can be started and tested without
affecting the production service groups.

Copyright 2012 Symantec Corporation. All rights reserved.

When you bring a fire drill service group online, the resources:
1 Create a snapshot of the replicated volume.
2 Mount the snapshot file system.
3 Start the application or database.

88

VCS logs any errors to enable you to identify configuration problems in your DR
site, as shown in the following examples. Starting with VCS 5.0, EMC SRDF and
Hitachi Truecopy agents are enhanced to take advantage of device tagging in
Volume Manager. This enables hardware snapshots to be imported and used for a
fire drill capability without any scripting or tasks required to be used outside of
VCS.
Note: The Percentage in slide based on Symantec Disaster Recovery Study
published 22 November 2010.

58

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Successful fire drill example


The slide shows example VCS Engine log file entries for a successful fire drill test
of an Oracle service group named oragrp_fd. In this case, all resources came
online, and the database is running using data files on the snapshot volumes.

Copyright 2012 Symantec Corporation. All rights reserved.

You can configure the VCS Management Console to run regularly scheduled fire
drills to ensure you are continuously validating your DR environment.

89

Lesson 5 High Availability in the Enterprise


Copyright 2012 Symantec Corporation. All rights reserved.

59

DR configuration problem example


One common DR configuration problem is storage configuration driftan object
is changed at the primary site and that change is not propagated to the secondary
site.

Copyright 2012 Symantec Corporation. All rights reserved.

In this example, a database administrator adds a volume to a disk group on the


primary site, and then creates a new tablespace. This DBA is not aware that the
volume needs to be added to the replication configuration, so the tablespace is not
replicated to the secondary site.

90

510

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Failed fire drill example


This slide shows VCS engine log entries indicating the problem with the fire drill
caused by data created on the primary site but not replicated to the secondary site.
The log includes the Oracle startup output resulting from bringing the Oracle (Orafd3_fd) resource online on the secondary site. The Oracle error output identifies a
problem with the /oradata2/tb2.dbf file, which is not present at the
secondary site.

Copyright 2012 Symantec Corporation. All rights reserved.

Other examples of configuration drift that cause problems for site migration
include:
Expired licenses
Operating system and application patch version mismatches
Configuration files not synchronized

91

Lesson 5 High Availability in the Enterprise


Copyright 2012 Symantec Corporation. All rights reserved.

511

Managing replication
You can use VOM to simplify configuration of storage and replication objects used
in replicated data clusters (RDCs) and global clusters with both 4.x- and 5.x-based
systems.
To generate reports about the details of data transfer between source and target
replication hosts, download the Add-on for Veritas Volume Replicator Bandwidth
Reporting from sort.symantec.com.

Copyright 2012 Symantec Corporation. All rights reserved.

After you upload the add-on to the VOM repository, you can deploy the solution to
the management server and all hosts with replication solutions.

92

512

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Site-wide notification and migration


VOM simplifies management of disaster recovery, providing policy-based
monitoring of critical events, with automatic notification of administrators when
faults occur.
The Management Console also provides valuable diagnostic tools, such as:
Uptime analysis
Configuration analysis
Agent inventory
Additional predefined reports that can be scheduled and saved

Copyright 2012 Symantec Corporation. All rights reserved.

Perhaps most important in a DR environment, VOM enables you to perform an


entire site migration and recovery with a single action.

93

Lesson 5 High Availability in the Enterprise


Copyright 2012 Symantec Corporation. All rights reserved.

513

Disaster Recovery Advisor


Veritas Disaster Recovery Advisor (DRA) monitors high availability and disaster
recovery configurations to ensure data center recoverability. By scanning systems
across the data center to ensure that existing HA/DR plans are applied seamless,
Disaster Recovery Advisor helps to limit the risk of infrastructure and application
downtime.

Copyright 2012 Symantec Corporation. All rights reserved.

DRA scans storage, servers, databases, clusters, and replication infrastructures


using a knowledge base containing over 4,600 risk signatures. When gaps are
discovered, the software alerts the system administrator so the issues can be
resolved before business operations are impacted. As an agentless solution, the
implementation of Disaster Recovery Advisor is unobtrusive and non-disruptive,
operating in read-only mode. Monitoring and management is simple with status
dashboards that provide detailed insight into the data center's environment.

94

514

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Virtualization support

Copyright 2012 Symantec Corporation. All rights reserved.

Server virtualization approaches

95

Most customers are using more than one virtualization methodology within their
IT infrastructure. Each major UNIX operating system vendor has a virtualization
technology, each with unique terms such as control domain, host, guest, and virtual
I/O. The slide illustrates four virtualization approaches and examples of platforms
that utilize that type of virtualization.
Hardware partitioning subdivides a server, allocating resources to partitions,
each running an operating system. Sun Domains and HP nPartitions are
example solutions using this technology.
Bare metal hypervisor consists of a hypervisor process running directly on the
server hardware, which enables creating and maintaining virtual machines.
VMware ESX server is an example of this technology.
Hosted hypervisor has hosted operating system running on the server and a
hypervisor running within the OS. I/O flows from the virtual machines,
through the hypervisor, to the local operating system drivers, and then to the
server hardware. Linux KVM and Solaris LDOM are examples using this
technology.
Finally, operating system container technology allows for the partitioning of
the OS into containers to control access to the applications running within the
containers.

Lesson 5 High Availability in the Enterprise


Copyright 2012 Symantec Corporation. All rights reserved.

515

Symantec solutions for virtualization

Copyright 2012 Symantec Corporation. All rights reserved.

The slide shows how Storage Foundation and VCS works in each of the
virtualization environments.
Hardware partitioning
Storage Foundation and VCS run within partitions and generally partitions are
not migrated between physical machines.
Bare metal hypervisor
Storage Foundation and VCS run within the virtual machines. The VMs may
be migrated to different physical servers by virtualization HA technology.
Hosted hypervisor
Storage Foundation and VCS can run on both the host operating system and
virtual machines. VCS can be used to make VMs highly available, however
clusters cannot contain both VMs and the physical machine hosting VMs.
Operating system containers
Storage Foundation and VCS can run on the host operating system and can
control the startup of the security container as well as the applications within
them.

96

516

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Symantec across physical and virtual environments


Storage Foundation High Availability 6.0 can be deployed in a number of
configurations in both physical and virtual environments. As shown in the
diagram, different SFHA components can be deployed on hosts or on virtual
machines running supported guest operating systems.

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Operations Manager can be used to manage storage and clustering in both
physical and virtual environments.

97

Lesson 5 High Availability in the Enterprise


Copyright 2012 Symantec Corporation. All rights reserved.

517

Building resiliant private clouds


The Storage Foundation High Availability solutions from Symantec enable IT
organizations to build resilient private clouds by transforming their existing
infrastructure. Organizations can manage entire business services end-to-end with
built in resiliency, even if the business service is run across multiple virtualization
technologies, operating systems, and storage platforms. All products in the
portfolio are tightly integrated to help IT organizations move confidently to a
private cloud architecture.

Copyright 2012 Symantec Corporation. All rights reserved.

SFHA extends the core concepts of virtualization, by pooling information


resources even across heterogeneous infrastructure, and making them available ondemand for infrastructure HA or management solutions to take appropriate action
to maintain service continuity.

98

518

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Summary

Copyright 2012 Symantec Corporation. All rights reserved.

This lesson summarized some of the features of Veritas Cluster Server, Storage
Foundation, and Veritas Operations Manager that specifically enhance high
availability and disaster recovery solutions in enterprise data centers.

99

Lesson 5 High Availability in the Enterprise


Copyright 2012 Symantec Corporation. All rights reserved.

519

Copyright 2012 Symantec Corporation. All rights reserved.

100 520

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Appendix A

Labs

101

Copyright 2012 Symantec Corporation. All rights reserved.

102 A2

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Lab 1: Service group dependencies


In this lab, you will examine a variety of service group dependency configurations
and behaviors. Online and offline service group propagation is also examined.
This lab contains the following exercises:
Exercise 1: Checking lab prerequisites
A verification that the virtual machines needed for this lab are powered on and
functioning is performed.

Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 2: Service group and resource orientation


The service groups and resources in the east cluster are examined.

103

Exercise 3: Service group dependencies scenario 1


A typical service group dependency is configured so that one service group will
have multiple parents causing one service group to have preference being online
on startup and failover over another service group.
Exercise 4: Service group dependencies scenario 2
A typical service group dependency is configured with a three level dependency
causing one service group to have preference being online on startup and failover
over another service group.
Exercise 5: Service group dependencies scenario 3
A typical multi-child service group dependency is configured so that one service
group has two child service group dependencies causing a service group to have
preference being online on startup and failover.

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

A3

Exercise 1: Checking lab prerequisites


In this exercise, you verify that the virtual machines needed for this lab are
powered on and functioning, that you are logged in using the proper account and
that any needed terminal windows are opened.
1

Verify that the following virtual machines shown are powered on, that you are
logged using the indicated account and that the indicated terminal windows are
opened.
mgt
There is no need to log into this virtual machine at this time.
sym3
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym4
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.

Copyright 2012 Symantec Corporation. All rights reserved.

Note: The terminal windows are referred to as hostname:terminal#


throughout the labs. For example: sym1:terminal1, sym1:terminal2,
sym2:terminal1, and sym2:terminal2. It is not necessary to label the
terminal windows and you may decide which is terminal1 and which is
terminal2.

104 A4

If you have machines running that are not used in this lab, shut down the
operating system and power the machines off.

Note: If you are completing the lab exercises in order, this will require you to
shutdown cps, sym1 and sym2, and power on and log into sym3 and
sym4.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 2: Service group and resource orientation


In this exercise, you examine the service groups and resources in the east cluster.

sym3
Note: The appsg and testappsg service groups were added in an early Install and
Configure lab using the Veritas Cluster Manager Java Console. If those
lab exercises were skipped, then consult with your instructor on how to
copy in a replacement main.cf file located at:
/student/labs/vcs/vcs60/maincf/IC/
main.cf.east.lab05
1

From sym3:terminal1, perform a summary status on the cluster by:

Copyright 2012 Symantec Corporation. All rights reserved.

Listing the state of each service group.


Verifying that all service groups are online on the sym3 system and
switching any that need to be switched.
Verifying that there are no service group dependencies.

105

Display the state of all resources by listing the resources for each service group
and the dependencies between each resource in each service group.

Determine if all resources in all service groups other than the ClusterService
service group are set to critical. If any resources are set to non critical then
modify them. Open, save and close the cluster configuration as appropriate.

Confirm that the FaultPropagation and ManageFaults service group


attributes are set to 1 and ALL respectively for each service group. Modify to
their default value if necessary. Open, save and close the cluster configuration
as appropriate.

Open the cluster configuration for update.

From sym3:terminal2, run the hastatus command in order to observe


cluster changes as they happen in subsequent exercises.

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

A5

Exercise 3: Service group dependencies scenario 1


In this exercise, you configure a typical service group dependency so that one
service group will have multiple parents causing one service group to have
preference being online on startup and failover over another service group.

sym3
Note: The service groups in this exercise will come online and go offline very
quickly. The hagrp -wait commands, while not necessary, have been
included to be consistent with other lab exercises.
1

From sym3:terminal1, take all of the service groups except the


ClusterService service group offline on all cluster systems.

Create and confirm dependencies between the appsg and dbsg and between
the websg and dbsg service groups as indicated below. Save but do not close
the VCS configuration.

Copyright 2012 Symantec Corporation. All rights reserved.

appsg depends on dbsg (online local firm)


websg depends on dbsg (online local firm)

106 A6

Use a single hagrp command with the propagate option to bring the dbsg,
appsg, and websg service groups online on sym3. Can all three service groups
be brought online using just one command? If not, then ensure that all three
service groups are brought online on sym3.

Use a single hagrp command to take the dbsg, appsg, and websg service
groups offline on sym3. Can all three service groups be brought offline using
just one command? If not, then ensure that all three service groups are brought
offline on sym3.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Using a single hagrp command with the propagate option, bring the dbsg
and websg service groups online on sym3. Attempt to bring the appsg service
group online on sym4. Were you successful? If not successful, then bring the
appsg service group online on sym3.

Fault the appsg service group by removing the /var/tmp/appip file


followed by a probe of the appip resource on sym3. Do any service groups
failover?

Clear the appip resource fault and bring the appsg service group back online
on sym3.

Fault the dbsg service group by removing the /var/tmp/dbip file followed
by a probe of the dbip resource on sym3. Do any service groups failover?

Clear the dbip resource fault and take the dbsg, websg, appsg, and testappsg
service groups offline on any system they may be online.

10 Link the testappsg service group as a parent of the appsg service group with
Copyright 2012 Symantec Corporation. All rights reserved.

an offline local dependency. Save, but do not close the VCS configuration.

107

Note: This will configure a preference for the appsg service group running on
a system over the testappsg service group and force the testappsg to
switch upon a successful failover of the appsg service group.

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

A7

11 Bring the dgsg, appsg, and websg service groups online on sym3. Attempt to

bring the testappsg service group online on sym3. Were you successful? If
not, attempt to bring the testappsg service group online on sym4.

12 Fault the dbsg service group by removing the /var/tmp/dbip file followed

by a probe of the dbip resource on sym3. Do any service groups failover?

13 Take the testappsg service group offline on sym3 and clear the dbip resource

fault. Then, use a single command to take the dbsg, websg, appsg, and
testappsg service groups offline on any system where they may be online.
14 Unlink the service group dependencies between the testappsg and appsg

Copyright 2012 Symantec Corporation. All rights reserved.

service groups and between the dbsg, websg and appsg service groups. Save,
but do not close the VCS configuration.

108 A8

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 4: Service group dependencies scenario 2


In this exercise, you configure a typical service group dependency with a three
level dependency causing one service group to have preference being online on
startup and failover over another service group.

sym3
Note: The service groups in this exercise will come online and go offline very
quickly. The hagrp -wait commands, while not necessary, have been
included to be consistent with other lab exercises.

From sym3:terminal1, create dependencies between the appsg and dbsg


service groups, between the websg and appsg service groups and between the
testappsg and appsg service groups as indicated below. Save but do not close
the VCS configuration.
appsg depends on dbsg (online local firm)
websg depends on appsg (online local firm)
testappsg depends on appsg (offline local)
Bring the testappsg service group online on sym3. Use a single hagrp
command with the propagate option to bring the dbsg, appsg, and websg
service groups online on sym3. Were you successful? If not, then switch the
testappsg service group to sym4 and retry. Can all three service groups be
brought online using just one command? If not, then ensure that all three
service groups are brought online on sym3.

Copyright 2012 Symantec Corporation. All rights reserved.

109

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

A9

Copyright 2012 Symantec Corporation. All rights reserved.

110

A10

Using a single hagrp command with the propagate option, take the dbsg,
appsg, and websg service groups offline on sym3. Can all three service
groups be brought offline using just one command? If not, then ensure that all
three service groups are brought offline on sym3.

Using a single hagrp command with the propagate option, bring the dbsg,
appsg, and websg service groups online on sym3.

Fault the appsg service group by removing the /var/tmp/appip file


followed by a probe of the appip resource on sym3. Do any service groups
failover?

Clear the appip resource fault and use a single hagrp command with the
propagate option to bring the appsg service group back online on sym3.

Fault the dbsg service group by removing the /var/tmp/dbip file followed
by a probe of the dbip resource on sym3. Do any service groups failover?

Take the testappsg service group offline on sym3 and clear the dbip resource
fault. Then, use a single command to bring the dbsg, websg, appsg, and
testappsg service groups offline on any system they may be online.

Unlink the service group dependencies between the testappsg and appsg
service groups, between the dbsg and appsg service groups and between the
websg and appsg service groups. Save, but do not close the VCS
configuration.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 5: Service group dependencies scenario 3


In this exercise, you configure a typical multi-child service group dependency so
that one service group has two child service group dependencies causing a service
group to have preference being online on startup and failover.

sym3
Note: The service groups in this exercise will come online and go offline very
quickly. The hagrp -wait commands, while not necessary, have been
included to be consistent with other lab exercises.

From sym3:terminal1, create dependencies between the websg and appsg


service groups and websg and dbsg service groups as indicated below. Save but
do not close the VCS configuration.

Copyright 2012 Symantec Corporation. All rights reserved.

websg depends on appsg (online local firm)


websg depends on dbsg (online local firm)
testappsg depends on appsg (offline local)

111

Bring the testappsg service group online on sym4. Using a single hagrp
command with the propagate option, bring the dbsg, appsg, and websg
service groups online on sym3. Were you able to bring the websg, dbsg, and
appsg service groups online with the online propagate option?

Using a single hagrp command with the propagate option, take the websg,
dbsg, and appsg and service groups offline on sym3. Can all three service
groups be brought offline using just one command? If not, then ensure that all
three service groups are brought offline on sym3.

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

A11

Using a single hagrp command with the propagate option, bring the dbsg,
appsg, and websg service groups online on sym3.

Fault the websg service group by removing the /var/tmp/webvip file


followed by a probe of the webvip resource on sym3. Do any service groups
failover?

Clear the webvip resource fault and bring the websg service group back online
on sym3.

Fault the dbsg service group by removing the /var/tmp/dbip file followed
by a probe of the dbip resource on sym3. Do any service groups failover?

Take the appsg service group offline on sym3. Then, bring the websg, appsg,
and dbsg service groups online on sym4. Account for the testappsg service
group.

Clear the dbip resource fault.

10 Unlink the service group dependencies between the testappsg and appsg

Copyright 2012 Symantec Corporation. All rights reserved.

service groups, between the dbsg and appsg service groups, and between the
websg and appsg service groups. Save and close the VCS configuration.

112

11 From sym3:terminal2, terminate the hastatus command.

End of lab

A12

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Lab 2: Merging clusters


In this lab, you will merge the east cluster to the west cluster forming one
four-node west cluster.
This lab contains the following exercises:
Exercise 1: Checking lab prerequisites
A verification that the virtual machines needed for this lab are powered on and
functioning is performed.
Exercise 2: Merge cluster preparation
Two clusters are prepared for merging.

Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 3: Merging cluster infrastructure


The systems from one cluster are added to another cluster.

113

Exercise 4: Adding service groups and resources


The service groups and resources from one of the original are added to the merged
cluster.
Exercise 5: Merged cluster verification
The merged cluster is verified.

Lab 2: Merging clusters

A13
Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 1: Checking lab prerequisites


In this exercise, you verify that the virtual machines needed for this lab are
powered on and functioning, that you are logged in using the proper account and
that any needed terminal windows are opened.
1

mgt
There is no need to log into this virtual machine at this time.
sym1
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym2
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym3
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym4
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.

Copyright 2012 Symantec Corporation. All rights reserved.

114

Verify that the following virtual machines shown are powered on, that you are
logged using the indicated account and that the indicated terminal windows are
opened.

Note: The terminal windows are referred to as hostname:terminal#


throughout the labs. For example: sym1:terminal1, sym1:terminal2,
sym2:terminal1, and sym2:terminal2. It is not necessary to label the
terminal windows and you may decide which is terminal1 and which is
terminal2.

A14

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

If you have machines running that are not used in this lab, shut down the
operating system and power the machines off.

Copyright 2012 Symantec Corporation. All rights reserved.

Note: If you are completing the lab exercises in order, this will require you to
power on and log into sym1 and sym2.

115

Lab 2: Merging clusters

A15
Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 2: Merge cluster preparation


In this exercise, you prepare two clusters for merging.

sym1
Note: In this lab there are no iSCSI LUNs that are visible to all four systems, so
merging enabled I/O fencing using disks cannot be part of the cluster
merge lab. Therefore, for lab purposes, I/O fencing is re-configured to
disabled mode on both clusters.

From sym1:terminal1, navigate to the /opt/VRTS/install directory and


run the installsfha script with the fencing option using the following
information.
Reconfigure fencing in disabled mode.
Stop VCS and apply the fencing changes.
Do not view the summary file.

Use the gabconfig and service vxfen status commands to confirm


that I/O fencing is configured. Use the vxfenadm command to confirm that
I/O fencing is running in disabled mode.

sym3

Copyright 2012 Symantec Corporation. All rights reserved.

116

From sym3:terminal1, navigate to the /opt/VRTS/install directory and


run the installsfha script with the fencing option using the following
information.
Reconfigure fencing in disabled mode.
Stop VCS and apply the fencing changes.
Do not view the summary file.

A16

Use the gabconfig and service vxfen status commands to confirm


that I/O fencing is configured. Use the vxfenadm command to confirm that
I/O fencing is running in disabled mode.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Note: In this lab, there is a conflicting duplicate named appsg service group in
both the west and east clusters. This would cause errors during a merge.
The appsg service group in the west cluster would have precedence so the
appsg service group in the east cluster must be renamed or, as performed
in this lab, deleted from the configuration.
5

Take the appsg service group offline and list its resources and dependencies.

Open the VCS configuration for update and delete each resource in the appsg
service group in resource dependency order. Then, delete the appsg service
group and save and close the VCS configuration.

Navigate to the /etc/VRTSvcs/conf/config directory and use the


hacf -cftocmd command to create a main.cmd file. Copy the
main.cmd file to eastmain.cmd.

Edit the eastmain.cmd file to remove all commands other than those
related to the websg, testappsg, dbsg service groups, their resources, and their
resource dependencies.

Copyright 2012 Symantec Corporation. All rights reserved.

Note: Alternatively, copy the


/student/labs/vcs/vcs60/maincf/CM/eastmain.cmd
file to /etc/VRTSvcs/conf/config/eastmain.cmd.

117

After you have made the edits the contents of the file should be:
hagrp -add dbsg
hagrp -modify dbsg SystemList sym3 0 sym4 1
hagrp -modify dbsg AutoStartList sym3 sym4
hagrp -modify dbsg SourceFile "./main.cf"
hares -add dbdg FileOnOff dbsg
hares -modify dbdg PathName "/var/tmp/dbdg"
hares -modify dbdg Enabled 1
hares -add dbip FileOnOff dbsg
hares -modify dbip PathName "/var/tmp/dbip"
hares -modify dbip Enabled 1
hares -add dblistener FileOnOff dbsg
hares -modify dblistener PathName "/var/tmp/
dblistener"
hares -modify dblistener Enabled 1
hares -add dbmnt FileOnOff dbsg
hares -modify dbmnt PathName "/var/tmp/dbmnt"

Lab 2: Merging clusters

A17
Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

hares -modify dbmnt Enabled 1


hares -add dbnic FileOnOff dbsg
hares -modify dbnic PathName "/var/tmp/dbnic"
hares -modify dbnic Enabled 1
hares -add dboracle FileOnOff dbsg
hares -modify dboracle PathName "/var/tmp/
dboracle"
hares -modify dboracle Enabled 1
hares -add dbvol FileOnOff dbsg
hares -modify dbvol PathName "/var/tmp/dbvol"
hares -modify dbvol Enabled 1
hagrp -add testappsg
hagrp -modify testappsg SystemList sym3 0 sym4 1
hagrp -modify testappsg AutoStartList sym3 sym4
hagrp -modify testappsg Operators oper
hagrp -modify testappsg SourceFile "./main.cf"
hares -add testappdg FileOnOff testappsg
hares -modify testappdg PathName "/var/tmp/
testappdg"
hares -modify testappdg Enabled 1
hares -add testappmnt FileOnOff testappsg
hares -modify testappmnt PathName "/var/tmp/
testappmnt"
hares -modify testappmnt Enabled 1
hares -add testappnic FileOnOff testappsg
hares -modify testappnic PathName "/var/tmp/
testappnic"
hares -modify testappnic Enabled 1
hares -add testappproc FileOnOff testappsg
hares -modify testappproc PathName "/var/tmp/
testappproc"
hares -modify testappproc Enabled 1
hares -add testappip FileOnOff testappsg
hares -modify testappip PathName "/var/tmp/
testappip"
hares -modify testappip Enabled 1
hares -add testappvol FileOnOff testappsg
hares -modify testappvol PathName "/var/tmp/
testappvol"
hares -modify testappvol Enabled 1

118

A18

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

119

hagrp -add websg


hagrp -modify websg SystemList sym3 0 sym4 1
hagrp -modify websg AutoStartList sym3 sym4
hagrp -modify websg Operators oper
hagrp -modify websg SourceFile "./main.cf"
hares -add webapache FileOnOff websg
hares -modify webapache PathName "/var/tmp/
webapache"
hares -modify webapache Enabled 1
hares -add webdg FileOnOff websg
hares -modify webdg PathName "/var/tmp/webdg"
hares -modify webdg Enabled 1
hares -add webmnt FileOnOff websg
hares -modify webmnt PathName "/var/tmp/webmnt"
hares -modify webmnt Enabled 1
hares -add webnic FileOnOff websg
hares -modify webnic PathName "/var/tmp/webnic"
hares -modify webnic Enabled 1
hares -add webvip FileOnOff websg
hares -modify webvip PathName "/var/tmp/webvip"
hares -modify webvip Enabled 1
hares -add webvol FileOnOff websg
hares -modify webvol PathName "/var/tmp/webvol"
hares -modify webvol Enabled 1
hares -link dbip dbnic
hares -link dblistener dbip
hares -link dblistener dboracle
hares -link dbmnt dbvol
hares -link dboracle dbmnt
hares -link dbvol dbdg
hares -link testappmnt testappvol
hares -link testappproc testappmnt
hares -link testappproc testappip
hares -link testappip testappnic
hares -link testappvol testappdg
hares -link webapache webmnt
hares -link webapache webvip
hares -link webmnt webvol
hares -link webvip webnic
hares -link webvol webdg
Lab 2: Merging clusters

A19
Copyright 2012 Symantec Corporation. All rights reserved.

Shutdown VCS, IO fencing, GAB, and LLT on both nodes of the east cluster.

10 Rename the /etc/VRTSvcs/conf/config/main.cf file to

main.cf.east on sym3 and sym4.

Copyright 2012 Symantec Corporation. All rights reserved.

Note: In order for the merge utility to work correctly, the main.cf files on
the east cluster nodes must not exist.

120 A20

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 3: Merging cluster infrastructure


In this exercise, you add the systems from one cluster to another cluster.

sym1
1

From sym1:terminal1, navigate to the /opt/VRTS/install directory and


run the ./installsfha -addnode command taking note of the merge
pre-requisites and using the following information.

Enter sym1 as the name of one node of the west cluster.


Enter sym3 and sym4 as the systems to be added to the west cluster.
Use eth4 and eth5 as the private links and eth0 as the low priority link.
Use eth0 as the public NIC used for the cluster virtual IP.
Record the log file located in the /opt/VRTS/install/logs
directory. Optionally, review the log file.
Do not view the summary file.

Copyright 2012 Symantec Corporation. All rights reserved.

Log file:

121

Perform a summary status of the cluster noting that none of the former east
cluster service groups have been added.

Use the gabconfig command to check GAB status.

Use the vxfenadm command to check I/O fencing.

Use the lltstat command to check the LLT configured and active status.
Optionally, performed this step on all cluster nodes.

Display the contents of the /etc/llttab, /etc/llthosts, and


/etc/gabtab files. Optionally, perform this step on all cluster nodes.

Lab 2: Merging clusters

A21
Copyright 2012 Symantec Corporation. All rights reserved.

Display the /etc/VRTSvcs/conf/config/main.cf file noticing the


following elements.
\

The cluster name is west.


The cluster virtual ip is 10.10.2.51.
All four systems are defined as cluster members.
The SystemList and AutoStartList service group attributes for the
ClusterService service group configure all four cluster systems.
The appsg, nfssg, and orasg service groups from the original west cluster
are configured and remain restricted to sym1 and sym2.
The websg, dbsg, and testappsg from the east cluster are not configured.

Copyright 2012 Symantec Corporation. All rights reserved.

122 A22

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 4: Adding service groups and resources


In this exercise, you add the service groups and resources from one of the original
clusters to the merged cluster.

sym3
1

From sym3:terminal1, navigate to the /etc/VRTSvcs/conf/config


directory and open the VCS configuration for update

Run the sh -x ./eastmain.cmd command and observe the output taking


notice of the VCS warning when failing to add the VCS user named oper as an
Operator for the testappsg and websg service groups.
Note: The oper user add is not part of the eastmain.cmd file and would
need to be re-added and re-configured later on.
Save and close the VCS configuration.

Copyright 2012 Symantec Corporation. All rights reserved.

123

Lab 2: Merging clusters

A23
Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 5: Merged cluster verification


In this exercise, you verify the merged cluster.

sym3
1

From sym3:terminal1, perform a summary status on the cluster taking note of


the service groups that are not online.

Display /etc/VRTSvcs/conf/config/main.cf file and notice the


following.
The oper user from the old east cluster is not defined nor is the Operator
service group attribute for the testappsg and websg service groups set to
the oper user.
The SystemList and AutoStartList service group attributes for the
ClusterService service group have been modified for all four cluster
systems.
The SystemList and AutoStartList service group attributes for the appsg,
nfssg, and orasg service groups remain set to sym1 and sym2.
Note: These service groups cannot be online on sym3 and sym4 due to
shared LUN restrictions.
The SystemList and AutoStartList service group attributes for the websg,
testappsg, and dbsg service groups remain set to sym3 and sym4.

Copyright 2012 Symantec Corporation. All rights reserved.

Note: These service groups could be online on sym1 and sym2 and could
be modified to accommodate that.

Note: Consideration needs to be given to activated triggers configured at any


service group and/or resource level if the service group is going to be
re-configured to run on a new cluster system.

End of lab

124 A24

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Lab 3: Failover policies


In this lab, you use the Veritas Cluster Server Simulator with a preconfigured
main.cf file to show how to configure and test failover policies using the Veritas
Cluster Manager Java Console.
This lab contains the following exercises:
Exercise 1: Checking lab prerequisites
A verification that the virtual machines needed for this lab are powered on and
functioning is performed.

Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 2: Starting the simulator


The simulation used in the lab is examined and the Veritas VCS Simulator is
started.

125

Exercise 3: Testing priority failover policy


The default priority failover policy is examined.
Exercise 4: Testing load failover policy
The load failover policy is examined.
Exercise 5: Testing prerequisites and limits
Prerequisites and limits are examined and added to the load failover policy
settings.
Exercise 6: Stopping the simulator
The simulation used in the lab is examined and the Veritas VCS Simulator is
terminated.

Lab 3: Failover policies

A25
Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 1: Checking lab prerequisites


In this exercise, you verify that the virtual machines needed for this lab are
powered on and functioning, that you are logged in using the proper account and
that any needed terminal windows are opened.
1

Verify that the following virtual machines shown are powered on, that you are
logged using the indicated account and that the indicated terminal windows are
opened.
mgt
There is no need to log into this virtual machine at this time.
winclient
Log in with credentials:
Account: Administrator
Password: veritas
Note: The terminal windows are referred to as hostname:terminal#
throughout the labs. For example: sym1:terminal1, sym1:terminal2,
sym2:terminal1, and sym2:terminal2. It is not necessary to label the
terminal windows and you may decide which is terminal1 and which is
terminal2.

Copyright 2012 Symantec Corporation. All rights reserved.

126 A26

If you have machines running that are not used in this lab, shut down the
operating system and power the machines off.

Note: If you are completing the lab exercises in order, this will require you to
shutdown sym1, sym2, sym3 and sym4, and power on and log into
winclient.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 2: Starting the simulator


In this exercise, you examine the simulation used in the lab and start the Veritas
VCS Simulator.

winclient
1

From the winclient desktop, use Windows Explorer to navigate to the


C:\Progam Files (x86)\VERITAS\VCS Simulator\wlm\
conf\config folder.

Open the main.cf file using the WordPad application.

Without making any changes, review the contents of the main.cf file. Then,
close the Wordpad and Windows Explorer windows.

Start the Veritas VCS Simulator.

Copyright 2012 Symantec Corporation. All rights reserved.

Note: The VCS Simulator may take a moment to display as it discovers any
running simulations even thought there will not be any.

127

Verify the wlm simulation.

Start the wlm simulation.


Note: The status of the cluster name will have a green check mark upon
successful startup.

Launch the VCS Java Console and log in using the following credentials.
User name: admin
Password: password

Lab 3: Failover policies

A27
Copyright 2012 Symantec Corporation. All rights reserved.

From the Java Console, verify that the Member Systems and their Priority
and their AutoStartList attribute are set to the values shown in the following
table.
Service Group

Member Systems (Priority)

AutoStartList

A1

S1 (1), S2 (2), S3 (3), S4 (4)

S1

A2

S1 (1), S2 (2), S3 (3), S4 (4)

S1

B1

S1 (4), S2 (1), S3 (2), S4 (3)

S2

B2

S1 (4), S2 (1), S3 (2), S4 (3)

S2

C1

S1 (3), S2 (4), S3 (1), S4 (2)

S3

C2

S1 (3), S2 (4), S3 (1), S4 (2)

S3

D1

S1 (2), S2 (3), S3 (4), S4 (1)

S4

D2

S1 (2), S2 (3), S3 (4), S4 (1)

S4

Copyright 2012 Symantec Corporation. All rights reserved.

128 A28

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 3: Testing priority failover policy


In this exercise, you examine the default priority failover policy.

winclient
1

From the Java Console, verify that the failover policy of all service groups is
Priority.

Verify that all service groups are online as shown in the following table.

Service Group

System S1

A1

Online

A2

Online

System S2

B1

Online

B2

Online

System S3

C1

Online

C2

Online

System S4

D1

Online

D2

Online

If it faults, where will the A1 service group fail over to? Verify this failover by
faulting a critical resource in the A1 service group.
Service Group A1 will fail over to:

Copyright 2012 Symantec Corporation. All rights reserved.

129

If the A1 service group faults again without clearing the previous fault, where
should it fail over? Verify the failover by faulting a critical resource in the A1
service group.
Service Group A1 will fail over to:

Clear the faults in the A1 service group. Where will the service group fail over
to now? Verify the failover by faulting a critical resource in the A1 service
group.
Service Group A1 will fail over to:

Clear the faults in the A1 service group.

Lab 3: Failover policies

A29
Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 4: Testing load failover policy


In this exercise, you examine the load failover policy.

winclient
1

From the Java Console, open the cluster configuration for update.

Set the FailOverPolicy attribute to Load for the eight service groups.

Set the Load attribute for each service group as shown in the following table.

Copyright 2012 Symantec Corporation. All rights reserved.

130 A30

Service Group

Load

A1

75

A2

75

B1

75

B2

75

C1

50

C2

50

D1

50

D2

50

Set the S1 and S2 systems Capacity attributes to 200.


Note: The S3 and S4 systems Capacity attribute will not be changed and
defaults to 100.

Verify that the systems AvailableCapacity attribute is as shown in the


following table.
Service Group

AvailableCapacity

S1

50

S2

50

S3

S4

0
Veritas Cluster Server 5.1 for UNIX: Install and Configure

Copyright 2012 Symantec Corporation. All rights reserved.

Save the configuration changes.

Verify that all service groups are online as shown in the following table.

Service Group

System S1

A1

Online

A2

Online

System S2

B1

Online

B2

Online

System S3

C1

Online

C2

Online

System S4

D1

Online

D2

Online

If it faults, where will the A1 service group fail over to? Verify this failover by
faulting a critical resource in the A1 service group.
Service Group A1 will fail over to:

Verify that the systems AvailableCapacity attribute is as shown in the


following table.
Service Group

AvailableCapacity

S1

125

S2

-25

S3

S4

Copyright 2012 Symantec Corporation. All rights reserved.

10 If the S2 system fails, where will the A1, B1 and B2 service groups that are

131

Online on S2 fail over? Verify this failover by powering off the S2 system in
Cluster Manager.
The A1 service group will fail over to:
The B1 service group will fail over to:
The B2 service group will fail over to:

Lab 3: Failover policies

A31
Copyright 2012 Symantec Corporation. All rights reserved.

11 Verify that the systems AvailableCapacity attribute is as shown in the

following table.
Service Group

AvailableCapacity

S1

-25

S2

200

S3

-75

S4

12 Power up the S2 system, clear all faults, and return the service groups to their

startup locations.
13 Verify that the systems AvailableCapacity attribute is as shown in the

following table.
Service Group

AvailableCapacity

S1

50

S2

50

S3

S4

14 Save and close the configuration changes.

Copyright 2012 Symantec Corporation. All rights reserved.

Note: Saving changes to the configuration is not necessary for purposes of


this lab. However, if you want to view the modifications to the
main.cf file then save the changes.

132 A32

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 5: Testing prerequisites and limits


In this exercise, you examine the prerequisites and limits and add them to the load
failover policy settings.

winclient
1

From the Java Console, open the cluster configuration for update.

Set the Limits attribute for each system to the key ABGroup with a value of 3.

Set the Prerequisites attribute for the A1, A2, B1 and B2 service groups to be
ABGroup with a value of 1.
CAUTION Do not modify the Prerequisites attribute for the C1, C2, D1,
and D2 service groups.

Save the configuration changes.

If the S1 system fails, where will the A1 and A2 service groups that are Online
on S1 fail over? Verify this failover by powering off the S1 system in Cluster
Manager. Explain the failover actions that occurred.
The A1 service group will fail over to:

Copyright 2012 Symantec Corporation. All rights reserved.

The A2 service group will fail over to:

133

Lab 3: Failover policies

A33
Copyright 2012 Symantec Corporation. All rights reserved.

If the S2 system fails, where will the A1, B1 and B2 service groups that are
Online on S2 fail over? Verify this failover by powering off the S2 system in
Cluster Manager. Explain the failover actions that occurred.
The A1 service group will fail over to:
The B1 service group will fail over to:
The B2 service group will fail over to:

If the S3 system fails, where will the A2, B1, C1 and C2 service groups that
are Online on S3 fail over? Verify this failover by powering off the S3 system
in Cluster Manager. Explain the failover actions that occurred.
The A2 service group will fail over to:
The B1 service group will fail over to:
The C1 service group will fail over to:
The C2 service group will fail over to:

Copyright 2012 Symantec Corporation. All rights reserved.

134 A34

S4

Save and close the configuration changes.


Note: Saving changes to the configuration is not necessary for purposes of
this lab. However, if you want to view the modifications to the
main.cf file then save the changes.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 6: Stopping the simulator


In this exercise, you terminate the simulation and the Veritas VCS Simulator.

winclient
1

Log out from Cluster Explorer.

From the Simulator Java Console, stop the wlm cluster.

Verify the configuration.


Note: If you intend to move this configuration in to a running cluster, you
must ensure the main.cf syntax is correct. You can verify the
configuration only in the Simulator GUI after the cluster is stopped.

Close the Simulator Java Console.


Note: The Veritas Cluster Manager Java Console should close as well. If it
does not then close it manually.

Copyright 2012 Symantec Corporation. All rights reserved.

End of lab

135

Lab 3: Failover policies

A35
Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

136 A36

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Lab 4: Creating a parallel network service group


In this lab application networking is reconfigured to be more efficient.
This lab contains the following exercises:
Exercise 1: Checking lab prerequisites
A verification that the virtual machines needed for this lab are powered on and
functioning is performed.
Exercise 2: Configuring a common parallel network service group
A common parallel network service group and resources is configured.

Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 3: Replacing NIC resources with Proxy resources


The pre-existing resources of type NIC supporting application VIPs in each
application service group are replaced with resources of type Proxy.

137

Lab 4: Creating a parallel network service group


Copyright 2012 Symantec Corporation. All rights reserved.

A37

Exercise 1: Checking lab prerequisites


In this exercise, you verify that the virtual machines needed for this lab are
powered on and functioning, that you are logged in using the proper account and
that any needed terminal windows are opened.

Copyright 2012 Symantec Corporation. All rights reserved.

138 A38

Verify that the following virtual machines shown are powered on, that you are
logged using the indicated account and that the indicated terminal windows are
opened.
mgt
There is no need to log into this virtual machine at this time.
sym1
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym2
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym3
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym4
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
Note: The terminal windows are referred to as hostname:terminal#
throughout the labs. For example: sym1:terminal1, sym1:terminal2,
sym2:terminal1, and sym2:terminal2. It is not necessary to label the
terminal windows and you may decide which is terminal1 and which is
terminal2.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

If you have machines running that are not used in this lab, shut down the
operating system and power the machines off.

Copyright 2012 Symantec Corporation. All rights reserved.

Note: If you are completing the lab exercises in order, this will require you to
shutdown winclient, and power on and log into sym1, sym2, sym3
and sym4.

139

Lab 4: Creating a parallel network service group


Copyright 2012 Symantec Corporation. All rights reserved.

A39

Exercise 2: Configuring a common parallel network service group


In this exercise, you configure a common parallel network service group and
resources.

Copyright 2012 Symantec Corporation. All rights reserved.

sym1

140 A40

From sym1:terminal1, use the haconf command to open the cluster


configuration for update.

Use the hagrp command to create a service group named netsg.

Modify the SystemList to allow the netsg service group to run on the sym1,
sym2, sym3, and sym4 in that order.

Modify the AutoStartList attribute to allow the service group to start on all
systems.

Modify the Parallel service group to configure the netsg service group as a
parallel service group.

Display the service group attributes for the netsg service group to confirm your
input and key default service group attribute values.

Save the VCS configuration, but do not close it.

Add a resource of type NIC named netnic to the netsg service group.

Modify the Device and Critical resource attributes for the netnic resource
using the following information.
Device is eth0
Critical set to 0 (zero)

10 Display the resource attribute values for the netnic resource to confirm your

input and key default resource attribute values.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

11 Enable the netnic resource and verify that it is enabled. Display the state of the

netnic resource to ensure that it is online on both cluster systems.


Note: It is not necessary to bring a resource of type NIC online.
12 Display the state of the netsg service group.

Note: The service group state is offline even though the NIC resource state is
online. This is because VCS does not consider persistent resources
when determining the state of a service group.
13 Save the VCS configuration, but do not close it.
14 Add a resource of type Phantom named netphantom to the netsg service

group.
15 Set the Critical resource attribute to 0 (zero) for the netphantom resource.
16 Display the resource attribute values for the netphantom resource to confirm

your input and key default resource attribute values.


17 Enable the netphantom resource and verify that it is enabled. Display the state

of the netphantom resource to ensure that it is online on both cluster systems.


Note: It is not necessary to bring a resource of type Phantom online.

Copyright 2012 Symantec Corporation. All rights reserved.

18 Display the state of the netsg service group.

141

Note: The service group state is offline even though the NIC resource state is
online. This is because VCS does not consider persistent resources
when determining the state of a service group.

19 Save the VCS configuration, but do not close it.

Lab 4: Creating a parallel network service group


Copyright 2012 Symantec Corporation. All rights reserved.

A41

Exercise 3: Replacing NIC resources with Proxy resources


In this exercise, you replace the pre-existing resources of type NIC that supporting
application VIPs in each application service group with resources of type Proxy.

sym1
1

From sym1:terminal1, identify the names of the resources of type NIC that are
configured.

Ignoring the resource of type NIC named netnic, list the resource
dependencies associated with the listed resources of type NIC.

Replace the resource of type NIC named appnic with a resource of type Proxy
named appproxy using the following information.

Replace the resource of type NIC named csgnic with a resource of type Proxy
named csgproxy using the following information.

Copyright 2012 Symantec Corporation. All rights reserved.

142 A42

Set the Critical attribute to 0 (zero).


Set the TargetResName attribute to netnic.
Set Enabled to 1 (one).
Link to the appip resource.

Set the Critical attribute to 0 (zero).


Set the TargetResName attribute to netnic.
Set Enabled to 1 (one).
Link to the notifier and webip resources.

Replace the resource of type NIC named nfsnic with a resource of type Proxy
named nfsproxy using the following information.

Set the Critical attribute to 0 (zero).


Set the TargetResName attribute to netnic.
Set Enabled to 1 (one).
Link to the nfsip resource.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Replace the resource of type NIC named oranic with a resource of type Proxy
named oraproxy using the following information.

Set the Critical attribute to 0 (zero).


Set the TargetResName attribute to netnic.
Set Enabled to 1 (one).
Link to the oraip resource.

Save and close the VCS configuration.

Copyright 2012 Symantec Corporation. All rights reserved.

End of lab

143 Lab 4: Creating a parallel network service group

Copyright 2012 Symantec Corporation. All rights reserved.

A43

Copyright 2012 Symantec Corporation. All rights reserved.

144 A44

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Appendix B

Lab Solutions

145

Copyright 2012 Symantec Corporation. All rights reserved.

146 B2

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Lab 1: Service group dependencies


In this lab, you will examine a variety of service group dependency configurations
and behaviors. Online and offline service group propagation is also examined.
This lab contains the following exercises:
Exercise 1: Checking lab prerequisites
A verification that the virtual machines needed for this lab are powered on and
functioning is performed.

Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 2: Service group and resource orientation


The service groups and resources in the east cluster are examined.
Exercise 3: Service group dependencies scenario 1
A typical service group dependency is configured so that one service group will
have multiple parents causing one service group to have preference being online
on startup and failover over another service group.
Exercise 4: Service group dependencies scenario 2
A typical service group dependency is configured with a three level dependency
causing one service group to have preference being online on startup and failover
over another service group.
Exercise 5: Service group dependencies scenario 3
A typical multi-child service group dependency is configured so that one service
group has two child service group dependencies causing a service group to have
preference being online on startup and failover.

147 Lab 1: Service group dependencies

Copyright 2012 Symantec Corporation. All rights reserved.

B3

Exercise 1: Checking lab prerequisites


In this exercise, you verify that the virtual machines needed for this lab are
powered on and functioning, that you are logged in using the proper account and
that any needed terminal windows are opened.
1

Verify that the following virtual machines shown are powered on, that you are
logged using the indicated account and that the indicated terminal windows are
opened.
mgt
There is no need to log into this virtual machine at this time.
sym3
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym4
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.

Copyright 2012 Symantec Corporation. All rights reserved.

Note: The terminal windows are referred to as hostname:terminal#


throughout the labs. For example: sym1:terminal1, sym1:terminal2,
sym2:terminal1, and sym2:terminal2. It is not necessary to label the
terminal windows and you may decide which is terminal1 and which is
terminal2.

148 B4

If you have machines running that are not used in this lab, shut down the
operating system and power the machines off.

Note: If you are completing the lab exercises in order, this will require you to
shutdown cps, sym1 and sym2, and power on and log into sym3 and
sym4.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 2: Service group and resource orientation


In this exercise, you examine the service groups and resources in the east cluster.

sym3
Note: The appsg and testappsg service groups were added in an early Install and
Configure lab using the Veritas Cluster Manager Java Console. If those
lab exercises were skipped, then consult with your instructor on how to
copy in a replacement main.cf file located at:
/student/labs/vcs/vcs60/maincf/IC/
main.cf.east.lab05
1

From sym3:terminal1, perform a summary status on the cluster by:


Listing the state of each service group.
Verifying that all service groups are online on the sym3 system and
switching any that need to be switched.
Verifying that there are no service group dependencies.
Solution

Copyright 2012 Symantec Corporation. All rights reserved.

hastatus -sum
-- SYSTEM STATE
-- System

State

Frozen

A
A

RUNNING
RUNNING

0
0

sym3
sym4

-- GROUP STATE
-- Group

System

Probed

AutoDisabled

State

B
B
B
B
B
B
B
B
B
B

sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Y
Y
Y
Y
Y
Y
Y
Y
Y
Y

N
N
N
N
N
N
N
N
N
N

ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE

ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

hagrp -dep
VCS WARNING V-16-1-50035 No Group dependencies are configured

End of Solution

149 Lab 1: Service group dependencies

Copyright 2012 Symantec Corporation. All rights reserved.

B5

Display the state of all resources by listing the resources for each service group
and the dependencies between each resource in each service group.
Solution

Copyright 2012 Symantec Corporation. All rights reserved.

150 B6

hares -state
#Resource
appdg
appdg
appmnt
appmnt
appnic
appnic
appproc
appproc
appip
appip
appvol
appvol
csgnic
csgnic
dbdg
dbdg
dbip
dbip
dblistener
dblistener
dbmnt
dbmnt
dbnic
dbnic
dboracle
dboracle
dbvol
dbvol
notifier
notifier
testappdg
testappdg
testappmnt
testappmnt
testappnic
testappnic
testappproc
testappproc
testappip
testappip
testappvol
testappvol
webapache
webapache
webdg
webdg
webip
webip
webmnt
webmnt
webnic
webnic
webvip
webvip
webvol
webvol

Attribute
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
ONLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

hagrp -resources ClusterService


webip
csgnic
notifier

hares -dep | grep ClusterService


ClusterService notifier
ClusterService webip

csgnic
csgnic

hagrp -resources appsg


appproc
appdg
appmnt
appnic
appip
appvol

hares -dep | grep "^appsg"


appsg
appsg
appsg
appsg
appsg

appmnt
appproc
appproc
appip
appvol

appvol
appip
appmnt
appnic
appdg

hagrp -resources dbsg


dbdg
dbip
dblistener
dbmnt
dbnic
dboracle
dbvol

Copyright 2012 Symantec Corporation. All rights reserved.

151

hares -dep | grep dbsg


dbsg
dbsg
dbsg
dbsg
dbsg
dbsg

dbip
dblistener
dblistener
dbmnt
dboracle
dbvol

dbnic
dboracle
dbip
dbvol
dbmnt
dbdg

hagrp -resources testappsg


testappproc
testappdg
testappmnt
testappnic
testappip
testappvol

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

B7

hares -dep | grep testappsg


testappsg
testappsg
testappsg
testappsg
testappsg

testappmnt
testappproc
testappproc
testappip
testappvol

testappvol
testappip
testappmnt
testappnic
testappdg

hagrp -resources websg


webapache
webdg
webmnt
webnic
webvip
webvol

hares -dep | grep websg


websg
websg
websg
websg
websg

webapache
webapache
webmnt
webvip
webvol

webvip
webmnt
webvol
webnic
webdg

End of Solution

Determine if all resources in all service groups other than the ClusterService
service group are set to critical. If any resources are set to non critical then
modify them. Open, save and close the cluster configuration as appropriate.

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

152 B8

hares -list Critical=0 Group=appsg

hares -list Critical=0 Group=dbsg

hares -list Critical=0 Group=testappsg

hares -list Critical=0 Group=websg

End of Solution

Confirm that the FaultPropagation and ManageFaults service group


attributes are set to 1 and ALL respectively for each service group. Modify to
their default value if necessary. Open, save and close the cluster configuration
as appropriate.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Solution

hagrp -display -attribute ManageFaults


FaultPropagation
#Group
ClusterService
ClusterService
#
appsg
appsg
#
dbsg
dbsg
#
testappsg
testappsg
#
websg
websg

Attribute
FaultPropagation
ManageFaults

System
global
global

Value
1
ALL

FaultPropagation
ManageFaults

global
global

1
ALL

FaultPropagation
ManageFaults

global
global

1
ALL

FaultPropagation
ManageFaults

global
global

1
ALL

FaultPropagation
ManageFaults

global
global

1
ALL

End of Solution

Open the cluster configuration for update.


Solution

haconf -makerw
End of Solution

From sym3:terminal2, run the hastatus command in order to observe


cluster changes as they happen in subsequent exercises.

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

153

hastatus
testappproc
sym4
OFFLINE
testappip
sym3
ONLINE
testappip
sym4
OFFLINE
------------------------------------------------------------------------testappvol
sym3
ONLINE
testappvol
sym4
OFFLINE
webapache
sym3
OFFLINE
webapache
sym4
ONLINE
webdg
sym3
OFFLINE
------------------------------------------------------------------------webdg
sym4
ONLINE
group
resource
system
message
--------------- -------------------- -------------------- -------------------

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

B9

webmnt
sym3
OFFLINE
webmnt
sym4
ONLINE
webnic
sym3
OFFLINE
webnic
sym4
ONLINE
------------------------------------------------------------------------webvip
sym3
OFFLINE
webvip
sym4
ONLINE
webvol
sym3
OFFLINE
webvol
sym4
ONLINE

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

154 B10

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 3: Service group dependencies scenario 1


In this exercise, you configure a typical service group dependency so that one
service group will have multiple parents causing one service group to have
preference being online on startup and failover over another service group.

sym3
Note: The service groups in this exercise will come online and go offline very
quickly. The hagrp -wait commands, while not necessary, have been
included to be consistent with other lab exercises.
1

From sym3:terminal1, take all of the service groups except the


ClusterService service group offline on all cluster systems.
Solution

hagrp -offline dbsg -any


VCS NOTICE V-16-1-50733 Attempting to offline group on system sym3

hagrp -offline appsg -any


VCS NOTICE V-16-1-50733 Attempting to offline group on system sym3

hagrp -offline websg -any


VCS NOTICE V-16-1-50733 Attempting to offline group on system sym3

Copyright 2012 Symantec Corporation. All rights reserved.

155

hagrp -offline testappsg -any


VCS NOTICE V-16-1-50733 Attempting to offline group on system sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|

End of Solution

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

B11

Create and confirm dependencies between the appsg and dbsg and between
the websg and dbsg service groups as indicated below. Save but do not close
the VCS configuration.
appsg depends on dbsg (online local firm)
websg depends on dbsg (online local firm)
Solution

hagrp -link appsg dbsg online local firm

hagrp -link websg dbsg online local firm

hagrp -dep
#Parent
appsg
websg

Child
dbsg
dbsg

Relationship
online local firm
online local firm

haconf -dump

End of Solution

Use a single hagrp command with the propagate option to bring the dbsg,
appsg, and websg service groups online on sym3. Can all three service groups
be brought online using just one command? If not, then ensure that all three
service groups are brought online on sym3.

Both top level parent service groups are independent so only one of the parents and
the shared child dbsg service group can be brought online using the propagate option.

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

156 B12

hagrp -online -propagate websg -sys sym3


VCS NOTICE V-16-1-40147 Attempting to online group(s) websg dbsg on system
sym3.

hagrp -wait websg State ONLINE -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg

Attribute
State
State
State

System
sym3
sym4
sym3

Value
|ONLINE|
|OFFLINE|
|OFFLINE|

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

State
State
State
State
State
State
State

sym4
sym3
sym4
sym3
sym4
sym3
sym4

|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|

hagrp -online -propagate appsg -sys sym3


Note: The propagate option is not required nor is it invalid. The results are
the same as hagrp -online appsg -sys sym3.
VCS NOTICE V-16-1-40147 Attempting to online group(s) appsg on system sym3.

hagrp -wait appsg State ONLINE -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|

End of Solution

Copyright 2012 Symantec Corporation. All rights reserved.

157

Use a single hagrp command to take the dbsg, appsg, and websg service
groups offline on sym3. Can all three service groups be brought offline using
just one command? If not, then ensure that all three service groups are brought
offline on sym3.

All three service groups can be taken offline with one command.

Solution

hagrp -offline -propagate dbsg -sys sym3


VCS NOTICE V-16-1-40158 Attempting to offline group(s) dbsg websg appsg on
system sym3.

hagrp -wait dbsg State OFFLINE -sys sym3

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

B13

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|

End of Solution

Using a single hagrp command with the propagate option, bring the dbsg
and websg service groups online on sym3. Attempt to bring the appsg service
group online on sym4. Were you successful? If not successful, then bring the
appsg service group online on sym3.

The appsg service group cannot be brought online on sym4 due to service group
dependencies.

Solution

hagrp -online -propagate websg -sys sym3

Copyright 2012 Symantec Corporation. All rights reserved.

VCS NOTICE V-16-1-40147 Attempting to online group(s) websg dbsg on system


sym3.

158 B14

hagrp -wait websg State ONLINE -sys sym3

hagrp -online appsg -sys sym4


VCS WARNING V-16-1-10163 Group dependency is not met if group appsg goes
online on system sym4

hagrp -online appsg -sys sym3

hagrp -wait appsg State ONLINE -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg

Attribute
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

testappsg
testappsg
websg
websg

State
State
State
State

sym3
sym4
sym3
sym4

|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|

End of Solution

Fault the appsg service group by removing the /var/tmp/appip file


followed by a probe of the appip resource on sym3. Do any service groups
failover?

Only appsg faults. Neither the dbsg nor websg failover or change state.

Solution

rm /var/tmp/appip

hares -probe appip -sys sym3

hagrp -wait appsg State "OFFLINE|FAULTED" -sys


sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|OFFLINE|FAULTED|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

159

Clear the appip resource fault and bring the appsg service group back online
on sym3.
Solution

hares -clear appip -sys sym3

hagrp -online appsg -sys sym3

hagrp -wait appsg State ONLINE -sys sym3

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

B15

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|

End of Solution

Fault the dbsg service group by removing the /var/tmp/dbip file followed
by a probe of the dbip resource on sym3. Do any service groups failover?

The dbsg service group faults. The appsg and websg service groups are taken offline
on sym3. All three service groups failover to sym4.

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

160 B16

rm /var/tmp/dbip

hares -probe dbip -sys sym3

hagrp -wait dbsg State 'OFFLINE|FAULTED' -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|STOPPING|
|OFFLINE|
|OFFLINE|FAULTED|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|STOPPING|
|OFFLINE|

hagrp -wait appsg State ONLINE -sys sym4

hagrp -state
#Group
ClusterService
ClusterService
appsg

Attribute
State
State
State

System
sym3
sym4
sym3

Value
|ONLINE|
|OFFLINE|
|OFFLINE|

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

State
State
State
State
State
State
State

sym4
sym3
sym4
sym3
sym4
sym3
sym4

|ONLINE|
|OFFLINE|FAULTED|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|

End of Solution

Clear the dbip resource fault and take the dbsg, websg, appsg, and testappsg
service groups offline on any system they may be online.
Solution

hares -clear dbip -sys sym3

hagrp -offline -propagate dbsg -any


VCS NOTICE V-16-1-40173 Attempting to offline group(s) dbsg websg appsg on
system sym4.

hagrp -wait dbsg State OFFLINE -sys sym4

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

161

10 Link the testappsg service group as a parent of the appsg service group with

an offline local dependency. Save, but do not close the VCS configuration.
Note: This will configure a preference for the appsg service group running on
a system over the testappsg service group and force the testappsg to
switch upon a successful failover of the appsg service group.
Solution

hagrp -link testappsg appsg offline local

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

B17

hagrp -dep
#Parent
appsg
testappsg
websg
haconf -dump

Child
dbsg
appsg
dbsg

Relationship
online local firm
offline local
online local firm

haconf -dump

End of Solution

11 Bring the dgsg, appsg, and websg service groups online on sym3. Attempt to

bring the testappsg service group online on sym3. Were you successful? If
not, attempt to bring the testappsg service group online on sym4.
The testappsg service group cannot be brought online on the same system as the
appsg service group.
Solution

hagrp -online -propagate websg -sys sym3

Copyright 2012 Symantec Corporation. All rights reserved.

VCS NOTICE V-16-1-40147 Attempting to online group(s) websg dbsg on system


sym3.

162 B18

hagrp -wait websg State ONLINE -sys sym3

hagrp -online appsg -sys sym3

hagrp -wait appsg State ONLINE -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|

hagrp -online testappsg -sys sym3


VCS WARNING V-16-1-10163 Group dependency is not met if group testappsg goes
online on system sym3

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

hagrp -online testappsg -sys sym4

hagrp -wait testappsg State ONLINE -sys sym4

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|

End of Solution

12 Fault the dbsg service group by removing the /var/tmp/dbip file followed

by a probe of the dbip resource on sym3. Do any service groups failover?


The dbsg service group faults. The appsg and websg service groups are taken offline
on sym3. The testappsg service group is switched from sym4 to sym3. The dbsg,
appsg, and websg service groups failover to sym4.

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

163

rm /var/tmp/dbip

hares -probe dbip -sys sym3

hagrp -wait dbsg State 'OFFLINE|FAULTED' -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|PARTIAL|STOPPING|
|OFFLINE|
|OFFLINE|FAULTED|
|OFFLINE|
|OFFLINE|
|ONLINE|
|PARTIAL|STOPPING|
|OFFLINE|

Note: Depending on your timing, appsg may show OFFLINE on both


systems.

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

B19

hagrp -wait appsg State ONLINE -sys sym4

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|FAULTED|
|ONLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|

End of Solution

13 Take the testappsg service group offline on sym3 and clear the dbip resource

fault. Then, use a single command to take the dbsg, websg, appsg, and
testappsg service groups offline on any system where they may be online.
Solution

hagrp -offline testappsg -sys sym3

hares -clear dbip -sys sym3

hagrp -offline -propagate dbsg -any

Copyright 2012 Symantec Corporation. All rights reserved.

VCS NOTICE V-16-1-40173 Attempting to offline group(s) dbsg websg appsg on


system sym4.

164 B20

hagrp -wait dbsg State OFFLINE -sys sym4

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|

End of Solution

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

14 Unlink the service group dependencies between the testappsg and appsg

service groups and between the dbsg, websg and appsg service groups. Save,
but do not close the VCS configuration.
Solution

hagrp -unlink testappsg appsg

hagrp -unlink websg dbsg

hagrp -unlink appsg dbsg

hagrp -dep
VCS WARNING V-16-1-50035 No Group dependencies are configured

haconf -dump

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

165

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

B21

Exercise 4: Service group dependencies scenario 2


In this exercise, you configure a typical service group dependency with a three
level dependency causing one service group to have preference being online on
startup and failover over another service group.

sym3
Note: The service groups in this exercise will come online and go offline very
quickly. The hagrp -wait commands, while not necessary, have been
included to be consistent with other lab exercises.

From sym3:terminal1, create dependencies between the appsg and dbsg


service groups, between the websg and appsg service groups and between the
testappsg and appsg service groups as indicated below. Save but do not close
the VCS configuration.
appsg depends on dbsg (online local firm)
websg depends on appsg (online local firm)
testappsg depends on appsg (offline local)

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

166 B22

hagrp -link appsg dbsg online local firm

hagrp -link websg appsg online local firm

hagrp -link testappsg appsg offline local

hagrp -dep
#Parent
appsg
testappsg
websg

Child
dbsg
appsg
appsg

Relationship
online local firm
offline local
online local firm

haconf -dump

End of Solution

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Bring the testappsg service group online on sym3. Use a single hagrp
command with the propagate option to bring the dbsg, appsg, and websg
service groups online on sym3. Were you successful? If not, then switch the
testappsg service group to sym4 and retry. Can all three service groups be
brought online using just one command? If not, then ensure that all three
service groups are brought online on sym3.

The testappsg is already online on sym3 through normal mechanisms so the appsg
service group cannot come online there. It could, under the proper circumstances, fail
over to sym3 preferentially, but it cannot just come online there with the testappsg
already online.
All three service groups can be brought online using a single command. The dbsg
service group is brought online, followed by the appsg service group and finally the
websg service group.

Solution

hagrp -online testappsg -sys sym3

hagrp -wait testappsg State ONLINE -sys sym3

hagrp -online -propagate websg -sys sym3

Copyright 2012 Symantec Corporation. All rights reserved.

VCS WARNING V-16-1-40148 Cannot online group appsg on system sym3.

167

hagrp -switch testappsg -to sym4

hagrp -wait testappsg State ONLINE -sys sym4

hagrp -online -propagate websg -sys sym3


VCS NOTICE V-16-1-40147 Attempting to online group(s) websg appsg dbsg on
system sym3.

hagrp -wait websg State ONLINE -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg

Attribute
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|

B23

dbsg
testappsg
testappsg
websg
websg

State
State
State
State
State

sym4
sym3
sym4
sym3
sym4

|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|

End of Solution

Using a single hagrp command with the propagate option, take the dbsg,
appsg, and websg service groups offline on sym3. Can all three service
groups be brought offline using just one command? If not, then ensure that all
three service groups are brought offline on sym3.

All three service groups can be taken offline with one command. The testappsg
service group remains online.

Solution

hagrp -offline -propagate dbsg -sys sym3

Copyright 2012 Symantec Corporation. All rights reserved.

VCS NOTICE V-16-1-40158 Attempting to offline group(s) dbsg appsg websg on


system sym3.

168 B24

hagrp -wait dbsg State OFFLINE -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|

End of Solution

Using a single hagrp command with the propagate option, bring the dbsg,
appsg, and websg service groups online on sym3.
Solution

hagrp -online -propagate websg -sys sym3


VCS NOTICE V-16-1-40147 Attempting to online group(s) websg appsg dbsg on
system sym3.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

hagrp -wait websg State ONLINE -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|

End of Solution

Fault the appsg service group by removing the /var/tmp/appip file


followed by a probe of the appip resource on sym3. Do any service groups
failover?

The appsg faults. The websg is taken offline. No service groups fail over.

Solution

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

169

rm /var/tmp/appip

hares -probe appip -sys sym3

hagrp -wait appsg State "OFFLINE|FAULTED" -sys


sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|OFFLINE|FAULTED|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|PARTIAL|STOPPING|
|OFFLINE|

End of Solution

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

B25

Clear the appip resource fault and use a single hagrp command with the
propagate option to bring the appsg service group back online on sym3.
Solution

hares -clear appip -sys sym3

hagrp -online -propagate websg -sys sym3

hagrp -wait websg State ONLINE -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|

End of Solution

Fault the dbsg service group by removing the /var/tmp/dbip file followed
by a probe of the dbip resource on sym3. Do any service groups failover?

The dbsg service group faults. The appsg and websg service groups are brought
offline on sym3. The testappsg service group is switched from sym4 to sym3. The

Copyright 2012 Symantec Corporation. All rights reserved.

dbsg, appsg, and websg service groups failover to sym4.

170 B26

Solution

rm /var/tmp/dbip

hares -probe dbip -sys sym3

hagrp -wait dbsg State 'OFFLINE|FAULTED' -sys sym3

hagrp -state
#Group
Attribute
ClusterService State
ClusterService State

System
sym3
sym4

Value
|ONLINE|
|OFFLINE|

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

State
State
State
State
State
State
State
State

sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

|ONLINE|STOPPING|
|OFFLINE|
|OFFLINE|FAULTED|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|

hagrp -wait appsg State ONLINE -sys sym4

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|FAULTED|
|ONLINE|
|PARTIAL|STARTING|
|OFFLINE|
|OFFLINE|
|ONLINE|

End of Solution

Take the testappsg service group offline on sym3 and clear the dbip resource
fault. Then, use a single command to bring the dbsg, websg, appsg, and
testappsg service groups offline on any system they may be online.

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

171

hagrp -offline testappsg -sys sym3

hares -clear dbip -sys sym3

hagrp -offline -propagate dbsg -any


VCS NOTICE V-16-1-40173 Attempting to offline group(s) dbsg appsg on system
sym4.

hagrp -wait dbsg State OFFLINE -sys sym4

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg

Attribute
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|

B27

websg
websg

State
State

sym3
sym4

|OFFLINE|
|OFFLINE|

End of Solution

Unlink the service group dependencies between the testappsg and appsg
service groups, between the dbsg and appsg service groups and between the
websg and appsg service groups. Save, but do not close the VCS
configuration.
Solution

hagrp -unlink testappsg appsg

hagrp -unlink websg appsg

hagrp -unlink appsg dbsg

hagrp -dep
VCS WARNING V-16-1-50035 No Group dependencies are configured

haconf -dump

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

172 B28

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 5: Service group dependencies scenario 3


In this exercise, you configure a typical multi-child service group dependency so
that one service group has two child service group dependencies causing a service
group to have preference being online on startup and failover.

sym3
Note: The service groups in this exercise will come online and go offline very
quickly. The hagrp -wait commands, while not necessary, have been
included to be consistent with other lab exercises.

From sym3:terminal1, create dependencies between the websg and appsg


service groups and websg and dbsg service groups as indicated below. Save but
do not close the VCS configuration.
websg depends on appsg (online local firm)
websg depends on dbsg (online local firm)
testappsg depends on appsg (offline local)

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

173

hagrp -link websg appsg online local firm

hagrp -link websg dbsg online local firm

hagrp -link testappsg appsg offline local

hagrp -dep
#Parent
testappsg
websg
websg

Child
appsg
appsg
dbsg

Relationship
offline local
online local firm
online local firm

haconf -dump

End of Solution

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

B29

Bring the testappsg service group online on sym4. Using a single hagrp
command with the propagate option, bring the dbsg, appsg, and websg
service groups online on sym3. Were you able to bring the websg, dbsg, and
appsg service groups online with the online propagate option?

Yes, the online propagate option worked as expected.

Solution

hagrp -online testappsg -sys sym4

hagrp -wait testappsg State ONLINE -sys sym4

hagrp -online -propagate websg -sys sym3


VCS NOTICE V-16-1-40147 Attempting to online group(s) websg appsg dbsg on
system sym3.

hagrp -wait websg State ONLINE -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

174 B30

Using a single hagrp command with the propagate option, take the websg,
dbsg, and appsg and service groups offline on sym3. Can all three service
groups be brought offline using just one command? If not, then ensure that all
three service groups are brought offline on sym3.

No, all three service groups cannot be taken offline with one command. The appsg
service group needs to be taken offline explicitly in this case.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Solution

hagrp -offline -propagate dbsg -sys sym3


VCS NOTICE V-16-1-40158 Attempting to offline group(s) dbsg websg on system
sym3.

hagrp -wait dbsg State OFFLINE -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|

hagrp -offline appsg -sys sym3

hagrp -wait appsg State OFFLINE -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

175

Using a single hagrp command with the propagate option, bring the dbsg,
appsg, and websg service groups online on sym3.
Solution

hagrp -online -propagate websg -sys sym3


VCS NOTICE V-16-1-40147 Attempting to online group(s) websg appsg dbsg on
system sym3.

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

B31

hagrp -wait websg State ONLINE -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|

End of Solution

Fault the websg service group by removing the /var/tmp/webvip file


followed by a probe of the webvip resource on sym3. Do any service groups
failover?

The websg faults. No service groups change state or fail over.

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

176 B32

rm /var/tmp/webvip

hares -probe webvip -sys sym3

hagrp -wait websg State "OFFLINE|FAULTED" -sys


sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|FAULTED|
|OFFLINE|

End of Solution

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Clear the webvip resource fault and bring the websg service group back online
on sym3.
Solution

hares -clear webvip -sys sym3

hagrp -online websg -sys sym3

hagrp -wait websg State ONLINE -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|

End of Solution

Fault the dbsg service group by removing the /var/tmp/dbip file followed
by a probe of the dbip resource on sym3. Do any service groups failover?

The dbsg service group faults on sym3. The websg service group is taken offline on
sym3. The appsg service group is unaffected and no failover occurs. The testappsg

Copyright 2012 Symantec Corporation. All rights reserved.

service group remains online on sym4.

177

Solution

rm /var/tmp/dbip

hares -probe dbip -sys sym3

hagrp -wait dbsg State 'OFFLINE|FAULTED' -sys sym3

hagrp -state
#Group
Attribute
ClusterService State
ClusterService State

System
sym3
sym4

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

Value
|ONLINE|
|OFFLINE|

B33

appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

State
State
State
State
State
State
State
State

sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

|ONLINE|
|OFFLINE|
|OFFLINE|FAULTED|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|

End of Solution

Take the appsg service group offline on sym3. Then, bring the websg, appsg,
and dbsg service groups online on sym4. Account for the testappsg service
group.
Solution

hagrp -offline appsg -sys sym3

hagrp -wait appsg State OFFLINE -sys sym3

hagrp -switch testappsg -to sym3

hagrp -wait testappsg State OFFLINE -sys sym3

hagrp -online -propagate websg -sys sym4

Copyright 2012 Symantec Corporation. All rights reserved.

VCS NOTICE V-16-1-40147 Attempting to online group(s) websg appsg on system


sym4.

178 B34

hagrp -wait websg State ONLINE -sys sym4

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|FAULTED|
|ONLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|

End of Solution

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Clear the dbip resource fault.


Solution

hares -clear dbip -sys sym3

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|

End of Solution

10 Unlink the service group dependencies between the testappsg and appsg

service groups, between the dbsg and appsg service groups, and between the
websg and appsg service groups. Save and close the VCS configuration.
Solution

hagrp -unlink testappsg appsg

hagrp -unlink websg appsg

hagrp -unlink websg dbsg

hagrp -dep

Copyright 2012 Symantec Corporation. All rights reserved.

VCS WARNING V-16-1-50035 No Group dependencies are configured

179

haconf -dump -makero

End of Solution

Lab 1: Service group dependencies


Copyright 2012 Symantec Corporation. All rights reserved.

B35

11 From sym3:terminal2, terminate the hastatus command.


Solution

From sym3:terminal2, press Ctrl-C.


End of Solution

Copyright 2012 Symantec Corporation. All rights reserved.

End of lab

180 B36

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Lab 2: Merging clusters


In this lab, you will merge the east cluster to the west cluster forming one
four-node west cluster.
This lab contains the following exercises:
Exercise 1: Checking lab prerequisites
A verification that the virtual machines needed for this lab are powered on and
functioning is performed.
Exercise 2: Merge cluster preparation
Two clusters are prepared for merging.

Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 3: Merging cluster infrastructure


The systems from one cluster are added to another cluster.

181

Exercise 4: Adding service groups and resources


The service groups and resources from one of the original are added to the merged
cluster.
Exercise 5: Merged cluster verification
The merged cluster is verified.

Lab 2: Merging clusters

B37
Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 1: Checking lab prerequisites


In this exercise, you verify that the virtual machines needed for this lab are
powered on and functioning, that you are logged in using the proper account and
that any needed terminal windows are opened.

Copyright 2012 Symantec Corporation. All rights reserved.

182 B38

Verify that the following virtual machines shown are powered on, that you are
logged using the indicated account and that the indicated terminal windows are
opened.
mgt
There is no need to log into this virtual machine at this time.
sym1
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym2
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym3
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym4
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
Note: The terminal windows are referred to as hostname:terminal#
throughout the labs. For example: sym1:terminal1, sym1:terminal2,
sym2:terminal1, and sym2:terminal2. It is not necessary to label the
terminal windows and you may decide which is terminal1 and which is
terminal2.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

If you have machines running that are not used in this lab, shut down the
operating system and power the machines off.

Copyright 2012 Symantec Corporation. All rights reserved.

Note: If you are completing the lab exercises in order, this will require you to
power on and log into sym1 and sym2.

183

Lab 2: Merging clusters

B39
Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 2: Merge cluster preparation


In this exercise, you prepare two clusters for merging.

sym1
Note: In this lab there are no iSCSI LUNs that are visible to all four systems, so
merging enabled I/O fencing using disks cannot be part of the cluster
merge lab. Therefore, for lab purposes, I/O fencing is re-configured to
disabled mode on both clusters.

From sym1:terminal1, navigate to the /opt/VRTS/install directory and


run the installsfha script with the fencing option using the following
information.
Reconfigure fencing in disabled mode.
Stop VCS and apply the fencing changes.
Do not view the summary file.
Solution

cd /opt/VRTS/install

./installsfha -fencing
Checking communication on sym1
Checking release compatibility on sym1
Checking VCS installation on sym1

Copyright 2012 Symantec Corporation. All rights reserved.

Cluster information verification:

184 B40

Cluster Name: west


Cluster ID Number: 5
Systems: sym1 sym2
Would you like to configure I/O fencing on the cluster? [y,n,q]

y
Checking
Done
Checking
Checking
Checking

communication on sym1 ....................................


release compatibility on sym1 ............................ Done
VCS installation on sym1 .................. Version 6.0.000.000
communication on sym2 ....................................

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Done
Checking release compatibility on sym2 ............................ Done
Checking VCS installation on sym2 .................. Version 6.0.000.000
Fencing is already started in enabled mode, do you want to reconfigure it?
[y,n,q] (y)

y
Fencing configuration
1) Configure Coordination Point client based fencing
2) Configure disk based fencing
3) Configure fencing in disabled mode
4) Online fencing migration
Select the fencing mechanism to be configured in this Application Cluster:

3
Installer will stop VCS before applying fencing configuration. To make sure
VCS shuts down successfully, unfreeze any frozen service group and unmount
the mounted file systems in the cluster.
Are you ready to stop VCS and apply fencing configuration on all nodes at
this time? [y,n,q,b] (y)

y
Deleting Coordination Point Agent Service Group westfensg ......... Done
Stopping VCS on sym1 ..............................................
Done
Stopping Fencing on sym1 ..........................................
Done
Stopping VCS on sym2 ..............................................
Done
Stopping Fencing on sym2 ..........................................
Done
Configuring fencing in disabled mode on sym1 ...................... Done
Configuring fencing in disabled mode on sym2 ...................... Done
Updating main.cf without fencing ..................................
Done
Starting VCS on sym1 ..............................................
Done
Starting VCS on sym2 ..............................................
Done

Copyright 2012 Symantec Corporation. All rights reserved.

I/O Fencing configuration .........................................

185

Done
I/O Fencing configuration completed successfully
...
installsfha log files, summary file, and response file are saved at:
/opt/VRTS/install/logs/installsfha-201112081226AJb
Would you like to view the summary file? [y,n,q] (n)

End of Solution

Lab 2: Merging clusters

B41
Copyright 2012 Symantec Corporation. All rights reserved.

Use the gabconfig and service vxfen status commands to confirm


that I/O fencing is configured. Use the vxfenadm command to confirm that
I/O fencing is running in disabled mode.
Solution

gabconfig -a
GAB Port Memberships
===============================================================
Port a gen
1b5701 membership 01
Port b gen
1b5707 membership 01
Port h gen
1b570a membership 01

service vxfen status


running

vxfenadm -d
I/O Fencing Cluster Information:
================================
Fencing Protocol Version: 201
Fencing Mode: Disabled
Cluster Members:
* 0 (sym1)
1 (sym2)
RFSM State Information:
node
0 in state 8 (running)
node
1 in state 8 (running)

End of Solution

sym3

Copyright 2012 Symantec Corporation. All rights reserved.

186 B42

From sym3:terminal1, navigate to the /opt/VRTS/install directory and


run the installsfha script with the fencing option using the following
information.
Reconfigure fencing in disabled mode.
Stop VCS and apply the fencing changes.
Do not view the summary file.
Solution

cd /opt/VRTS/install

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

./installsfha -fencing
Checking communication on sym3
Checking release compatibility on sym3
Checking VCS installation on sym3
Cluster information verification:
Cluster Name: east
Cluster ID Number: 777
Systems: sym3 sym4
Would you like to configure I/O fencing on the cluster? [y,n,q]

y
Checking
Done
Checking
Checking
Checking
Done
Checking
Checking

communication on sym3 ....................................


release compatibility on sym3 ............................ Done
VCS installation on sym3 .................. Version 6.0.000.000
communication on sym4 ....................................
release compatibility on sym4 ............................ Done
VCS installation on sym4 .................. Version 6.0.000.000

Fencing is already started in enabled mode, do you want to reconfigure it?


[y,n,q] (y)

y
Fencing configuration
1) Configure Coordination Point client based fencing
2) Configure disk based fencing
3) Configure fencing in disabled mode
4) Online fencing migration
Select the fencing mechanism to be configured in this Application Cluster:

Copyright 2012 Symantec Corporation. All rights reserved.

187

3
Installer will stop VCS before applying fencing configuration. To make sure
VCS shuts down successfully, unfreeze any frozen service group and unmount
the mounted file systems in the cluster.
Are you ready to stop VCS and apply fencing configuration on all nodes at
this time? [y,n,q,b] (y)

y
Stopping VCS on sym3 .............................................
Stopping Fencing on sym3 .........................................
Stopping VCS on sym4 .............................................
Stopping Fencing on sym4 .........................................
Configuring fencing in disabled mode on sym3 ......................
Configuring fencing in disabled mode on sym4 ......................
Updating main.cf without fencing .................................
Starting VCS on sym3 .............................................
Starting VCS on sym4 .............................................

Done
Done
Done
Done
Done
Done
Done
Done
Done

I/O Fencing configuration ........................................ Done

Lab 2: Merging clusters

B43
Copyright 2012 Symantec Corporation. All rights reserved.

I/O Fencing configuration completed successfully


...
/opt/VRTS/install/logs/installsfha-201112081226Zat
Would you like to view the summary file? [y,n,q] (n)

End of Solution

Use the gabconfig and service vxfen status commands to confirm


that I/O fencing is configured. Use the vxfenadm command to confirm that
I/O fencing is running in disabled mode.
Solution

gabconfig -a
GAB Port Memberships
===============================================================
Port a gen
1b5701 membership 01
Port b gen
1b5707 membership 01
Port h gen
1b570a membership 01

service vxfen status


running

Copyright 2012 Symantec Corporation. All rights reserved.

188 B44

vxfenadm -d
I/O Fencing Cluster Information:
================================
Fencing Protocol Version: 201
Fencing Mode: Disabled
Cluster Members:
* 0 (sym3)
1 (sym4)
RFSM State Information:
node
0 in state 8 (running)
node
1 in state 8 (running)

End of Solution

Note: In this lab, there is a conflicting duplicate named appsg service group in
both the west and east clusters. This would cause errors during a merge.
The appsg service group in the west cluster would have precedence so the
appsg service group in the east cluster must be renamed or, as performed
in this lab, deleted from the configuration.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Take the appsg service group offline and list its resources and dependencies.
Solution

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg

Attribute
State
State
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|

hagrp -dep
VCS WARNING V-16-1-50035 No Group dependencies are configured

hagrp -offline appsg -any


Attempting to offline group on system sym3

hagrp -wait appsg State OFFLINE -sys sym3

hagrp -state appsg


#Group
appsg
appsg

Copyright 2012 Symantec Corporation. All rights reserved.

189

Attribute
State
State

System
sym3
sym4

Value
|OFFLINE|
|OFFLINE|

hagrp -resources appsg


appdg
appmnt
appnic
appproc
appip
appvo

hares -dep | grep "^appsg"


appsg
appsg
appsg
appsg
appsg

appmnt
appproc
appproc
appip
appvol

appvol
appip
appmnt
appnic
appdg

End of Solution

Lab 2: Merging clusters

B45
Copyright 2012 Symantec Corporation. All rights reserved.

Open the VCS configuration for update and delete each resource in the appsg
service group in resource dependency order. Then, delete the appsg service
group and save and close the VCS configuration.
Solution

haconf -makerw

hares -delete appproc

hares -delete appmnt

hares -delete appvol

hares -delete appdg

hares -delete appip

hares -delete appnic

hagrp -resources appsg

hagrp -delete appsg

hagrp -state
#Group
ClusterService
ClusterService
dbsg
dbsg
testappsg
testappsg
websg
websg

Copyright 2012 Symantec Corporation. All rights reserved.

190 B46

Attribute
State
State
State
State
State
State
State
State

System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|

haconf -dump -makero

End of Solution

Navigate to the /etc/VRTSvcs/conf/config directory and use the


hacf -cftocmd command to create a main.cmd file. Copy the
main.cmd file to eastmain.cmd.
Solution

cd /etc/VRTSvcs/conf/config

hacf -cftocmd .
Veritas Cluster Server 5.1 for UNIX: Install and Configure
Copyright 2012 Symantec Corporation. All rights reserved.

cp -p main.cmd eastmain.cmd

ls *.cmd
eastmain.cmd

main.cmd

End of Solution

Edit the eastmain.cmd file to remove all commands other than those
related to the websg, testappsg, dbsg service groups, their resources, and their
resource dependencies.

Copyright 2012 Symantec Corporation. All rights reserved.

Note: Alternatively, copy the


/student/labs/vcs/vcs60/maincf/CM/eastmain.cmd
file to /etc/VRTSvcs/conf/config/eastmain.cmd.

191

After you have made the edits the contents of the file should be:
hagrp -add dbsg
hagrp -modify dbsg SystemList sym3 0 sym4 1
hagrp -modify dbsg AutoStartList sym3 sym4
hagrp -modify dbsg SourceFile "./main.cf"
hares -add dbdg FileOnOff dbsg
hares -modify dbdg PathName "/var/tmp/dbdg"
hares -modify dbdg Enabled 1
hares -add dbip FileOnOff dbsg
hares -modify dbip PathName "/var/tmp/dbip"
hares -modify dbip Enabled 1
hares -add dblistener FileOnOff dbsg
hares -modify dblistener PathName "/var/tmp/
dblistener"
hares -modify dblistener Enabled 1
hares -add dbmnt FileOnOff dbsg
hares -modify dbmnt PathName "/var/tmp/dbmnt"
hares -modify dbmnt Enabled 1
hares -add dbnic FileOnOff dbsg
hares -modify dbnic PathName "/var/tmp/dbnic"
hares -modify dbnic Enabled 1
hares -add dboracle FileOnOff dbsg
hares -modify dboracle PathName "/var/tmp/
dboracle"
hares -modify dboracle Enabled 1
hares -add dbvol FileOnOff dbsg

Lab 2: Merging clusters

B47
Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

192 B48

hares -modify dbvol PathName "/var/tmp/dbvol"


hares -modify dbvol Enabled 1
hagrp -add testappsg
hagrp -modify testappsg SystemList sym3 0 sym4 1
hagrp -modify testappsg AutoStartList sym3 sym4
hagrp -modify testappsg Operators oper
hagrp -modify testappsg SourceFile "./main.cf"
hares -add testappdg FileOnOff testappsg
hares -modify testappdg PathName "/var/tmp/
testappdg"
hares -modify testappdg Enabled 1
hares -add testappmnt FileOnOff testappsg
hares -modify testappmnt PathName "/var/tmp/
testappmnt"
hares -modify testappmnt Enabled 1
hares -add testappnic FileOnOff testappsg
hares -modify testappnic PathName "/var/tmp/
testappnic"
hares -modify testappnic Enabled 1
hares -add testappproc FileOnOff testappsg
hares -modify testappproc PathName "/var/tmp/
testappproc"
hares -modify testappproc Enabled 1
hares -add testappip FileOnOff testappsg
hares -modify testappip PathName "/var/tmp/
testappip"
hares -modify testappip Enabled 1
hares -add testappvol FileOnOff testappsg
hares -modify testappvol PathName "/var/tmp/
testappvol"
hares -modify testappvol Enabled 1
hagrp -add websg
hagrp -modify websg SystemList sym3 0 sym4 1
hagrp -modify websg AutoStartList sym3 sym4
hagrp -modify websg Operators oper
hagrp -modify websg SourceFile "./main.cf"
hares -add webapache FileOnOff websg
hares -modify webapache PathName "/var/tmp/
webapache"
hares -modify webapache Enabled 1

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

193

hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares

-add webdg FileOnOff websg


-modify webdg PathName "/var/tmp/webdg"
-modify webdg Enabled 1
-add webmnt FileOnOff websg
-modify webmnt PathName "/var/tmp/webmnt"
-modify webmnt Enabled 1
-add webnic FileOnOff websg
-modify webnic PathName "/var/tmp/webnic"
-modify webnic Enabled 1
-add webvip FileOnOff websg
-modify webvip PathName "/var/tmp/webvip"
-modify webvip Enabled 1
-add webvol FileOnOff websg
-modify webvol PathName "/var/tmp/webvol"
-modify webvol Enabled 1
-link dbip dbnic
-link dblistener dbip
-link dblistener dboracle
-link dbmnt dbvol
-link dboracle dbmnt
-link dbvol dbdg
-link testappmnt testappvol
-link testappproc testappmnt
-link testappproc testappip
-link testappip testappnic
-link testappvol testappdg
-link webapache webmnt
-link webapache webvip
-link webmnt webvol
-link webvip webnic
-link webvol webdg

Solution

vi eastmain.cmd

Remove the appropriate lines.

Lab 2: Merging clusters

B49
Copyright 2012 Symantec Corporation. All rights reserved.

From the vi editor in command mode, type: :wq

End of Solution

Shutdown VCS, IO fencing, GAB, and LLT on both nodes of the east cluster.
Solution

hastop -all

gabconfig -a
GAB Port Memberships
===============================================================
Port a gen
a9e001 membership 01
Port b gen
a9e007 membership 01

service vxfen stop


Stopping vxfen..
Stopping vxfen.. Done

ssh sym4 service vxfen stop


Stopping vxfen..
Stopping vxfen.. Done

service gab stop


Stopping GAB:

ssh sym4 service gab stop


Stopping GAB:

Copyright 2012 Symantec Corporation. All rights reserved.

194 B50

service llt stop


Stopping LLT:

ssh sym4 service llt stop


Stopping LLT:

gabconfig -a
GAB gabconfig ERROR V-15-2-25022 unknown error

ssh sym4 gabconfig -a


GAB gabconfig ERROR V-15-2-25022 unknown error

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

lltstat -nvv configured


LLT lltstat ERROR V-14-2-15000 open /dev/llt failed: No such file or
directory

ssh sym4 lltstat -nvv configured


LLT lltstat ERROR V-14-2-15000 open /dev/llt failed: No such file or
directory

End of Solution

10 Rename the /etc/VRTSvcs/conf/config/main.cf file to

main.cf.east on sym3 and sym4.


Note: In order for the merge utility to work correctly, the main.cf files on
the east cluster nodes must not exist.
Solution

mv main.cf main.cf.east

ls -l main.cf
ls: main.cf No such file or directory

ssh sym4 mv /etc/VRTSvcs/conf/config/main.cf


/etc/VRTSvcs/conf/config/main.cf.east

ssh sym4 ls -l /etc/VRTSvcs/conf/config/main.cf


main.cf: No such file or directory

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

195

Lab 2: Merging clusters

B51
Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 3: Merging cluster infrastructure


In this exercise, you add the systems from one cluster to another cluster.

sym1
1

From sym1:terminal1, navigate to the /opt/VRTS/install directory and


run the ./installsfha -addnode command taking note of the merge
pre-requisites and using the following information.

Enter sym1 as the name of one node of the west cluster.


Enter sym3 and sym4 as the systems to be added to the west cluster.
Use eth4 and eth5 as the private links and eth0 as the low priority link.
Use eth0 as the public NIC used for the cluster virtual IP.
Record the log file located in the /opt/VRTS/install/logs
directory. Optionally, review the log file.
Do not view the summary file.
Log file:

The log file will be in the form of

installsfha-yyyymmddhhmmxxx
Solution

cd /opt/VRTS/install

./installsfha -addnode
Logs are being written to /var/tmp/installsfha-201112081249IHO while
installsfha
is in progress.

Copyright 2012 Symantec Corporation. All rights reserved.

Following are the prerequisites to add a node to the cluster:

196 B52

* The cluster to which you want to add the node must have all required
SFHA rpms installed
* New node must have all required SFHA rpms installed
* SFHA must be running on the cluster to which you want to add a node
* There should be no prior running SFHA processes on the new node
* New node must have the same SFHA version as that of the existing
cluster nodes

Refer to the SFHA Installation Guide for more details

Press [Enter] to continue:

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Press Enter.
Enter one node of the SFHA cluster to which you would like to add one or
more new nodes:

sym1
Checking communication on sym1 ....................................
Done
Checking release compatibility on sym1 ............................ Done
Following cluster information detected:
Cluster Name: west
Cluster ID: 5
Systems: sym1 sym2
Is this information correct? [y,n,q] (y)

y
Checking communication on sym2 ....................................
Done
Checking VCS running state on sym1 ................................ Done
Checking VCS running state on sym2 ................................ Done
Enter the system names separated by spaces to add to the cluster:

sym3 sym4
Checking
Done
Checking
Checking
Done
Checking

communication on sym3 ....................................


release compatibility on sym3 ............................ Done
communication on sym4 ....................................
release compatibility on sym4 ............................ Done

Do you want to add the system(s) sym3 sym4 to the cluster west? [y,n,q] (y)

Copyright 2012 Symantec Corporation. All rights reserved.

197

y
Checking
Checking
Checking
Checking
Checking

installed
installed
installed
installed
installed

product on cluster west ............ SFHA 6.0.000.000


product on sym3 .................... SFHA 6.0.000.000
packages on sym3 ............................... Done
product on sym4 .................... SFHA 6.0.000.000
packages on sym4 ............................... Done

Each node to be added to the cluster should have the same LLT heartbeat
configuration as the existing node(s).
To connect to the existing node(s) properly, each new node is required to be
configured with 2 private and 1 low-priority heartbeat links.
Discovering NICs on sym3 ...... Discovered eth0 eth1 eth2 eth3 eth4 eth5
Enter the NIC for the first private heartbeat link on sym3: [q,?] (eth4)

eth4
Enter the NIC for the second private heartbeat link on sym3: [b,q,?] (eth5)

Lab 2: Merging clusters

B53
Copyright 2012 Symantec Corporation. All rights reserved.

eth5
Enter the NIC for the low-priority heartbeat link on sym3: [b,q,?] (eth0)

eth0
Are you using the same NICs for private heartbeat links on all systems?
[y,n,q,b,?]

y
Checking
Checking
Checking
Checking
Checking
Checking

media
media
media
media
media
media

speed
speed
speed
speed
speed
speed

Private Heartbeat NICs


link1=eth4
link2=eth5
Low-Priority Heartbeat
link-lowpri1=eth0
Private Heartbeat NICs
link1=eth4
link2=eth5
Low-Priority Heartbeat
link-lowpri1=eth0

for
for
for
for
for
for

eth4
eth5
eth4
eth5
eth4
eth5

on
on
on
on
on
on

sym3
sym3
sym4
sym4
sym1
sym1

.........................
.........................
.........................
.........................
.........................
.........................

1000Mb/s
1000Mb/s
1000Mb/s
1000Mb/s
1000Mb/s
1000Mb/s

for sym3:

NIC for sym3:


for sym4:

NIC for sym4:

Is this information correct? [y,n,q] (y)

y
Cluster Virtual IP is configured on the cluster west
A public NIC device is required by following services on each of the newly
added nodes:
Cluster Virtual IP
Active NIC devices discovered on sym3: eth0 eth1 eth2 eth3 eth4 eth0

Copyright 2012 Symantec Corporation. All rights reserved.

Enter the NIC for the VCS to use on sym3: (eth0)

198 B54

m eth0
Is eth0 the public NIC used by other new nodes? [y,n,q] (y)

y
Starting llt
Starting
Starting
Starting
Done
Checking
Updating
Updating
Starting
Starting

on sym3 .............................................. Done


llt on sym4 ............................................. Done
gab on sym3 ............................................. Done
gab on sym4 ..............................................
GAB control port seed .......... Checking GAB control port
csgnic resource .........................................
webip resource ..........................................
vxfen on sym3 ...........................................
vxfen on sym4 ...........................................

seed
Done
Done
Done
Done

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Starting
Starting
Starting
Starting

amf
had
amf
had

on
on
on
on

sym3
sym3
sym4
sym4

.............................................
.............................................
.............................................
.............................................

Done
Done
Done
Done

Addnode completed successfully


installsfha log files, summary file, and response file are saved at:
/opt/VRTS/install/logs/installsfha-201112081249IHO

End of Solution

Perform a summary status of the cluster noting that none of the former east
cluster service groups have been added.
Solution

hastatus -sum
-- SYSTEM STATE
-- System

State

Frozen

A
A
A
A

RUNNING
RUNNING
RUNNING
RUNNING

0
0
0
0

sym1
sym2
sym3
sym4

-- GROUP STATE
-- Group

System

B
B
B
B
B
B
B
B
B
B

sym1
sym2
sym3
sym4
sym1
sym2
sym1
sym2
sym1
sym2

ClusterService
ClusterService
ClusterService
ClusterService
appsg
appsg
nfssg
nfssg
orasg
orasg

Probed
Y
Y
Y
Y
Y
Y
Y
Y
Y

AutoDisabled
N
N
N
N
N
N
N
N
N

State
ONLINE
OFFLINE
OFFLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

199

Use the gabconfig command to check GAB status.


Solution

gabconfig -a
GAB Port Memberships
===============================================================
Port a gen
1b5702 membership 0123
Port b gen
1b5709 membership 0123
Port h gen
1b570c membership 0123

End of Solution

Lab 2: Merging clusters

B55
Copyright 2012 Symantec Corporation. All rights reserved.

Use the vxfenadm command to check I/O fencing.


Solution

vxfenadm -d
I/O Fencing Cluster Information:
================================
Fencing Protocol Version: 201
Fencing Mode: Disabled
Cluster Members:
* 0
1
2
3

(sym1)
(sym2)
(sym3)
(sym4)

RFSM State Information:


node
0 in state 8 (running)
node
1 in state 8 (running)
node
2 in state 8 (running)
node
3 in state 8 (running)

End of Solution

Use the lltstat command to check the LLT configured and active status.
Optionally, performed this step on all cluster nodes.
Solution

lltstat -nvv configured


LLT node information:
Node
* 0 sym1

Copyright 2012 Symantec Corporation. All rights reserved.

1 sym2

200 B56

Link

Status

Address

eth4
eth5
eth0

UP
UP
UP

00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE

eth4
eth5
eth0

UP
UP
UP

00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72

eth4
eth5
eth0

UP
UP
UP

00:0C:29:97:FB:5D
00:0C:29:97:FB:67
00:0C:29:97:FB:35

eth4
eth5
eth0

UP
UP
UP

00:0C:29:C4:1D:0A
00:0C:29:C4:1D:14
00:0C:29:C4:1D:E2

OPEN

2 sym3

OPEN

3 sym4

State
OPEN

OPEN

lltstat -nvv active


LLT node information:
Node
* 0 sym1

State
OPEN

Link
eth4
eth5

Status
UP
UP

Address
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

1 sym2

eth0

UP

00:0C:29:5A:B1:EE

eth4
eth5
eth0

UP
UP
UP

00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72

eth4
eth5
eth0

UP
UP
UP

00:0C:29:97:FB:5D
00:0C:29:97:FB:67
00:0C:29:97:FB:35

eth4
eth5
eth0

UP
UP
UP

00:0C:29:C4:1D:0A
00:0C:29:C4:1D:14
00:0C:29:C4:1D:E2

OPEN

2 sym3

OPEN

3 sym4

OPEN

End of Solution

Display the contents of the /etc/llttab, /etc/llthosts, and


/etc/gabtab files. Optionally, perform this step on all cluster nodes.

Solution

cat /etc/llthosts
0
1
2
3

sym1
sym2
sym3
sym4

cat /etc/llttab
set-node sym3
set-cluster 5
link eth4 eth-00:0c:29:97:fb:5d - ether - link eth5 eth-00:0c:29:97:fb:67 - ether - link-lowpri eth0 eth-00:0c:29:97:fb:35 - ether - -

cat /etc/gabtab
/sbin/gabconfig -c -n4

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

201

Display the /etc/VRTSvcs/conf/config/main.cf file noticing the


following elements.

The cluster name is west.


The cluster virtual ip is 10.10.2.51.
All four systems are defined as cluster members.
The SystemList and AutoStartList service group attributes for the
ClusterService service group configure all four cluster systems.
The appsg, nfssg, and orasg service groups from the original west cluster
are configured and remain restricted to sym1 and sym2.
The websg, dbsg, and testappsg from the east cluster are not configured.

Lab 2: Merging clusters

B57
Copyright 2012 Symantec Corporation. All rights reserved.

Solution

Note: Only the main.cf file though the ClusterService service group
definition is shown.
more /etc/VRTSvcs/conf/config/main.cf
...
include
include
include
include
include

"OracleASMTypes.cf"
"types.cf"
"Db2udbTypes.cf"
"OracleTypes.cf"
"SybaseTypes.cf"

cluster west (
UserNames = { admin = bQRjQLqNRmRRpZRlQO }
ClusterAddress = "10.10.2.51"
Administrators = { admin }
)
system sym1 (
)
system sym2 (
)
system sym3 (
)
system sym4 (
)

Copyright 2012 Symantec Corporation. All rights reserved.

group ClusterService (
SystemList = { sym1 = 0, sym2 = 1, sym3 = 2, sym4 = 3 }
AutoStartList = { sym1, sym2, sym3, sym4 }
OnlineRetryLimit = 3
OnlineRetryInterval = 120
)

202 B58

IP webip (
Device = eth0
Address = "10.10.2.51"
NetMask = "255.255.255.0"
)
NIC csgnic (
Device = eth0
)
NotifierMngr notifier (
SmtpServer = "mgt.example.com"
SmtpServerVrfyOff = 1
SmtpRecipients = { student = Warning }
)
notifier requires csgnic
webip requires csgnic
...

End of Solution

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 4: Adding service groups and resources


In this exercise, you add the service groups and resources from one of the original
clusters to the merged cluster.

sym3
1

From sym3:terminal1, navigate to the /etc/VRTSvcs/conf/config


directory and open the VCS configuration for update
Solution

cd /etc/VRTSvcs/conf/config

haconf -makerw

End of Solution

Run the sh -x ./eastmain.cmd command and observe the output taking


notice of the VCS warning when failing to add the VCS user named oper as an
Operator for the testappsg and websg service groups.
Note: The oper user add is not part of the eastmain.cmd file and would
need to be re-added and re-configured later on.

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

sh -x ./eastmain.cmd
+ hagrp -add dbsg
VCS NOTICE V-16-1-10136 Group added; populating SystemList and setting the
Parallel attribute recommended before adding resources
+ hagrp -modify dbsg SystemList sym3 0 sym4 1
+ hagrp -modify dbsg AutoStartList sym3 sym4
+ hagrp -modify dbsg SourceFile ./main.cf
+ hares -add dbdg FileOnOff dbsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify dbdg PathName /var/tmp/dbdg
+ hares -modify dbdg Enabled 1
+ hares -add dbip FileOnOff dbsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify dbip PathName /var/tmp/dbip

203 Lab 2: Merging clusters

B59
Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

204 B60

+ hares -modify dbip Enabled 1


+ hares -add dblistener FileOnOff dbsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify dblistener PathName /var/tmp/dblistener
+ hares -modify dblistener Enabled 1
+ hares -add dbmnt FileOnOff dbsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify dbmnt PathName /var/tmp/dbmnt
+ hares -modify dbmnt Enabled 1
+ hares -add dbnic FileOnOff dbsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify dbnic PathName /var/tmp/dbnic
+ hares -modify dbnic Enabled 1
+ hares -add dboracle FileOnOff dbsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify dboracle PathName /var/tmp/dboracle
+ hares -modify dboracle Enabled 1
+ hares -add dbvol FileOnOff dbsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify dbvol PathName /var/tmp/dbvol
+ hares -modify dbvol Enabled 1
+ hagrp -add testappsg
VCS NOTICE V-16-1-10136 Group added; populating SystemList and setting the
Parallel attribute recommended before adding resources
+ hagrp -modify testappsg SystemList sym3 0 sym4 1
+ hagrp -modify testappsg AutoStartList sym3 sym4
+ hagrp -modify testappsg Operators oper
VCS WARNING V-16-1-50109 User oper is not a valid Cluster user
+ hagrp -modify testappsg SourceFile ./main.cf
+ hares -add testappdg FileOnOff testappsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify testappdg PathName /var/tmp/testappdg
+ hares -modify testappdg Enabled 1
+ hares -add testappmnt FileOnOff testappsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify testappmnt PathName /var/tmp/testappmnt
+ hares -modify testappmnt Enabled 1
+ hares -add testappnic FileOnOff testappsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify testappnic PathName /var/tmp/testappnic
+ hares -modify testappnic Enabled 1
+ hares -add testappproc FileOnOff testappsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify testappproc PathName /var/tmp/testappproc
+ hares -modify testappproc Enabled 1
+ hares -add testappip FileOnOff testappsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify testappip PathName /var/tmp/testappip
+ hares -modify testappip Enabled 1
+ hares -add testappvol FileOnOff testappsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify testappvol PathName /var/tmp/testappvol
+ hares -modify testappvol Enabled 1
+ hagrp -add websg
VCS NOTICE V-16-1-10136 Group added; populating SystemList and setting the

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Parallel attribute recommended before adding resources


+ hagrp -modify websg SystemList sym3 0 sym4 1
+ hagrp -modify websg AutoStartList sym3 sym4
+ hagrp -modify websg Operators oper
VCS WARNING V-16-1-50109 User oper is not a valid Cluster user
+ hagrp -modify websg SourceFile ./main.cf
+ hares -add webapache FileOnOff websg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be
agent monitors
+ hares -modify webapache PathName /var/tmp/webapache
+ hares -modify webapache Enabled 1
+ hares -add webdg FileOnOff websg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be
agent monitors
+ hares -modify webdg PathName /var/tmp/webdg
+ hares -modify webdg Enabled 1
+ hares -add webmnt FileOnOff websg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be
agent monitors
+ hares -modify webmnt PathName /var/tmp/webmnt
+ hares -modify webmnt Enabled 1
+ hares -add webnic FileOnOff websg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be
agent monitors
+ hares -modify webnic PathName /var/tmp/webnic
+ hares -modify webnic Enabled 1
+ hares -add webvip FileOnOff websg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be
agent monitors
+ hares -modify webvip PathName /var/tmp/webvip
+ hares -modify webvip Enabled 1
+ hares -add webvol FileOnOff websg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be
agent monitors
+ hares -modify webvol PathName /var/tmp/webvol
+ hares -modify webvol Enabled 1
+ hares -link dbip dbnic
+ hares -link dblistener dbip
+ hares -link dblistener dboracle
+ hares -link dbmnt dbvol
+ hares -link dboracle dbmnt
+ hares -link dbvol dbdg
+ hares -link testappmnt testappvol
+ hares -link testappproc testappmnt
+ hares -link testappproc testappip
+ hares -link testappip testappnic
+ hares -link testappvol testappdg
+ hares -link webapache webmnt
+ hares -link webapache webvip
+ hares -link webmnt webvol
+ hares -link webvip webnic
+ hares -link webvol webdg

set before

set before

set before

set before

set before

set before

End of Solution

Save and close the VCS configuration.


Solution

haconf -dump -makero


End of Solution

205 Lab 2: Merging clusters

B61
Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 5: Merged cluster verification


In this exercise, you verify the merged cluster.

sym3
1

From sym3:terminal1, perform a summary status on the cluster taking note of


the service groups that are not online.
Solution

Copyright 2012 Symantec Corporation. All rights reserved.

hastatus -sum

206 B62

-- SYSTEM STATE
-- System

State

Frozen

A
A
A
A

RUNNING
RUNNING
RUNNING
RUNNING

0
0
0
0

sym1
sym2
sym3
sym4

-- GROUP STATE
-- Group

System

B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B

sym1
sym2
sym3
sym4
sym1
sym2
sym3
sym4
sym1
sym2
sym1
sym2
sym3
sym4
sym3
sym4

ClusterService
ClusterService
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
nfssg
nfssg
orasg
orasg
testappsg
testappsg
websg
websg

Probed
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y

AutoDisabled
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N

State
ONLINE
OFFLINE
OFFLINE
OFFLINE
ONLINE
OFFLINE
OFFLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
OFFLINE
OFFLINE
OFFLINE
OFFLINE

End of Solution

Display /etc/VRTSvcs/conf/config/main.cf file and notice the


following.
The oper user from the old east cluster is not defined nor is the Operator
service group attribute for the testappsg and websg service groups set to
the oper user.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

The SystemList and AutoStartList service group attributes for the


ClusterService service group have been modified for all four cluster
systems.
The SystemList and AutoStartList service group attributes for the appsg,
nfssg, and orasg service groups remain set to sym1 and sym2.
Note: These service groups cannot be online on sym3 and sym4 due to
shared LUN restrictions.
The SystemList and AutoStartList service group attributes for the websg,
testappsg, and dbsg service groups remain set to sym3 and sym4.
Note: These service groups could be online on sym1 and sym2 and could
be modified to accommodate that.

Note: Consideration needs to be given to activated triggers configured at any


service group and/or resource level if the service group is going to be
re-configured to run on a new cluster system.
Solution

Copyright 2012 Symantec Corporation. All rights reserved.

more /etc/VRTSvcs/conf/config/main.cf
...
cluster west (
UserNames = { admin = bQRjQLqNRmRRpZRlQO }
ClusterAddress = "10.10.2.51"
Administrators = { admin }
)
...
group ClusterService (
SystemList = { sym1 = 0, sym2 = 1, sym3 = 2, sym4 = 3 }
AutoStartList = { sym1, sym2, sym3, sym4 }
OnlineRetryLimit = 3
OnlineRetryInterval = 120
)
...
group appsg (
SystemList = { sym1 = 0, sym2 = 1 }
AutoStartList = { sym1, sym2 }
TriggersEnabled @sym1 = { NOFAILOVER, RESFAULT }
TriggersEnabled @sym2 = { NOFAILOVER, RESFAULT }
)
...
group nfssg (
SystemList = { sym1 = 0, sym2 = 1 }
AutoStartList = { sym1, sym2 }
)
...
group orasg (
SystemList = { sym1 = 0, sym2 = 1 }
AutoStartList = { sym1, sym2 }

207 Lab 2: Merging clusters

B63
Copyright 2012 Symantec Corporation. All rights reserved.

)
...
group websg (
SystemList = { sym3 = 0, sym4 = 1 }
AutoStartList = { sym3, sym4 }
)
...
group testappsg (
SystemList = { sym3 = 0, sym4 = 1 }
AutoStartList = { sym3, sym4 }
)
...
group dbsg (
SystemList = { sym3 = 0, sym4 = 1 }
AutoStartList = { sym3, sym4 }
...

End of Solution

Copyright 2012 Symantec Corporation. All rights reserved.

End of lab

208 B64

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Lab 3: Failover policies


In this lab, you use the Veritas Cluster Server Simulator with a preconfigured
main.cf file to show how to configure and test failover policies using the Veritas
Cluster Manager Java Console.
This lab contains the following exercises:
Exercise 1: Checking lab prerequisites
A verification that the virtual machines needed for this lab are powered on and
functioning is performed.

Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 2: Starting the simulator


The simulation used in the lab is examined and the Veritas VCS Simulator is
started.
Exercise 3: Testing priority failover policy
The default priority failover policy is examined.
Exercise 4: Testing load failover policy
The load failover policy is examined.
Exercise 5: Testing prerequisites and limits
Prerequisites and limits are examined and added to the load failover policy
settings.
Exercise 6: Stopping the simulator
The simulation used in the lab is examined and the Veritas VCS Simulator is
terminated.

209 Lab 3: Failover policies

B65
Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 1: Checking lab prerequisites


In this exercise, you verify that the virtual machines needed for this lab are
powered on and functioning, that you are logged in using the proper account and
that any needed terminal windows are opened.
1

Verify that the following virtual machines shown are powered on, that you are
logged using the indicated account and that the indicated terminal windows are
opened.
mgt
There is no need to log into this virtual machine at this time.
winclient
Log in with credentials:
Account: Administrator
Password: veritas
Note: The terminal windows are referred to as hostname:terminal#
throughout the labs. For example: sym1:terminal1, sym1:terminal2,
sym2:terminal1, and sym2:terminal2. It is not necessary to label the
terminal windows and you may decide which is terminal1 and which is
terminal2.

Copyright 2012 Symantec Corporation. All rights reserved.

210 B66

If you have machines running that are not used in this lab, shut down the
operating system and power the machines off.

Note: If you are completing the lab exercises in order, this will require you to
shutdown sym1, sym2, sym3 and sym4, and power on and log into
winclient.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 2: Starting the simulator


In this exercise, you examine the simulation used in the lab and start the Veritas
VCS Simulator.

winclient
1

From the winclient desktop, use Windows Explorer to navigate to the


C:\Progam Files (x86)\VERITAS\VCS Simulator\wlm\
conf\config folder.
Solution

From the desktop, double-click the Computer icon.

From the left pane, expand and select Computer > C: >
Program Files (x86) > VERITAS > VCS Simulator > wlm > conf >
config.

End of Solution

Open the main.cf file using the WordPad application.

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

211

Right-click the main.cf file and select Open.

From the pop-up window, select Select a program from a list of installed
programs.

Click OK.

Clear Always use the selected program to open this kind of file.

Select WordPad.

Click OK.

End of Solution

Lab 3: Failover policies

B67
Copyright 2012 Symantec Corporation. All rights reserved.

Without making any changes, review the contents of the main.cf file. Then,
close the Wordpad and Windows Explorer windows.
Solution

From the Wordpad window, review the contents of the main.cf file.

Click the Close button that appears as an X in the upper right hand corner
of the window.

From the Windows Explorer window, click the Close button that appears
as an X in the upper right hand corner of the window.

End of Solution

Start the Veritas VCS Simulator.


Note: The VCS Simulator may take a moment to display as it discovers any
running simulations even thought there will not be any.
Solution

From the desktop, double-click the Veritas VCS Simulator Java


Console icon.
Note: Alternately, from the start bar, select Start > All programs >
Symantec > Veritas VCS Simulator Java Console.

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

212 B68

Verify the wlm simulation.


Solution

From the Symantec Veritas Cluster Server Simulation window, from the
left pane list, select wlm.

From the task (right) pane, click Verify Configuration.

From the pop-up window, click OK.

End of Solution

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Start the wlm simulation.


Note: The status of the cluster name will have a green check mark upon
successful startup.
Solution

From the left pane list, select wlm.

From the task (right) pane, Click Start Cluster.

End of Solution

Launch the VCS Java Console and log in using the following credentials.
User name: admin
Password: password
Solution

From the left pane list, select wlm.

From the task (right) pane, Click Launch Console.

In the User name field, type: admin

In the Password field, type: password

Click OK.

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

213

Lab 3: Failover policies

B69
Copyright 2012 Symantec Corporation. All rights reserved.

From the Java Console, verify that the Member Systems and their Priority
and their AutoStartList attribute are set to the values shown in the following
table.
Service Group

Member Systems (Priority)

AutoStartList

A1

S1 (1), S2 (2), S3 (3), S4 (4)

S1

A2

S1 (1), S2 (2), S3 (3), S4 (4)

S1

B1

S1 (4), S2 (1), S3 (2), S4 (3)

S2

B2

S1 (4), S2 (1), S3 (2), S4 (3)

S2

C1

S1 (3), S2 (4), S3 (1), S4 (2)

S3

C2

S1 (3), S2 (4), S3 (1), S4 (2)

S3

D1

S1 (2), S2 (3), S3 (4), S4 (1)

S4

D2

S1 (2), S2 (3), S3 (4), S4 (1)

S4

Solution

From the left pane, select wlm > A1.

From the right pane, select the Properties tab.

Verify the AutoStartList attribute.

From the System Details section, verify Member Systems and their
Priority.

Repeat steps a through d for the A2, B1, B2, C1, C2, D1 and D2 service
groups.

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

214 B70

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 3: Testing priority failover policy


In this exercise, you examine the default priority failover policy.

winclient
1

From the Java Console, verify that the failover policy of all service groups is
Priority.
Solution

From the left pane, select wlm > A1.

From the right pane, verify that the FailOverPolicy attribute is set to
Priority.

Repeat steps a and b for the A2, B1, B2, C1, C2, D1 and D2 service
groups.

End of Solution

Copyright 2012 Symantec Corporation. All rights reserved.

215

Verify that all service groups are online as shown in the following table.
Service Group

System S1

A1

Online

A2

Online

System S2

B1

Online

B2

Online

System S3

C1

Online

C2

Online

System S4

D1

Online

D2

Online

Solution

From the left pane, select wlm.

Lab 3: Failover policies

B71
Copyright 2012 Symantec Corporation. All rights reserved.

From the right pane, select the Status tab.


Note: This view is referred to as the Cluster Status view.

Verify the service groups are online as shown in the above table.

End of Solution

If it faults, where will the A1 service group fail over to? Verify this failover by
faulting a critical resource in the A1 service group.
Service Group A1 will fail over to:

S2

Solution

From left pane, expand wlm > A1 > FileOnOff > A1File.

Right-click wlm > A1 > FileOnOff > A1File and select


Fault Resource > S1.

From the pop-up window, click Yes.

From the right pane, notice that the A1 service group shows faulted on S1,
but Online on S2.

End of Solution

Copyright 2012 Symantec Corporation. All rights reserved.

216 B72

If the A1 service group faults again without clearing the previous fault, where
should it fail over? Verify the failover by faulting a critical resource in the A1
service group.
Service Group A1 will fail over to:

S3

Solution

From the left pane, right-click wlm > A1> FileOnOff > A1File and select
Fault Resource > S2.

From the pop-up window, click Yes.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

From the right pane, notice that the A1 service group shows faulted on S1,
and S2 but Online on S3.

End of Solution

Clear the faults in the A1 service group. Where will the service group fail over
to now? Verify the failover by faulting a critical resource in the A1 service
group.
Service Group A1 will fail over to:

S1

Solution

From the left pane, right-click wlm > A1 > FileOnOff > A1File and select
Clear Fault > Auto.

Right-click wlm > A1 > FileOnOff > A1File and select


Fault Resource > S3.

From the pop-up window, click Yes.

From the right pane, notice that the A1 service group shows faulted on S3,
but Online on S1.

End of Solution

Clear the faults in the A1 service group.


Solution

Copyright 2012 Symantec Corporation. All rights reserved.

From the left pane, right-click wlm > A1 > FileOnOff > A1File and select
Clear Fault > Auto.

217

End of Solution

Lab 3: Failover policies

B73
Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 4: Testing load failover policy


In this exercise, you examine the load failover policy.

winclient
1

From the Java Console, open the cluster configuration for update.
Solution

From the menu, select File > Open Configuration.


End of Solution

Set the FailOverPolicy attribute to Load for the eight service groups.

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

218 B74

From the left pane, select wlm > A1.

From the right pane, select the Properties tab.

To the right of the FailOverPolicy attribute click the Edit button.

From the Edit Attribute window, from the Scaler Value drop-down menu,
select Load.
Note: The drop-down menu will appear when you click in the Scaler
Value field.

Click OK.

Repeat steps a through e for the A2, B1, B2, C1, C2, D1 and D2 service
groups.

End of Solution

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Set the Load attribute for each service group as shown in the following table.
Service Group

Load

A1

75

A2

75

B1

75

B2

75

C1

50

C2

50

D1

50

D2

50

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

219

From the left pane, select wlm > A1.

From the right pane, select the Properties tab.

Click Show all attributes.

Scroll to the Load attribute and click the Edit icon.

From the Edit Attribute window, in the Scaler Value field, type: 75

Click OK.

Close the Attributes View window.

Repeat steps a through g for the A2, B1, B2, C1, C2, D1 and D2 service
groups using the Load value shown in the table above.

End of Solution

Set the S1 and S2 systems Capacity attributes to 200.


Note: The S3 and S4 systems Capacity attribute will not be changed and
defaults to 100.
Solution

From the left pane, select the Systems tab.

Lab 3: Failover policies

B75
Copyright 2012 Symantec Corporation. All rights reserved.

Select wlm > S1.

From the right pane, select the Properties tab.

Click Show all attributes.

Scroll to the Capacity attribute and click the Edit icon.

From the pop-up window, in the Scaler Value field, type: 200

Click OK.

Close the Attributes View window.

Repeat steps a through h for the S2 system.

End of Solution

Verify that the systems AvailableCapacity attribute is as shown in the


following table.
Service Group

AvailableCapacity

S1

50

S2

50

S3

S4

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

220 B76

From the left pane, select wlm > S1.

From the right pane, select the Properties tab.

Click Show all attributes.

Verify that the AvaliableCapacity is set to the value shown in the table
above.

Close the Attributes View window.

Repeat steps a through e for the S2, S3 and S4 systems.

End of Solution

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Save the configuration changes.


Solution

From the menu, select File >Save Configuration.


End of Solution

Verify that all service groups are online as shown in the following table.
Service Group

System S1

A1

Online

A2

Online

System S2

B1

Online

B2

Online

System S3

C1

Online

C2

Online

System S4

D1

Online

D2

Online

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

221

From the left pane, select the Service Groups tab.

Select wlm.

From the right pane, select the Status tab.

Verify the service groups are online as shown in the above table.

End of Solution

If it faults, where will the A1 service group fail over to? Verify this failover by
faulting a critical resource in the A1 service group.
Service Group A1 will fail over to:

S2

Solution

From left pane, expand wlm > A1 > FileOnOff > A1File.

Lab 3: Failover policies

B77
Copyright 2012 Symantec Corporation. All rights reserved.

Right-click wlm > A1 > FileOnOff > A1File and select


Fault Resource > S1.

From the pop-up window, click Yes.

From the right pane, notice that the A1 service group shows faulted on S1,
but Online on S2.

End of Solution

Verify that the systems AvailableCapacity attribute is as shown in the


following table.
Service Group

AvailableCapacity

S1

125

S2

-25

S3

S4

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

222 B78

From the left pane, select the Systems tab.

Select wlm > S1.

From the right pane, select the Properties tab.

Click Show all attributes.

Verify that the AvaliableCapacity is set to the value shown in the table
above.

Close the Attributes View window.

Repeat steps b through e for the S2, S3 and S4 systems.

End of Solution

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

10 If the S2 system fails, where will the A1, B1 and B2 service groups that are

Online on S2 fail over? Verify this failover by powering off the S2 system in
Cluster Manager.
The A1 service group will fail over to:

S3

The B1 service group will fail over to:

S1

The B2 service group will fail over to:

S1

Solution

From the left pane, right-click wlm > S2 and select on Power Off.

Select wlm.

From the right pane, select the Status tab.

Notice that the A1 service group shows faulted on S1, but Online on S3.

Notice that the B1 and B2 service groups show Online on S1.

End of Solution

11 Verify that the systems AvailableCapacity attribute is as shown in the

Copyright 2012 Symantec Corporation. All rights reserved.

following table.
Service Group

AvailableCapacity

S1

-25

S2

200

S3

-75

S4

Solution

From the left pane, select wlm > S1.

From the right pane, select the Properties tab.

Click Show all attributes.

Verify that the AvaliableCapacity is set to the value shown in the table
above.

Close the Attributes View window.

223 Lab 3: Failover policies

B79
Copyright 2012 Symantec Corporation. All rights reserved.

Repeat steps b through f for the S2, S3 and S4 systems.

End of Solution

12 Power up the S2 system, clear all faults, and return the service groups to their

startup locations.
Solution

From the left pane, right-click wlm > S2 and select Up > S2.

Select the Service Groups tab.

Right-click wlm > A1 and select Clear Fault > Auto.

Right-click wlm > A1 and select Switch To > S1.

From the pop-up window, click Yes.

Right-click wlm > B1 and select Switch To > S2.

From the pop-up window, click Yes.

Right-click wlm > B2 and select Switch To > S2.

From the pop-up window, click Yes.

Solution

13 Verify that the systems AvailableCapacity attribute is as shown in the

Copyright 2012 Symantec Corporation. All rights reserved.

following table.

224 B80

Service Group

AvailableCapacity

S1

50

S2

50

S3

S4

Solution

From the left pane, select the Systems tab.

Select wlm > S1.


Veritas Cluster Server 5.1 for UNIX: Install and Configure
Copyright 2012 Symantec Corporation. All rights reserved.

From the right pane, select the Properties tab.

Click Show all attributes.

Verify that the AvaliableCapacity is set to the value shown in the table
above.

Close the Attributes View window.

Repeat steps b through f for the S2, S3 and S4 systems.

End of Solution

14 Save and close the configuration changes.

Note: Saving changes to the configuration is not necessary for purposes of


this lab. However, if you want to view the modifications to the
main.cf file then save the changes.
Solution

From the menu, select File >Close Configuration.

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

225 Lab 3: Failover policies

B81
Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 5: Testing prerequisites and limits


In this exercise, you examine the prerequisites and limits and add them to the load
failover policy settings.

winclient
1

From the Java Console, open the cluster configuration for update.
Solution

From the menu, select File > Open Configuration.


End of Solution

Set the Limits attribute for each system to the key ABGroup with a value of 3.

Copyright 2012 Symantec Corporation. All rights reserved.

Solution

226 B82

From the left pane, select the Systems tab.

Select wlm > S1.

From the right pane, select the Properties tab.

Click Show all Attributes.

Scroll to the Limits attribute and click the Edit icon.

From the Edit Attribute window, click the add an element button.

In the Key field, type: ABGroup

In the Value field, type: 3

Click OK.

Close the Attributes View window.

Repeat steps b through j for the S2, S3 and S4 systems.

End of Solution

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Set the Prerequisites attribute for the A1, A2, B1 and B2 service groups to be
ABGroup with a value of 1.
CAUTION Do not modify the Prerequisites attribute for the C1, C2, D1,
and D2 service groups.

Solution

From the left pane, select the Service Groups tab.

Select wlm > A1.

From the right pane, select the Properties tab.

Click Show all Attributes.

Scroll to the Prerequisites attribute and click the Edit icon.

From the Edit Attribute window, click the add an element button.

In the Key field, type: ABGroup

In the Value field, type: 1

Click OK.

Close the Attributes View window.

Repeat steps b through j for the A2, B1 and B2 service groups.

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

Save the configuration changes.


Solution

From the menu, select File >Save Configuration.


End of Solution

227 Lab 3: Failover policies

B83
Copyright 2012 Symantec Corporation. All rights reserved.

If the S1 system fails, where will the A1 and A2 service groups that are Online
on S1 fail over? Verify this failover by powering off the S1 system in Cluster
Manager. Explain the failover actions that occurred.
The A1 service group will fail over to:

S2

The A2 service group will fail over to:

S3

A1 failed over to S2, but A2 failed over to S3 because the limit was reached on S2.

Solution

From the left pane, select the Systems tab.

Select wlm.

From the right pane, select the Status tab.

From the left pane, right-click wlm > S1 and select on Power Off.

Select wlm.

From the right pane, select the Status tab.

Notice that the A1 service group shows Online on S2.

Notice that the A2 service group show Online on S3.

End of Solution

Copyright 2012 Symantec Corporation. All rights reserved.

228 B84

If the S2 system fails, where will the A1, B1 and B2 service groups that are
Online on S2 fail over? Verify this failover by powering off the S2 system in
Cluster Manager. Explain the failover actions that occurred.
The A1 service group will fail over to:

S4

The B1 service group will fail over to:

S3

The B2 service group will fail over to:

S4

A1 failed over to S4, B1 failed over to S3, and B2 failed over to S4 due to the values
of the Load attribute on each system.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Solution

From the left pane, right-click wlm > S2 and select on Power Off.

Select wlm.

From the right pane, select the Status tab.

Notice that the A1 service group shows Online on S4.

Notice that the B1 service group show Online on S3.

Notice that the B2 service group show Online on S4.

End of Solution

If the S3 system fails, where will the A2, B1, C1 and C2 service groups that
are Online on S3 fail over? Verify this failover by powering off the S3 system
in Cluster Manager. Explain the failover actions that occurred.
The A2 service group will fail over to:

S4

The B1 service group will fail over to:

B1 does not fail over and stays offline

The C1 service group will fail over to:

S4

The C2 service group will fail over to:

S4

All service groups fail over to S4 except B1. B1 is the last group to attempt to fail over
to S4. Since B1 has a prerequisite that cannot be met on S4, it cannot fail over. A1,

Copyright 2012 Symantec Corporation. All rights reserved.

A2, B1, and B2 can run on the same system. B1 stays offline.

Solution

From the left pane, right-click wlm > S3 and select on Power Off.

Select wlm.

From the right pane, select the Status tab.

Notice that the A2 service group shows Online on S4.

Notice that the B1 service group shows Offline on all systems.

229 Lab 3: Failover policies

B85
Copyright 2012 Symantec Corporation. All rights reserved.

Notice that the C1 service group show Online on S4.

Notice that the C2 service group show Online on S4.

End of Solution

Save and close the configuration changes.


Note: Saving changes to the configuration is not necessary for purposes of
this lab. However, if you want to view the modifications to the
main.cf file then save the changes.
Solution

From the menu, select File >Close Configuration.

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

230 B86

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 6: Stopping the simulator


In this exercise, you terminate the simulation and the Veritas VCS Simulator.

winclient
1

Log out from Cluster Explorer.


Solution

From the menu, select File >Log Out.


End of Solution

From the Simulator Java Console, stop the wlm cluster.


Solution

From the left pane list, select wlm.

From the task (right) pane, click Stop Cluster.

End of Solution

Copyright 2012 Symantec Corporation. All rights reserved.

231

Verify the configuration.


Note: If you intend to move this configuration in to a running cluster, you
must ensure the main.cf syntax is correct. You can verify the
configuration only in the Simulator GUI after the cluster is stopped.
Solution

From the task (right) pane, click Verify Configuration.

From the pop-up window, click OK.

End of Solution

Lab 3: Failover policies

B87
Copyright 2012 Symantec Corporation. All rights reserved.

Close the Simulator Java Console.


Note: The Veritas Cluster Manager Java Console should close as well. If it
does not then close it manually.
Solution

Click the Close button that appears as an X in the upper right hand corner
of the window.
End of Solution

Copyright 2012 Symantec Corporation. All rights reserved.

End of lab

232 B88

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Lab 4: Creating a parallel network service group


In this lab application networking is reconfigured to be more efficient.
This lab contains the following exercises:
Exercise 1: Checking lab prerequisites
A verification that the virtual machines needed for this lab are powered on and
functioning is performed.
Exercise 2: Configuring a common parallel network service group
A common parallel network service group and resources is configured.

Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 3: Replacing NIC resources with Proxy resources


The pre-existing resources of type NIC supporting application VIPs in each
application service group are replaced with resources of type Proxy.

233 Lab 4: Creating a parallel network service group

Copyright 2012 Symantec Corporation. All rights reserved.

B89

Exercise 1: Checking lab prerequisites


In this exercise, you verify that the virtual machines needed for this lab are
powered on and functioning, that you are logged in using the proper account and
that any needed terminal windows are opened.

Copyright 2012 Symantec Corporation. All rights reserved.

234 B90

Verify that the following virtual machines shown are powered on, that you are
logged using the indicated account and that the indicated terminal windows are
opened.
mgt
There is no need to log into this virtual machine at this time.
sym1
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym2
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym3
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym4
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
Note: The terminal windows are referred to as hostname:terminal#
throughout the labs. For example: sym1:terminal1, sym1:terminal2,
sym2:terminal1, and sym2:terminal2. It is not necessary to label the
terminal windows and you may decide which is terminal1 and which is
terminal2.

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

If you have machines running that are not used in this lab, shut down the
operating system and power the machines off.

Copyright 2012 Symantec Corporation. All rights reserved.

Note: If you are completing the lab exercises in order, this will require you to
shutdown winclient, and power on and log into sym1, sym2, sym3
and sym4.

235 Lab 4: Creating a parallel network service group

Copyright 2012 Symantec Corporation. All rights reserved.

B91

Exercise 2: Configuring a common parallel network service group


In this exercise, you configure a common parallel network service group and
resources.

sym1
1

From sym1:terminal1, use the haconf command to open the cluster


configuration for update.
Solution

haconf -makerw
End of Solution

Use the hagrp command to create a service group named netsg.


Solution

hagrp -add netsg


VCS NOTICE V-16-1-10136 Group added; populating SystemList and setting the
Parallel attribute recommended before adding resources

Solution

Copyright 2012 Symantec Corporation. All rights reserved.

236 B92

Modify the SystemList to allow the netsg service group to run on the sym1,
sym2, sym3, and sym4 in that order.
Solution

hagrp -modify netsg SystemList sym1 0 sym2 1


sym3 2 sym4 3
End of Solution

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Modify the AutoStartList attribute to allow the service group to start on all
systems.
Solution

hagrp -modify netsg AutoStartList sym1 sym2 sym3


sym4
End of Solution

Modify the Parallel service group to configure the netsg service group as a
parallel service group.
Solution

hagrp -modify netsg Parallel 1


End of Solution

Display the service group attributes for the netsg service group to confirm your
input and key default service group attribute values.
Solution

Copyright 2012 Symantec Corporation. All rights reserved.

hagrp -display netsg | more


#Group
netsg
netsg
netsg
netsg
netsg
netsg
netsg
netsg
netsg
netsg
...
netsg
...
netsg
sym3
...

Attribute
AdministratorGroups
Administrators
Authority
AutoFailOver
AutoRestart
AutoStart
AutoStartIfPartial
AutoStartList
AutoStartPolicy
ClusterFailOverPolicy

System
global
global
global
global
global
global
global
global
global
global

Value

Parallel

global

SystemList
sym4

global

0
1
1
1
1
sym1
Order
Manual

sym1

sym2

sym3

sym4

sym2

End of Solution

237 Lab 4: Creating a parallel network service group

Copyright 2012 Symantec Corporation. All rights reserved.

B93

Save the VCS configuration, but do not close it.


Solution

haconf -dump
End of Solution

Add a resource of type NIC named netnic to the netsg service group.
Solution

hares -add netnic NIC netsg


VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors

End of Solution

Modify the Device and Critical resource attributes for the netnic resource
using the following information.
Device is eth0
Critical set to 0 (zero)
Solution

hares -modify netnic Device eth0

hares -modify netnic Critical 0

End of Solution

10 Display the resource attribute values for the netnic resource to confirm your

Copyright 2012 Symantec Corporation. All rights reserved.

input and key default resource attribute values.

238 B94

Solution

hares -display netnic | more


#Resource
netnic
netnic
netnic
netnic
netnic
...
netnic
...

Attribute
Group
Type
AutoStart
Critical
Enabled

System
global
global
global
global
global

Value
netsg
NIC
1
0
0

Device

global

eth0

End of Solution

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

11 Enable the netnic resource and verify that it is enabled. Display the state of the

netnic resource to ensure that it is online on both cluster systems.


Note: It is not necessary to bring a resource of type NIC online.
Solution

hares -modify netnic Enabled 1

hares -value netnic Enabled


1

hares -state netnic


#Resource
netnic
netnic
netnic
netnic

Attribute
State
State
State
State

System
sym1
sym2
sym3
sym4

Value
ONLINE
ONLINE
ONLINE
ONLINE

End of Solution

12 Display the state of the netsg service group.

Note: The service group state is offline even though the NIC resource state is
online. This is because VCS does not consider persistent resources
when determining the state of a service group.
Solution

Copyright 2012 Symantec Corporation. All rights reserved.

hagrp -state netsg


#Group
netsg
netsg
netsg
netsg

Attribute
State
State
State
State

System
sym1
sym2
sym3
sym4

Value
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|

End of Solution

13 Save the VCS configuration, but do not close it.


Solution

haconf -dump
End of Solution

239 Lab 4: Creating a parallel network service group

Copyright 2012 Symantec Corporation. All rights reserved.

B95

14 Add a resource of type Phantom named netphantom to the netsg service

group.
Solution

hares -add netphantom Phantom netsg


VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors

End of Solution

15 Set the Critical resource attribute to 0 (zero) for the netphantom resource.
Solution

hares -modify netphantom Critical 0


End of Solution

16 Display the resource attribute values for the netphantom resource to confirm

your input and key default resource attribute values.


Solution

hares -display netphantom | more


#Resource
netphantom
netphantom
netphantom
netphantom
netphantom
...

Attribute
Group
Type
AutoStart
Critical
Enabled

System
global
global
global
global
global

Value
netsg
Phantom
1
0
0

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

240 B96

17 Enable the netphantom resource and verify that it is enabled. Display the state

of the netphantom resource to ensure that it is online on both cluster systems.


Note: It is not necessary to bring a resource of type Phantom online.
Solution

hares -modify netphantom Enabled 1

hares -value netphantom Enabled


1

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

hares -state netphantom


#Resource
netphantom
netphantom
netphantom
netphantom

Attribute
State
State
State
State

System
sym1
sym2
sym3
sym4

Value
ONLINE
ONLINE
ONLINE
ONLINE

End of Solution

18 Display the state of the netsg service group.

Note: The service group state is offline even though the NIC resource state is
online. This is because VCS does not consider persistent resources
when determining the state of a service group.
Solution

hagrp -state netsg


#Group
netsg
netsg
netsg
netsg

Attribute
State
State
State
State

System
sym1
sym2
sym3
sym4

Value
|ONLINE|
|ONLINE|
|ONLINE|
|ONLINE|

End of Solution

19 Save the VCS configuration, but do not close it.


Solution

haconf -dump

Copyright 2012 Symantec Corporation. All rights reserved.

End of Solution

241 Lab 4: Creating a parallel network service group

Copyright 2012 Symantec Corporation. All rights reserved.

B97

Exercise 3: Replacing NIC resources with Proxy resources


In this exercise, you replace the pre-existing resources of type NIC that supporting
application VIPs in each application service group with resources of type Proxy.

sym1
1

From sym1:terminal1, identify the names of the resources of type NIC that are
configured.
Solution

hares -list Type=NIC


appnic
appnic
csgnic
csgnic
csgnic
csgnic
netnic
netnic
netnic
netnic
nfsnic
nfsnic
oranic
oranic

sym1
sym2
sym1
sym2
sym3
sym4
sym1
sym2
sym3
sym4
sym1
sym2
sym1
sym2

End of Solution

Copyright 2012 Symantec Corporation. All rights reserved.

242 B98

Ignoring the resource of type NIC named netnic, list the resource
dependencies associated with the listed resources of type NIC.
Solution

hares -dep appnic csgnic nfsnic oranic


#Group
ClusterService
ClusterService
appsg
nfssg
orasg

Parent
notifier
webip
appip
nfsip
oraip

Child
csgnic
csgnic
appnic
nfsnic
oranic

End of Solution

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Replace the resource of type NIC named appnic with a resource of type Proxy
named appproxy using the following information.

Set the Critical attribute to 0 (zero).


Set the TargetResName attribute to netnic.
Set Enabled to 1 (one).
Link to the appip resource.

Solution

hares -delete appnic

hares -add appproxy Proxy appsg


VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors

hares -modify appproxy Critical 0

hares -modify appproxy TargetResName netnic

hares -display appproxy | more


#Resource
appproxy
appproxy
appproxy
appproxy
appproxy
...
appproxy
...

Attribute
Group
Type
AutoStart
Critical
Enabled

System
global
global
global
global
global

Value
appsg
Proxy
1
0
0

TargetResName

global

netnic

hares -modify appproxy Enabled 1

hares -value appproxy Enabled

Copyright 2012 Symantec Corporation. All rights reserved.

hares -state appproxy


#Resource
appproxy
appproxy

Attribute
State
State

System
sym1
sym2

hares -link appip appproxy

hares -dep | grep appproxy


appsg

appip

Value
ONLINE
ONLINE

appproxy

End of Solution

243 Lab 4: Creating a parallel network service group

Copyright 2012 Symantec Corporation. All rights reserved.

B99

Replace the resource of type NIC named csgnic with a resource of type Proxy
named csgproxy using the following information.

Set the Critical attribute to 0 (zero).


Set the TargetResName attribute to netnic.
Set Enabled to 1 (one).
Link to the notifier and webip resources.

Solution

hares -delete csgnic

hares -add csgproxy Proxy ClusterService


VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors

hares -modify csgproxy Critical 0

hares -modify csgproxy TargetResName netnic

hares -display csgproxy | more


#Resource
csgproxy
csgproxy
csgproxy
csgproxy
csgproxy
...
csgproxy
...

Attribute
Group
Type
AutoStart
Critical
Enabled

System
global
global
global
global
global

Value
ClusterService
Proxy
1
0
0

TargetResName

global

netnic

hares -modify csgproxy Enabled 1

hares -value csgproxy Enabled

Copyright 2012 Symantec Corporation. All rights reserved.

244B100

hares -state csgproxy


#Resource
csgproxy
csgproxy
csgproxy
csgproxy

Attribute
State
State
State
State

System
sym1
sym2
sym3
sym4

hares -link webip csgproxy

hares -link notifier csgproxy

Value
ONLINE
ONLINE
ONLINE
ONLINE

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

hares -dep | grep csgproxy


ClusterService notifier
ClusterService webip

csgproxy
csgproxy

End of Solution

Replace the resource of type NIC named nfsnic with a resource of type Proxy
named nfsproxy using the following information.

Set the Critical attribute to 0 (zero).


Set the TargetResName attribute to netnic.
Set Enabled to 1 (one).
Link to the nfsip resource.

Solution

hares -delete nfsnic

hares -add nfsproxy Proxy nfssg

Copyright 2012 Symantec Corporation. All rights reserved.

VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors

hares -modify nfsproxy Critical 0

hares -modify nfsproxy TargetResName netnic

hares -display nfsproxy | more


#Resource
nfsproxy
nfsproxy
nfsproxy
nfsproxy
nfsproxy
...
nfsproxy
...

Attribute
Group
Type
AutoStart
Critical
Enabled

System
global
global
global
global
global

Value
nfssg
Proxy
1
0
0

TargetResName

global

netnic

hares -modify nfsproxy Enabled 1

hares -value nfsproxy Enabled


1

hares -state nfsproxy


#Resource
nfsproxy
nfsproxy

Attribute
State
State

245 Lab 4: Creating a parallel network service group

System
sym1
sym2

Value
ONLINE
ONLINE

Copyright 2012 Symantec Corporation. All rights reserved.

B101

hares -link nfsip nfsproxy

hares -dep | grep nfsproxy


nfssg

nfsip

nfsproxy

End of Solution

Replace the resource of type NIC named oranic with a resource of type Proxy
named oraproxy using the following information.

Set the Critical attribute to 0 (zero).


Set the TargetResName attribute to netnic.
Set Enabled to 1 (one).
Link to the oraip resource.

Solution

hares -delete oranic

hares -add oraproxy Proxy orasg

Copyright 2012 Symantec Corporation. All rights reserved.

VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors

246 B102

hares -modify oraproxy Critical 0

hares -modify oraproxy TargetResName netnic

hares -display oraproxy | more


#Resource
oraproxy
oraproxy
oraproxy
oraproxy
oraproxy
...
oraproxy
...

Attribute
Group
Type
AutoStart
Critical
Enabled

System
global
global
global
global
global

Value
orasg
Proxy
1
0
0

TargetResName

global

netnic

hares -modify oraproxy Enabled 1

hares -value oraproxy Enabled


1

hares -state oraproxy


#Resource
oraproxy
oraproxy

Attribute
State
State

System
sym1
sym2

Value
ONLINE
ONLINE

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

hares -link oraip oraproxy

hares -dep | grep oraproxy


orasg

oraip

oraproxy

End of Solution

Save and close the VCS configuration.


Solution

haconf -dump -makero


End of Solution

Copyright 2012 Symantec Corporation. All rights reserved.

End of lab

247 Lab 4: Creating a parallel network service group

Copyright 2012 Symantec Corporation. All rights reserved.

B103

Copyright 2012 Symantec Corporation. All rights reserved.

248 B104

Veritas Cluster Server 5.1 for UNIX: Install and Configure


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Appendix C

Supplemental Content

249

Copyright 2012 Symantec Corporation. All rights reserved.

Service group dependenciesFailover process

250 C2

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

251

The following steps describe what happens when a service group in a service
group dependency relationship is faulted due to a critical resource fault:
1 The entire service group is taken offline due to the critical resource fault
together with any of its parent service groups that have an online firm or hard
dependency (online local firm, online global firm, online remote firm, or
online local hard).
2 Then a failover target is chosen from the SystemList of the service group based
on the failover policy and the restrictions brought by the service group
dependencies. Note that if the faulted service group is also the parent service
group in a service group dependency relationship, the service group
dependency has an impact on the choice of a target system. For example, if the
faulted service group has an online local (firm or soft) dependency with a child
service group that is online only on that system, no failover targets are
available.
3 If there are no other systems to which the service group can fail over, both the
child service group and all of the parents that were already taken offline remain
offline.
4 If there is a failover target then VCS takes any child service group with an
online local hard dependency offline.
5 VCS then checks if there are any conflicting parent service groups that are
already online on the target system. These service groups can be parent service
groups that are linked with an offline local dependency or online remote soft
dependency. In either case, the parent service group is taken offline to enable
the child service group to start on that system.
6 If there is any child service group with an online local hard dependency, first
the child service group and then the service group that initiated the failover are
brought online.
7 After the service group is brought online successfully on the target system,
VCS takes any parent service groups offline that have an online local soft
dependency to the failed-over child.
8 Finally, VCS selects a failover target for any parent service groups that may
have been taken offline during steps 1, 5, or 7 and brings the parent service
group online on an available system.
9 If there are no target systems available to fail over the parent service group that
has been taken offline, the parent service group remains offline.

Appendix C Supplemental Content


Copyright 2012 Symantec Corporation. All rights reserved.

C3

Copyright 2012 Symantec Corporation. All rights reserved.

252 C4

Veritas Cluster Server 6.0 for UNIX: Cluster Management


Copyright 2012 Symantec Corporation. All rights reserved.