68-Veritas Cluster Server 6.0 For UNIX Cluster Management

Veritas Cluster
Server 6.0 for UNIX:

Cluster Management
100-002685-E
COURSE DEVELOPERS
Bilge Gerrits
Steve Hoffer
Siobhan Seeger
Pete Toemmes
LEAD SUBJECT MATTER

EXPERTS
Graeme Gofton
Sean Nockles
Brad Willer
TECHNICAL
CONTRIBUTORS AND
REVIEWERS
Copyright 2012 Symantec Corporation. All rights reserved.
Geoff Bergren
Kelli Cameron
Tomer Gurantz
Anthony Herr
James Kenney
Gene Henriksen
Bob Lucas
Paul Johnston
Rod Pixley
Clifford Barcliff
Danny Yonkers
Antonio Antonucci
Satoko Saito
Feng Liu

Symantec, the Symantec Logo, and VERITAS are trademarks or
registered trademarks of Symantec Corporation or its affiliates in
the U.S. and other countries. Other names may be trademarks of
their respective owners.
THIS PUBLICATION IS PROVIDED AS IS AND ALL
EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS
AND WARRANTIES, INCLUDING ANY IMPLIED
WARRANTY OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE
DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH
DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.
SYMANTEC CORPORATION SHALL NOT BE LIABLE FOR
INCIDENTAL OR CONSEQUENTIAL DAMAGES IN
CONNECTION WITH THE FURNISHING, PERFORMANCE,
OR USE OF THIS PUBLICATION. THE INFORMATION
CONTAINED HEREIN IS SUBJECT TO CHANGE WITHOUT
NOTICE.
No part of the contents of this book may be reproduced or
transmitted in any form or by any means without the written
permission of the publisher.
Veritas Cluster Server 6.0 for UNIX: Cluster Management
Symantec Corporation
World Headquarters
350 Ellis Street
Mountain View, CA 94043
United States
http://www.symantec.com
Table of Contents
Course Introduction
Lesson 1: Service Group Dependencies
Common application relationships ............................................................... 1-3
Service group dependencies........................................................................ 1-7
Service group dependency examples .......................................................... 1-9
Configuring service group dependencies ................................................... 1-12
Alternative methods of controlling interactions........................................... 1-16
Lesson 2: Reconfiguring Cluster Membership
Adding a system to a cluster ........................................................................ 2-3
Merging clusters ........................................................................................... 2-6
Additional reconfiguration tasks ................................................................. 2-10
Lesson 3: Startup and Failover Policies
Startup rules and policies ............................................................................. 3-3
Failover rules and policies.......................................................................... 3-10
Limits and Prerequisites ............................................................................. 3-18
Modeling startup and failover policies ........................................................ 3-22
Lesson 4: Alternate Network Configurations
Multiple service groups with NIC resources ................................................. 4-3
Multiple public interfaces .............................................................................. 4-8
Lesson 5: High Availability in the Enterprise

Veritas Operations Manager ........................................................................ 5-3
Disaster recovery enhancements................................................................. 5-8
Virtualization support.................................................................................. 5-15
Appendix A: Labs
Lab 1: Service group dependencies............................................................ A-3
Lab 2: Merging clusters .............................................................................. A-13
Lab 3: Failover policies............................................................................... A-25
Lab 4: Creating a parallel network service group ....................................... A-37
Appendix B: Lab Solutions
Lab 1: Service group dependencies............................................................ B-3
Lab 2: Merging clusters .............................................................................. B-37
Lab 3: Failover policies............................................................................... B-65
Lab 4: Creating a parallel network service group ....................................... B-89
Appendix C: Supplemental Content
Service group dependenciesFailover process......................................... C-2
Table of Contents
i
ii

Course Introduction
Veritas Cluster Server curriculum path
The Veritas Cluster Server for UNIX curriculum is a series of courses that are
designed to provide a full range of expertise with Veritas Cluster Server (VCS)
high availability solutionsfrom design through implementation.
Veritas Cluster Server for UNIX: Install and Configure
This course covers installation and configuration of common VCS
environments, focusing on two-node clusters running application and database
services.
Veritas Cluster Server for UNIX: Manage and Administer
This course focuses on multinode VCS clusters and advanced topics related to
managing more complex cluster configurations.
The course includes two participant guides:
Veritas Cluster Server for UNIX: Example Application Configurations
This guide provides examples of common service group configurations.
The instructor may opt to present some or all of this material in class,
depending on time constraints and student interest.
Veritas Cluster Server for UNIX: Cluster Management
This guide provides detailed information about configuring and managing
more complex clusters.
Veritas Cluster Server eLearning Library
The eLearning Library is available with bundled training options and includes
content on advanced high availability and disaster recovery features.
12

Lab design for the course

The diagram shows a conceptual view of the cluster design used as an example
throughout this course and implemented in hands-on lab exercises.
Each aspect of the cluster configuration is described in greater detail where
applicable in course lessons.
The cluster consists of:

Four nodes
Several high availability services
Fibre connections to SAN shared storage from each node through a switch
Two Ethernet interfaces for the cluster interconnect
Ethernet connections to the public network
Additional complexity is added to the design to illustrate certain aspects of cluster

configuration in later lessons.
Course Introduction
13
Course overview
This training provides comprehensive instruction on the deployment of advanced

features of Veritas Cluster Server (VCS). The course focuses on multinode VCS
clusters and advanced topics related to more complex cluster configurations, such
as service group dependencies and workload management.
14

Lesson 1
Service Group Dependencies
10
12

Common application relationships

Multitier services
In most high availability environments, a collection of applications work together
to provide a service. The relationships between these application service groups
must be managed by VCS to ensure that startup, shutdown, and failover
procedures are coordinated properly.
The example in the slide shows a typical three-tier model where:

A Web server provides an interface to network clients.
The Web server relies on an application that processes client requests.
The application processes requests from the Web server and relies on the
database to manage information.
11
The inherent relationships require that the database is started and running first,
then the application, and finally the Web server.
In a cluster environment, you must also consider which systems should run which
applications, as shown in the following examples.
Lesson 1 Service Group Dependencies

13
Example 1: Online on the same system

In this example of a two-tier service, both the Web server and the application must
be online on the same system. For example, the services may need to use
interprocess communication (IPC) mechanisms to exchange data.
Furthermore, the application must come online first, followed by the Web server. If
the application faults and fails over to another cluster node, the Web server must
then be brought online on that same failover target system.
12
14

Example 2: Online on different systems

In this example of a two-tier service, both the database and the application must be
online, but they cannot run on the same system. For example, the combined
resource requirements of each application may exceed the capacity of the systems,
and you want to ensure that they run on separate systems.
In this scenario, the database must come online first, and then the application is
started only after the database is running. If the application faults and fails over to
another cluster node, the database can stay online on the original system, as long as
the application is not restarted on that same system.
13

15
Example 3: Offline on the same system

One example relationship is where you have a test version of an application and
want to ensure that it does not interfere with the production version. You want to
give the production application precedence over the test version for all operations,
including manual offline, online, switch, and failover.
The difference between this example and the previous example, is that in this case,
neither application requires the other to be online. The only requirement is that the
other application cannot be online on a system when one application is brought
online.
14
16

Service group dependencies

VCS provides service group dependencies to manage application relationship
requirements in multi-application environments.
Service group dependency definitions
15
There are four basic criteria for defining how services interact when using service
group dependencies.
A service group can require another group to be online or offline in order to
start and run.
You can determine the startup order for service groups by designating one
group the child and another a parent. Parent groups depend on child groups. If
service group B requires service group A to be online in order to start, then B is
the parent and A is the child.
You can specify where the groups must be online or offline.
For all online dependencies, the child group must be online in order for the
parent to start. A location of local, global, or remote determines where the
parent can come online relative to where the child is online.
For offline local, there is no requirement for either service group to start
first. The child group must be offline for the parent to come online.
Failover behavior of linked service groups is specified by designating the
relationship soft, firm, or hard.
Detailed information about how these dependencies impact service group
operations is provided in Veritas Cluster Server Administrators Guide.

17
Failover types
The dependency type determines failover behavior for linked service groups.
Soft dependency
A soft dependency means that VCS does not immediately take the parent offline if
the child group faults.
When both groups are online, the child group can be taken offline while the parent
is online. Likewise, the parent group can be taken offline while the child is online.
The parent remains online if the child group faults and cannot fail over. The child
remains online if the parent group faults.
Firm dependency
A firm dependency means that VCS imposes additional constraints on the parent
and child service groups. Specifically:
The parent group must be taken offline when the child group faults.
When the child is brought online on another system, the parent group is
brought online on a system determined by the location type (local, global,
remote).
If the parent group faults, the child continues to run.
16
Hard dependency
A hard dependency means that VCS takes the other service group offline when
either the child or parent group faults.
18

Service group dependency examples

Online local example
In an online local dependency, a child service group must be online on the same
system before the parent service group can come online on that system.
Online local soft
An online local soft dependency designates that the parent service group remains
online when the child service group faults. If the child service group fails over and
is brought online on another system, the parent service group is then migrated to
that same system.
Online local firm
17
An online local firm is similar to soft, except the parent service group is taken
offline when the child faults. If the child fails over and is restarted on another
system, the parent is then also started on that system
Online local hard
In an online local hard dependency, the child service group is taken offline if the
parent service group faults. Online local is the only dependency supporting the
hard type.

19
Online remote example

In an online remote dependency, a child service group must be online on a remote
system before the parent service group can come online on the local system.
Online remote soft
An online remote soft dependency designates that the parent service group remains
online when the child service group faults, as long as the child service group
chooses another system on which to fail over. If the child service group chooses to
fail over to the system where the parent was online, the parent service group is
migrated to any other available system.
Online remote firm
An online remote firm is similar to soft, except the parent service group is taken
offline when the child faults.
18
110

Offline local example
In an offline local dependency, the parent service group can be started only if the
child service group is offline on the local system. Similarly, the child can only be
started if the parent is offline on the local system. This prevents conflicting
applications from running on the same system. This is commonly used when you
have a test version of an application running on the failover target system for the
production version of the application.
19

111
Configuring service group dependencies

Service group dependency rules
You can use service group dependencies to implement parent/child relationships

between applications. Before using service group dependencies to implement the
relationships between multiple application services, review the rules governing
these dependencies:
The child service group always has priority over the parent group.
Service groups can have multiple parent service groups.
This means that an application service can have multiple other application
services depending on it.
A service group can have multiple child service groups.
This means that an application service can be dependent on one or more
application services.
A group dependency tree can be no more than five levels deep.
Service groups cannot have cyclical dependencies.
20
112

Creating service group dependencies

You can create service group dependencies from the command-line interface using
the hagrp command, using Veritas Operations Manager Web GUI, or using the
Cluster Manager Java GUI. To create a dependency, link the groups and specify
the relationship (dependency) type, indicating whether it is soft, firm, or hard.
If not specified, service group dependencies are firm by default.
To configure service group dependencies using the Cluster Manager Java GUI,
you can either right-click the parent service group and select Link to display the
Link Service Groups view that is shown on the slide, or you can use the Service
Group View.
Linking constraints
21
To link a parent and child group with soft dependency, it is not required that the
child group be online if the parent is online. However, if the child group is also
online, the parent and child may not be linked in such a way that their online states
conflict with the type of link between the parent and child.
To link a parent and child group with firm dependency, the parent group must be
offline, or the parent and child group must be online in such a way that their online
states do not conflict with the type of link between the parent and child.
Removing service group dependencies
When removing a dependency, you do not need to specify the type of dependency,
because only one dependency is allowed between two service groups.

113
Modeling service group dependencies
You can use the Simulator on Windows to model how different types of links
affect service group behavior. This enables you to fully understand the
implications and the effects of different dependency configurations before you
configure links in your production environment.
22
114

Online and offline of linked service groups

You can use the -propagate option to the hagrp command to simplify
bringing linked service groups online and offline. You can also combine
-propagate with the -any and -sys options for specifying where the service
groups are brought online.
Note that propagation is supported only with local service group dependencies
type.
23

115
Alternative methods of controlling interactions

Using triggers to control service group interactions
VCS provides several event triggers that can be used to enforce service group
relationships where dependencies cannot be configured, as described in the
previous example. These triggers are useful for managing relationships between
service groups:
VCS runs the preonline script before bringing a service group online.
The PreOnline trigger must be enabled for each applicable service group by
setting the PreOnline service group attribute. For example, to enable the
preonline trigger for groupa, type:
hagrp -modify GroupB PreOnline 1
The preonline script must start the service group with the -nopre option
to hagrp.
The postonline script is run after a service group is brought online.
The postoffline script is run after a service group is taken offline.
24
The postonline and postoffline triggers are enabled automatically if the

script is present in the $VCS_HOME/bin/triggers directory. Be sure to copy
triggers to all systems in the cluster. When present, these triggers apply to all
service groups.
Consider implementing triggers only after investigating whether VCS native
facilities can be used to configure the desired behavior. Triggers add complexity,
requiring programming skills as opposed to simply configuring VCS objects and
attributes.
116

Managing multitier applications
One of the key tenets of a multitier architecture is the operational independence of

each layereach tier can make its own decisions without relying on an external
brain. This applies to high availability, as well. Each tier needs to have HA of its
own it should be able to independently handle the resilience of the application
component running in that tier.
25

117
Virtual Business Services

Virtual Business Services provide continuous high availability and reduce
frequency and duration of service disruptions for multi-tier business applications
running on heterogeneous operating systems and virtualization technologies.
A Virtual Business Service represents the multi-tier application as a single
consolidated entity and builds on the high availability and disaster recovery
provided for the individual tiers by Symantec products such as Veritas Cluster
Server and Symantec ApplicationHA.
VBS is configured and managed using Veritas Operations Manager (VOM)
version 4.1 and later.
In the example shown in the slide, the database cluster is the lowest level tier and is
the most critical component in this service. Typically, you would find the database
to be shared across multiple services as well. If something happens in the database
tier, for example, an active server fails or the database runs into software issues on
one of the servers, VCS ensures the database is failed over to another node in the
cluster. Additionally, disaster recovery may be configured so that if the production
site goes down, VCS fails the database over to the DR site.
26
The middle tier is an application cluster running on several servers in parallel in

order to meet user demands. VCS provide local high availability in this tier and
manages the dependencies on the database.
Finally, the web server top tier is running on Vmware virtual machines. Symantec
ApplicationHA is managing high availability for the Apache Web servers running
on Windows.
118

Fault propagation between clusters

When a service group that is running on a cluster that is a member of a Virtual
Business Service faults, the fault is propagated to other clusters similarly to service
group dependencies.
A restart type dependency is specific to a VBS configuration. In the example
configuration shown in the slide, if oracle_sg faults, oracle_apps_sg ignores the
fault and continues to run. If or when the oracle_sg restarts and is in a running
state, oracle_apps_sg restarts.
In other words, the parent service group with a restart dependency on a child
service group takes no action when a child service group faults until the child is
restarted.
27
This type of dependency is used when only start or stop ordering is required, and
no fault policy action is needed.

119
28
Labs and solutions for this lesson are located on the following pages.
Lab 1: Service group dependencies, page A-3.
Lab 1: Service group dependencies, page B-3.
120

Lesson 2
Reconfiguring Cluster Membership
29
30
22

Adding a system to a cluster

Adding a system to a running VCS cluster
The objective of this task is to add a new system to a running VCS cluster with no
or minimal impact on application services. Ensure that the cluster configuration is
modified so that the application services can make use of the new system in the
cluster.
Assumptions
31
For illustration purposes, these assumptions are used in describing how to perform
this task:
The VCS cluster consists of two or more systems, all of which are up and
running.
There are multiple service groups configured in the cluster. All of the service
groups are online somewhere in the cluster.
The new system to be added to the cluster does not have any VCS software.
The new system has the same version of operating system and VERITAS
Storage Foundation as the systems in the cluster.
The new system may not have all the required application software.
The storage devices can be connected to all systems.
Lesson 2 Reconfiguring Cluster Membership

23
Procedure for adding a system to a running cluster
32
24

33

25
Merging clusters
Merging two running VCS clusters
The objective of this task is to merge two running VCS clusters with no or minimal
impact on application services. Also, ensure that the cluster configuration is
modified so that the application services can make use of the systems from both
clusters.
Assumptions
The following is a list of assumptions that you need to take into account while
planning a procedure for this task:
All the systems in both clusters are up and running.
There are multiple service groups configured in both clusters. All service
groups are online somewhere in the cluster.
All the systems have the same version of operating system and Veritas Storage
Foundation.
The clusters do not necessarily have the same application services software.
New application software can be installed on the systems to support
application services of the other cluster.
The storage devices can be connected to all systems.
The cluster interconnects of both clusters are isolated before the merger.
34
For this example, assume that two two-node clusters are merged into a single fournode cluster.
26

Procedure for merging two running clusters
35

27
36
28

37

29
Additional reconfiguration tasks

Reconfiguring triggers and service groups
The objective of this task is to update the cluster configuration so that the
application services can make use of the systems from both clusters.
Assumptions
The following is a list of assumptions that you need to take into account while
planning a procedure for this task:
Application resources have been prepared
Service group definitions have been merged
Agents have been installed
38
210

Procedure for updating triggers and service groups
39

211
40
212

41
Lab 2: Merging clusters, page A-13.
Lab 2: Merging clusters, page B-37.

213
42
214

Lesson 3
Startup and Failover Policies
43
44
32

Startup rules and policies
Rules for automatic service group startup
45
The following conditions must be satisfied for a service group to be automatically

started:
The service group AutoStart attribute must be set to the default value of 1. If
this attribute is changed to 0, VCS leaves the service group offline and waits
for an administrative command to be issued to bring the service group online.
In addition, the service group definition must have at least one system in its
AutoStartList attribute. Also, the service group cannot be frozen.
All of the systems in the service groups SystemList must be in the running
state so that the service group can be probed on all systems in SystemList.
If there are systems on which the service group can run that have not joined the
cluster yet, VCS autodisables the service group until it is probed on all the
systems.
And dependencies on child service groups must have been met before a service
group can be automatically started.
Lesson 3 Startup and Failover Policies

33
Startup system selection
The startup system for a service group is chosen as follows:

1 All systems included in the AutoStartList attribute are initial candidates for
service group startup.
2 Systems are culled from this initial list according to these criteria:
Frozen systems are eliminated.
Systems where the service group has a faulted status are eliminated.
Systems that do not meet the service group requirements are eliminated, as
described in detail later in the lesson.
3 The target system is then chosen from this list based on the startup policy
defined for the service group.
46
34

Automatic startup policies

You can set the AutoStartPolicy attribute of a service group to one of these three
values:
Order: Systems are chosen in the order in which they are defined in the
AutoStartList attribute. This is the default policy for every service group.
Priority: Of the systems listed in the AutoStartList attribute, the system with
the lowest priority number in SystemList is selected.
Load: The system with the highest available capacity is selected.
The autostart policies are described in more detail in the following pages.
To configure the AutoStartPolicy attribute of a service group, execute:
hagrp -modify group AutoStartPolicy policy
47
where possible values for policy are Order, Priority, and Load. You can
also set this attribute using Veritas Operations Manager or the Cluster Manager
Java GUI.
Note: The configuration must be open to change service group attributes.

35
AutoStartPolicy=Order
When the AutoStartPolicy attribute of a service group is set to the default value of
Order, the first system available in AutoStartList is selected to bring the service
group online. The priority numbers in SystemList are ignored.
In the example shown on the slide, the A service group is brought online on s1,
although it is the system with the highest priority number in SystemList. Similarly,
the B service group is brought online on s2, and the C service group is brought
online on s3 because these are the first systems listed in the AutoStartList
attributes of the corresponding service groups.
Note: Because Order is the default value for the AutoStartPolicy attribute, it is
not listed in the service group definitions in the main.cf file.
48
36

AutoStartPolicy=Priority
When the AutoStartPolicy attribute of a service group is set to Priority, the system
with the lowest priority number in the SystemList that also appears in the
AutoStartList is selected as the target system during start-up. In this case, the order
of systems in the AutoStartList is ignored.
The same example service groups are now modified to use the Priority
AutoStartPolicy, as shown on the slide. In this example, the A service group is
brought online on s3, which has the lowest priority number in SystemList even
though it is listed as the last system in AutoStartList. Similarly, the B service group
is brought online on s1 (with priority number 0), and the C service group is
brought online on s1 (with priority number 1).
49
Notice how the startup systems have changed for the service groups by changing
the AutoStartPolicy attribute, although the SystemList and AutoStartList attributes
are the same for these two examples.

37
AutoStartPolicy=Load
When AutoStartPolicy is set to Load, VCS determines the target system based on
the existing workload of each system listed in the AutoStartList attribute and the
load that is added by the service group.
These attributes control load-based startup:
Capacity is a user-defined system attribute that contains a value representing
the total amount of load that the system can handle.
Load is a user-defined service group attribute that defines the amount of
capacity required to run the service group.
AvailableCapacity is a system attribute maintained by VCS that quantifies the
remaining available system load.
In the example displayed on the slide, three servers have Capacity set to 300, 200,
and 100, respectively. The s2 system is selected as the target system for starting C
because it has the highest AvailableCapacity value of 200 after A and B are started
on s1.
50
Determining Load and Capacity

You must determine a value for Load for each service group. This value is based
on how much of the system capacity is required to run the application service that
is managed by the service group.
38

When a service group is brought online, the value of its Load attribute is subtracted
from the system Capacity value, and AvailableCapacity is updated to reflect the
difference. Both the Capacity attribute of a system and the Load attribute of a
service group are static user-defined attributes based on your design criteria.
51

39
Failover rules and policies

Rules for automatic service group failover
The following conditions must be satisfied for a service group to be automatically

failed over after a fault:
The service group must contain a critical resource, and that resource must fault
or be a parent of a faulted resource.
The service group AutoFailOver attribute must be set to the default value of 1.
If this attribute is changed to 0, VCS leaves the service group offline and waits
for an administrative command to be issued to bring the service group online.
The ManageFaults attribute must be set to All, the default setting.
The service group cannot be frozen.
At least one of the systems in the service groups SystemList attribute must be
in a running state.
52
310

Failover system selection
The failover system for the service group is chosen as follows:

1 A subset of systems listed in the SystemList attribute is created first.
2 Systems that do not meet the service group requirements are eliminated,
including:
Frozen systems
Systems where the service group has a faulted status are eliminated from
the list.
Systems that do not meet the service group requirements are eliminated, as
described in detail later in the lesson.
3 The target system is chosen from this list based on the failover policy defined
for the service group.
53

311
Failover policies
VCS supports a variety of policies that determine how a system is selected when
service groups must migrate due to faults. The policy is configured by setting the
FailOverPolicy attribute to one of these values:
Priority: The system with the lowest priority number is preferred for failover
(default).
RoundRobin: The system with the least number of active service groups is
selected for failover.
Load: The system with the highest value of the AvailableCapacity system
attribute is selected for failover.
The policies are described in more detail in the following pages.
54
312

FailOverPolicy=Priority
When FailOverPolicy is set to Priority, VCS selects the system with the lowest
assigned value from the SystemList attribute.
For example, the C service group has three systems configured in the SystemList
attribute and the same order for AutoStartList values:
SystemList = {s3=0, s1=1, s2=2}
AutoStartList = {s3, s1, s2}
The C service group is initially started on s3 because it is the first system in

AutoStartList. If C faults on s3, VCS selects s1 as the failover target because it has
the lowest priority value for the remaining available systems.
55
Priority policy is the default behavior and is ideal for simple two-node clusters or
small clusters with few service groups.

313
FailOverPolicy=RoundRobin
The RoundRobin policy selects the system running the fewest service groups as
the failover target.
The RoundRobin policy is ideal for large clusters running many service groups
with essentially the same server load characteristics (for example, similar
databases or applications).
Take into account these properties of the RoundRobin policy:

Only systems listed in the SystemList attribute for the service group are
considered when VCS selects a failover target for all failover policies,
including RoundRobin.
A service group that is in the process of being brought online is not considered
an active service group until it is completely online.
56
Ties are determined by the order of systems in the SystemList attribute. For
example, if two failover target systems have the same number of service groups
running, the system listed first in the SystemList attribute is selected for failover.
314

FailOverPolicy=Load
When FailOverPolicy is set to Load, VCS determines the target system based on
the existing workload of each system listed in the SystemList attribute and the load
that is added by the service group.
These attributes control load-based failover:

Capacity is a system attribute that contains a value representing the total
amount of load that the system can handle.
Load is a service group attribute that defines the amount of capacity required to
run the service group.
AvailableCapacity is a system attribute maintained by VCS that quantifies the
remaining available system load.
57
In the example displayed in the slide, the three servers that remain running after
the s3 system fails have Capacity set to 300, 200, and 100. Each service group has
a fixed load defined by the user, which is subtracted from the system capacity to
find the AvailableCapacity value of a system.
When failover occurs, VCS checks the value of AvailableCapacity on each
potential targeteach system in the SystemList attribute for the service group
and starts the service group on the system with the highest value.
Note: In the event that no system has a high enough AvailableCapacity value for a
service group load, the service group still fails over to the system with the highest
value for AvailableCapacity, even if the resulting AvailableCapacity value is zero
or a negative number.

315
Configuring Load and Capacity

You can use VOM, the Cluster Manager Java GUI, or the command-line interface
to set the Capacity system attribute and the Load service group attribute.
To set Capacity from the command-line interface, use the hasys -modify
command as shown in this example:
hasys -modify s1 Capacity 300
To set Load from the CLI, use the hagrp -modify command as shown in this
example:
hagrp -modify dbsg Load 75
58
316

The loadwarning trigger
You can configure the loadwarning trigger to provide notification that a system
has sustained a predetermined load level for a specified period of time.
59
To configure the loadwarning trigger:

1 Create a loadwarning script in the /opt/VRTSvcs/bin/triggers
directory. You can copy the sample trigger script from /opt/VRTSvcs/
bin/sample_triggers as a starting point, and then modify it according
to your requirements.
2 Set the loadwarning attributes for the system:
Capacity: Load capacity for the system
LoadWarningLevel: The level at which load has reached a critical limit;
expressed as a percentage of the Capacity attribute
Default is 80 percent
LoadTimeThreshold: Length of time, in seconds, that a system must
remain at, or above, LoadWarningLevel before the trigger is run
Default is 600 seconds

317
Limits and Prerequisites

Startup example
VCS enables you to define the available resources on each system and the
corresponding requirements for these resources for each service group. Shared
memory, semaphores, and the number of processors or application instances are all
examples of resources that can be defined on a system.
Note: The resources that you define are arbitrarythey do not need to
correspond to physical or software resources. You then define the
corresponding prerequisites for a service group to come online on a system.
In a multinode, multiapplication services environment, VCS keeps track of the

available resources on a system by subtracting the resources already in use by
service groups online on each system from the maximum capacity for that
resource. When a new service group is brought online, VCS checks these available
resources against service group prerequisites; the service group cannot be brought
online on a system that does not have enough available resources to support the
application services.
60
Limits
The Limits system attribute is used to define the resources and the corresponding
capacity of each system for that resource. You can use any keyword for a resource
as long as you use the same keyword on all systems and service groups.
318

The example values displayed in the slide are set as follows:

On the first system, the Limits attribute setting in main.cf is:
Limits = { DBs=2 }
On the second two systems, the Limits attribute setting in main.cf is:
Limits = { DBs=1 }
On the fourth system, the Limits attribute setting in main.cf is:
Limits = { DBs=0 }
This value of DBs in this example is used to control how many Oracle service
groups can run on a system.
Service group Prerequisites
Prerequisites is a service group attribute that defines the set of resources needed to
run the service group. These values correspond to the Limits system attribute and
are set by the Prerequisites service group attribute. This main.cf configuration
corresponds to the E service group in the diagram:
Prerequisites = { DBs=1 }
CurrentLimits
CurrentLimits is an attribute maintained by VCS that contains the value of the
remaining available resources for a system. For example, if the limit for DBs is 2
and the A service group is online with a DBs prerequisite of 1, the CurrentLimits
setting for DBs is 1:
CurrentLimits = { DBs=1 }
Selecting a target system
Prerequisites are used to determine a subset of eligible systems on which a service

group can be started during failover or startup. When a list of eligible systems is
created, HAD then follows the configured policy for autostart or failover.
61
Note: A value of 0 is assumed for systems that do not have some or all of the
resources defined in their Limits attribute. Similarly, a value of 0 is
assumed for service groups that do not have some or all of the resources
defined in their Prerequisites attribute.

319
Failover example
The configuration in the slide shows how a failover target system is selected for
the C service group when the s3 system faults. Because C has a prerequisite of 1
for DBs and there are no systems with a high enough CurrentLimits value for DBs
to support running the group, C stays offline.
System Limits are hard values, meaning that if a system does not meet the
requirements specified in the Prerequisites attribute for a service group, the service
group cannot be started on that system.
Contrast this with the Load and Capacity, which are soft limits. If the
AvailableCapacity value is not high enough to satisfy the Load requirement for a
service group, VCS still fails over the service group, even if the AvailableCapacity
becomes a negative value.
Therefore, using Limits and Prerequisites may be a better method for controlling
service group startup and failover in cases where hard limits must be enforced. An
example case is an environment where licensing restrictions prevent two instances
of an application or database from running on a system.
62
320

Configuring Limits and Prerequisites

You can use the VCS GUI or command-line interface to set the Limits system
attribute and the Prerequisites service group attribute.
To set Limits from the command-line interface, use the hasys -modify
command as shown in the following example:
hasys -modify s2 Limits DBs 2
To set Prerequisites from the CLI, use the hagrp -modify command as shown
in this example:
hagrp -modify DBSG Prerequisites DBs 1
63
Notes:
To be able to set these attributes, open the VCS configuration to enable read/
write mode and ensure that the service groups that are already online on a
system do not violate the restrictions.
The order that the resources are defined within the Limits or Prerequisites
attributes is not important.

321
Modeling startup and failover policies

Using the Simulator
The VCS Simulator for Windows is a good tool for modeling the behavior that you
require before making changes to the running configuration. This enables you to
fully understand the implications and the effects of different workload
management configurations.
64
322

65
Lab 3: Failover policies, page A-25.
Lab 3: Failover policies, page B-65.
323
66
324

Lesson 4
Alternate Network Configurations
67
68
42

Multiple service groups with NIC resources

NIC resources in multiple service groups
Many clusters have multiple service groups, each containing NIC resources that
monitor the same physical network interface. In this case, VCS monitors the same
network interfacesay e1000g0 on Solarismany times, creating unnecessary
overhead and network traffic.
In addition to the overhead of many monitor cycles for the same resource, a
disadvantage of this configuration is the effect of changes in NIC hardware. If you
must change the network interface (for example, in the event the interface fails),
you must change the Device attribute for each NIC resource monitoring that
interface.
69
Lesson 4 Alternate Network Configurations

43
Using Proxy resources

You can use a Proxy resource to allow multiple service groups to monitor the same
network interfaces. This reduces the network traffic that results from having
multiple NIC resources in different service groups monitor the same interface.
The Proxy resource mirrors the status of another resource in a different service
group. The required attribute, TargetResName, is the name of the resource whose
status is reflected by the Proxy resource.
TargetSysName is an optional attribute specifies the name of the system on which

the target resource status is monitored. If no system is specified, the local system is
used as the target system.
70
44

Parallel network service groups

Parallel service groups run on multiple systems simultaneously. A network service
group can be configured as a parallel service group because the NIC resource is
persistent and can be online on multiple systems. With this configuration, all the
other service groups are configured with a Proxy to the NIC resource on their local
system.
A parallel service group is managed like any other service group in VCS, except
that it cannot be switched. The group is only started on a system listed in the
AutoStartList and the SystemList attributes. A parallel service group can also fail
over, in effect, if the service group faults on a system and there is an available
system (listed in the SystemList attribute) that is not already running the service
group. This is accomplished by VCS starting up the parallel group on the target
system.
71
To create a new parallel service group in a running cluster, set the Parallel attribute
to 1 (true) and then add resources.
You cannot change an existing failover service group that contains resources to a
parallel service group except by using the offline configuration procedure to edit
the main.cf file and add Parallel = 1 to the service group definition.

45
Using Phantom resources
The example network service group also requires a Phantom resource in order to
ensure that the service group status is displayed properly. A service group shows
an online status only when all of its nonpersistent resources are online. Therefore,
if a service group has only persistent resources, VCS considers the group offline,
even if the persistent resources are running properly. When a Phantom resource is
added, the status of the service group is shown as online.
72
46

Localizing a NIC resource

An attribute whose value applies to all systems is global in scope. An attribute
whose value applies on a per-system basis is local in scope. By default, all
attributes are global. Some attributes can be localized to enable you to specify
different values for different systems.
In the example displayed in the slide, the Device attribute for the NIC resource is
localized to enable you to specify a different interface for each system.
After creating the resource, you can localize attribute values using the hares
command, a GUI, or an offline configuration method. For example, when using the
CLI, type:
hares -local netnic Device
hares -modify netnic Device eth0 -sys s1
73
hares -modify netnic Device eth4 -sys s2
Any attribute can be localized. Network-related resources are common examples

for local attributes.

47
Multiple public interfaces

Trunked or bonded interfaces
Most UNIX operating systems have the capability to create a logical network
interface that represents a collection of physical network interfaces. This is often
referred to as port trunking or interface bonding.
For example, on Linux, you can use the bond driver to treat multiple network
interfaces as one logical interface. In this case, you can use the VCS NIC agent to
monitor a bond-type interface and an IP resource to bring up a virtual IP address
on the virtual bond interface.
74
48

The following table shows an example configuration where eth0 and eth1 are
controlled by the bond0 interface.
ifcfg-eth0
ifcfg-eth1
ifcfg-bond0
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=no
ne
USERCTL=no
PEERDNS=no
TYPE=Ethernet
MASTER=bond0
SLAVE=yes
DEVICE=eth1
ONBOOT=yes
BOOTPROTO=no
ne
USERCTL=no
PEERDNS=no
TYPE=Ethernet
MASTER=bond0
SLAVE=yes
DEVICE=bond0
ONBOOT=yes
USERCTL=no
TYPE=Ethernet
MTU=""
NETMASK=255.255.0.0
BOOTPROTO=none
IPADDR=10.10.2.20
BROADCAST=10.10.255.255
GATEWAY=10.10.2.1
NETWORK=10.10.2.0
The corresponding sample /etc/modules.conf file contains:

alias eth0 e1000
alias eth1 e1000
alias bond0 bonding
options bond0 miimom=100 mode=2
The following VCS resource definitions are contained in a service group to

monitor the bond0 interface and manage the virtual IP address.
IP webip (
Device = bond0
Address = "10.10.10.21"
)
NIC bond0nic (
75
Device = bond0
)
An advantage of using the Linux bond interface is that you can aggregate
interfaces to increase bandwidth.

49
Local network interface failover

With the availability of inexpensive network adapters, it is common to have many
network interfaces on each system. By allocating more than one network interface
to a service group, you can potentially avoid failover of the entire service group if
the interface fails. By moving the IP address on the failed interface to another
interface on the local system, you can eliminate or minimize downtime.
VCS provides this type of local failover with the MultiNICB and IPMultiNICB
resources for the Solaris, AIX, and HP-UX platforms. All platforms also support
legacy resource types, MultiNICA and IPMultiNIC, for local interface failover.
Advantages of local interface failover
Local interface failover can drastically reduce service interruptions to the clients.
Some applications have time-consuming shutdown and startup processes that
result in substantial downtime when the application fails over from one system to
another.
76
Failover between local interfaces can be completely transparent to users for some
applications.
Using multiple networks also makes it possible to eliminate any switch or hub
failures causing service group failover as long as the multiple interfaces on the
system are connected to separate hubs or switches.
410

Network resources overview

The MultiNICA agent is capable of monitoring multiple network interfaces, and if
one of these interfaces faults, VCS fails over the IP address defined by the
IPMultiNIC resource to the next available public network adapter.
The IPMultiNIC and MultiNICA resources provide essentially the same service as
the IP and NIC resources, but these resources monitor multiple interfaces instead
of a single interface. The dependency between these resources is the same as the
dependency between IP and NIC resources.
77
The MultiNICB and IPMultiNICB agents provide similar functionality to the

MultiNICA and IPMultiNIC agents with many additional features, such as:
Support for faster failover
Support for active/active interfaces
Support for failback
On Solaris only, these additional features are provided:
Support for the Solaris IPMP daemon
Support for trunked network interfaces on Solaris
See the Veritas Cluster Server Bundled Agents Reference Guide for your platform
for complete details about configuring these resource types.

411
Comparing MultiNICA and MultiNICB

Advantages of using MultiNICA and IPMultiNIC
Physical interfaces can be plumbed as needed by the agent, supporting an
active/passive configuration.
MultiNICA requires only one base IP address for the set of interfaces under its
control. This address can also be used as the administrative IP address for the
system.
MultiNICA does not require all interfaces to be part of a single IP subnet.
Advantages of using MultiNICB and IPMultiNICB

All interfaces under a particular MultiNICB resource are always configured
and have test IP addresses to speed failover.
MultiNICB failover is many times faster than that of MultiNICA.
Support for single and multiple interfaces eliminates the need for separate pairs
of NIC and IP, or MultiNICA and IPMultiNIC, for these interfaces.
MultiNICB and IPMultiNICB support failback of IP addresses.
MultiNICB and IPMultiNICB support manual movement of IP addresses
between working interfaces under the same MultiNICB resource without
changing the VCS configuration or disabling resources.
78
On Solaris, MultiNICB and IPMultiNICB support IPMP, interface groups, and

trunked ge and qfe interfaces.
412

79
Lab 4: Creating a parallel network service group, page A-37.
Lab 4: Creating a parallel network service group, page B-89.

413
80
414

Lesson 5
High Availability in the Enterprise
81
82
52

Veritas Operations Manager

Veritas Operations Manager provides a single, centralized management console for
the Veritas Storage Foundation and High Availability products. You can use it to
monitor, visualize, and manage storage and cluster resources, and generate reports.
Veritas Operations Manager enables administrators to centrally manage diverse
data center environments.
You can also use Veritas Operations Manager to manage hosts that do not have
Storage Foundation and High Availability products installed.
Simplified cluster management
Veritas Operations Manager (VOM) enables you to perform administrative tasks

and analysis for all clusters in an environment, across physical locations, virtual
environments, and operating system platforms.
83
You can also use Veritas Operations Manager to visualize the state of multi-tier
applications and all subcomponents components and to start or stop the entire
logical application in an ordered fashion. Virtual Business Services provides this
capability of associating service groups across clusters and managing the
relationships in a high availability and disaster recovery environment.
Lesson 5 High Availability in the Enterprise

53
VOM functional overview

VOM collects, stores, consolidates, and presents state information for all Storage
Foundation HA resources in a variety of views.
VOM uses a Web browser-based user interface to present information in a layered
form, starting with the concise dashboard view shown in the slide. The dashboard
view, presented upon login, contains highly summarized information about all
managed storage, server, and application resources at a glance.
Graphics and color coding are both used to draw attention to resources that require
attention because they are either faulted (not functioning) or are at risk (configured
to be fault-tolerant, but having one or more failed components that could result in
faulting if a further component failure occurs).
84
54

Distributed commands in a VOM domain

VOM provides centralized cluster visualization, monitoring, and control of the
entire distributed environment. The components of a VOM domainthe scope of
a Storage Foundation High Availability management environmentinclude:
Management serversstand-alone peer servers or clusters
One or more Web consoles
Managed hosts: VCS 4.x or 5.x nodes running a connector agent
VOM increases administrator efficiencies and reduces configuration errors by
enabling centralized deployment of configuration changes, such as by applying the
same change to many clusters with one command.
In the example shown in the slide, a new user account, OraOper, is given service
group operator privileges for all clusters containing Oracle service groups.
85

55
License deployment policies and reporting

When using keyless licensing, all cluster nodes must become managed hosts in a
VOM environment. The licenses can then be tracked using the license deployment
reporting feature in VOM.
A 60 day grace period is allowed to set up the VOM environment and configure
SFHA systems as managed hosts. After 60 days, warning messages are issued
every four hours until the system is configured as a managed host.
Note: Although you can install VCS without Storage Foundation and select
keyless licensing, the warning mechanism is only disabled when you install
Storage Foundation and add the host to the VOM domain.
You can use VOM license deployment policies and reports to ensure all systems in
the data center meet licensing requirements.
86
56

VOM solutions
Veritas Operations Manager solutions are independent and optional feature packs
that you can download and use to enhance the functionality of Veritas Operations
Manager. These solutions are grouped into the following categories:
Add-on
Package
Patch
Hotfix
87
A core group of add-ons are bundled in VOM solutions repository. Other add-ons
must be downloaded from sort.symantec.com and then uploaded to your
VOM repository. Any solutions available in the repository can be installed on the
Management Server, managed hosts, or on both. To deploy solutions you must
have domain administrative privileges.
VOM Deployment Management enables you to install a solution on selected hosts,
or a selected platform (for example AIX). When you run the installation process, a
deployment request is sent. You can view that deployment request in the
Deployment Requests page.
After a solution is installed, you must enable the solution to use the management
tools within the add-on.

57
Disaster recovery enhancements

Testing disaster recovery
Disaster recovery fire drill is a feature provided with the Global Cluster Option for
VCS that enables you to anticipate and address configuration problems at the DR
site prior to encountering an actual disaster.
A fire drill is implemented by configuring a clone service group with specialpurpose resources. These fire drill service groups are modified to use copies of the
primary sites data so that the applications can be started and tested without
affecting the production service groups.
When you bring a fire drill service group online, the resources:
1 Create a snapshot of the replicated volume.
2 Mount the snapshot file system.
3 Start the application or database.
88
VCS logs any errors to enable you to identify configuration problems in your DR
site, as shown in the following examples. Starting with VCS 5.0, EMC SRDF and
Hitachi Truecopy agents are enhanced to take advantage of device tagging in
Volume Manager. This enables hardware snapshots to be imported and used for a
fire drill capability without any scripting or tasks required to be used outside of
VCS.
Note: The Percentage in slide based on Symantec Disaster Recovery Study
published 22 November 2010.
58

Successful fire drill example

The slide shows example VCS Engine log file entries for a successful fire drill test
of an Oracle service group named oragrp_fd. In this case, all resources came
online, and the database is running using data files on the snapshot volumes.
You can configure the VCS Management Console to run regularly scheduled fire
drills to ensure you are continuously validating your DR environment.
89

59
DR configuration problem example

One common DR configuration problem is storage configuration driftan object
is changed at the primary site and that change is not propagated to the secondary
site.
In this example, a database administrator adds a volume to a disk group on the

primary site, and then creates a new tablespace. This DBA is not aware that the
volume needs to be added to the replication configuration, so the tablespace is not
replicated to the secondary site.
90
510

Failed fire drill example

This slide shows VCS engine log entries indicating the problem with the fire drill
caused by data created on the primary site but not replicated to the secondary site.
The log includes the Oracle startup output resulting from bringing the Oracle (Orafd3_fd) resource online on the secondary site. The Oracle error output identifies a
problem with the /oradata2/tb2.dbf file, which is not present at the
secondary site.
Other examples of configuration drift that cause problems for site migration
include:
Expired licenses
Operating system and application patch version mismatches
Configuration files not synchronized
91

511
Managing replication
You can use VOM to simplify configuration of storage and replication objects used
in replicated data clusters (RDCs) and global clusters with both 4.x- and 5.x-based
systems.
To generate reports about the details of data transfer between source and target
replication hosts, download the Add-on for Veritas Volume Replicator Bandwidth
Reporting from sort.symantec.com.
After you upload the add-on to the VOM repository, you can deploy the solution to
the management server and all hosts with replication solutions.
92
512

Site-wide notification and migration

VOM simplifies management of disaster recovery, providing policy-based
monitoring of critical events, with automatic notification of administrators when
faults occur.
The Management Console also provides valuable diagnostic tools, such as:
Uptime analysis
Configuration analysis
Agent inventory
Additional predefined reports that can be scheduled and saved
Perhaps most important in a DR environment, VOM enables you to perform an

entire site migration and recovery with a single action.
93

513
Disaster Recovery Advisor

Veritas Disaster Recovery Advisor (DRA) monitors high availability and disaster
recovery configurations to ensure data center recoverability. By scanning systems
across the data center to ensure that existing HA/DR plans are applied seamless,
Disaster Recovery Advisor helps to limit the risk of infrastructure and application
downtime.
DRA scans storage, servers, databases, clusters, and replication infrastructures

using a knowledge base containing over 4,600 risk signatures. When gaps are
discovered, the software alerts the system administrator so the issues can be
resolved before business operations are impacted. As an agentless solution, the
implementation of Disaster Recovery Advisor is unobtrusive and non-disruptive,
operating in read-only mode. Monitoring and management is simple with status
dashboards that provide detailed insight into the data center's environment.
94
514

Virtualization support
Server virtualization approaches
95
Most customers are using more than one virtualization methodology within their
IT infrastructure. Each major UNIX operating system vendor has a virtualization
technology, each with unique terms such as control domain, host, guest, and virtual
I/O. The slide illustrates four virtualization approaches and examples of platforms
that utilize that type of virtualization.
Hardware partitioning subdivides a server, allocating resources to partitions,
each running an operating system. Sun Domains and HP nPartitions are
example solutions using this technology.
Bare metal hypervisor consists of a hypervisor process running directly on the
server hardware, which enables creating and maintaining virtual machines.
VMware ESX server is an example of this technology.
Hosted hypervisor has hosted operating system running on the server and a
hypervisor running within the OS. I/O flows from the virtual machines,
through the hypervisor, to the local operating system drivers, and then to the
server hardware. Linux KVM and Solaris LDOM are examples using this
technology.
Finally, operating system container technology allows for the partitioning of
the OS into containers to control access to the applications running within the
containers.

515
Symantec solutions for virtualization
The slide shows how Storage Foundation and VCS works in each of the
virtualization environments.
Hardware partitioning
Storage Foundation and VCS run within partitions and generally partitions are
not migrated between physical machines.
Bare metal hypervisor
Storage Foundation and VCS run within the virtual machines. The VMs may
be migrated to different physical servers by virtualization HA technology.
Hosted hypervisor
Storage Foundation and VCS can run on both the host operating system and
virtual machines. VCS can be used to make VMs highly available, however
clusters cannot contain both VMs and the physical machine hosting VMs.
Operating system containers
Storage Foundation and VCS can run on the host operating system and can
control the startup of the security container as well as the applications within
them.
96
516

Symantec across physical and virtual environments

Storage Foundation High Availability 6.0 can be deployed in a number of
configurations in both physical and virtual environments. As shown in the
diagram, different SFHA components can be deployed on hosts or on virtual
machines running supported guest operating systems.
Veritas Operations Manager can be used to manage storage and clustering in both
physical and virtual environments.
97

517
Building resiliant private clouds

The Storage Foundation High Availability solutions from Symantec enable IT
organizations to build resilient private clouds by transforming their existing
infrastructure. Organizations can manage entire business services end-to-end with
built in resiliency, even if the business service is run across multiple virtualization
technologies, operating systems, and storage platforms. All products in the
portfolio are tightly integrated to help IT organizations move confidently to a
private cloud architecture.
SFHA extends the core concepts of virtualization, by pooling information

resources even across heterogeneous infrastructure, and making them available ondemand for infrastructure HA or management solutions to take appropriate action
to maintain service continuity.
98
518

Summary
This lesson summarized some of the features of Veritas Cluster Server, Storage
Foundation, and Veritas Operations Manager that specifically enhance high
availability and disaster recovery solutions in enterprise data centers.
99

519
100 520

Appendix A
Labs
101
102 A2

Lab 1: Service group dependencies

In this lab, you will examine a variety of service group dependency configurations
and behaviors. Online and offline service group propagation is also examined.
This lab contains the following exercises:
Exercise 1: Checking lab prerequisites
A verification that the virtual machines needed for this lab are powered on and
functioning is performed.
Exercise 2: Service group and resource orientation

The service groups and resources in the east cluster are examined.
103
Exercise 3: Service group dependencies scenario 1

A typical service group dependency is configured so that one service group will
have multiple parents causing one service group to have preference being online
on startup and failover over another service group.
A typical service group dependency is configured with a three level dependency
causing one service group to have preference being online on startup and failover
over another service group.
A typical multi-child service group dependency is configured so that one service
group has two child service group dependencies causing a service group to have
preference being online on startup and failover.

A3

In this exercise, you verify that the virtual machines needed for this lab are
powered on and functioning, that you are logged in using the proper account and
that any needed terminal windows are opened.
1
Verify that the following virtual machines shown are powered on, that you are
logged using the indicated account and that the indicated terminal windows are
opened.
mgt
There is no need to log into this virtual machine at this time.
sym3
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym4
Account: root
Password: veritas
Note: The terminal windows are referred to as hostname:terminal#

throughout the labs. For example: sym1:terminal1, sym1:terminal2,
sym2:terminal1, and sym2:terminal2. It is not necessary to label the
terminal windows and you may decide which is terminal1 and which is
terminal2.
104 A4
If you have machines running that are not used in this lab, shut down the
operating system and power the machines off.
Note: If you are completing the lab exercises in order, this will require you to
shutdown cps, sym1 and sym2, and power on and log into sym3 and
sym4.
Veritas Cluster Server 5.1 for UNIX: Install and Configure


In this exercise, you examine the service groups and resources in the east cluster.
sym3
Note: The appsg and testappsg service groups were added in an early Install and
Configure lab using the Veritas Cluster Manager Java Console. If those
lab exercises were skipped, then consult with your instructor on how to
copy in a replacement main.cf file located at:
/student/labs/vcs/vcs60/maincf/IC/
main.cf.east.lab05
1
From sym3:terminal1, perform a summary status on the cluster by:
Listing the state of each service group.

Verifying that all service groups are online on the sym3 system and
switching any that need to be switched.
Verifying that there are no service group dependencies.
105
Display the state of all resources by listing the resources for each service group
and the dependencies between each resource in each service group.
Determine if all resources in all service groups other than the ClusterService
service group are set to critical. If any resources are set to non critical then
modify them. Open, save and close the cluster configuration as appropriate.
Confirm that the FaultPropagation and ManageFaults service group

attributes are set to 1 and ALL respectively for each service group. Modify to
their default value if necessary. Open, save and close the cluster configuration
as appropriate.
Open the cluster configuration for update.
From sym3:terminal2, run the hastatus command in order to observe

cluster changes as they happen in subsequent exercises.

A5

In this exercise, you configure a typical service group dependency so that one
service group will have multiple parents causing one service group to have
preference being online on startup and failover over another service group.
sym3
Note: The service groups in this exercise will come online and go offline very
quickly. The hagrp -wait commands, while not necessary, have been
included to be consistent with other lab exercises.
1
From sym3:terminal1, take all of the service groups except the

ClusterService service group offline on all cluster systems.
Create and confirm dependencies between the appsg and dbsg and between
the websg and dbsg service groups as indicated below. Save but do not close
the VCS configuration.
appsg depends on dbsg (online local firm)

websg depends on dbsg (online local firm)
106 A6
Use a single hagrp command with the propagate option to bring the dbsg,
appsg, and websg service groups online on sym3. Can all three service groups
be brought online using just one command? If not, then ensure that all three
service groups are brought online on sym3.
Use a single hagrp command to take the dbsg, appsg, and websg service
groups offline on sym3. Can all three service groups be brought offline using
just one command? If not, then ensure that all three service groups are brought
offline on sym3.

Using a single hagrp command with the propagate option, bring the dbsg
and websg service groups online on sym3. Attempt to bring the appsg service
group online on sym4. Were you successful? If not successful, then bring the
appsg service group online on sym3.
Fault the appsg service group by removing the /var/tmp/appip file

followed by a probe of the appip resource on sym3. Do any service groups
failover?
Clear the appip resource fault and bring the appsg service group back online
on sym3.
Fault the dbsg service group by removing the /var/tmp/dbip file followed
by a probe of the dbip resource on sym3. Do any service groups failover?
Clear the dbip resource fault and take the dbsg, websg, appsg, and testappsg
service groups offline on any system they may be online.
10 Link the testappsg service group as a parent of the appsg service group with
an offline local dependency. Save, but do not close the VCS configuration.
107
Note: This will configure a preference for the appsg service group running on
a system over the testappsg service group and force the testappsg to
switch upon a successful failover of the appsg service group.

A7
11 Bring the dgsg, appsg, and websg service groups online on sym3. Attempt to
bring the testappsg service group online on sym3. Were you successful? If
not, attempt to bring the testappsg service group online on sym4.
12 Fault the dbsg service group by removing the /var/tmp/dbip file followed
13 Take the testappsg service group offline on sym3 and clear the dbip resource
fault. Then, use a single command to take the dbsg, websg, appsg, and
testappsg service groups offline on any system where they may be online.
14 Unlink the service group dependencies between the testappsg and appsg
service groups and between the dbsg, websg and appsg service groups. Save,
but do not close the VCS configuration.
108 A8


In this exercise, you configure a typical service group dependency with a three
level dependency causing one service group to have preference being online on
startup and failover over another service group.
sym3
From sym3:terminal1, create dependencies between the appsg and dbsg

service groups, between the websg and appsg service groups and between the
testappsg and appsg service groups as indicated below. Save but do not close
websg depends on appsg (online local firm)
testappsg depends on appsg (offline local)
Bring the testappsg service group online on sym3. Use a single hagrp
command with the propagate option to bring the dbsg, appsg, and websg
service groups online on sym3. Were you successful? If not, then switch the
testappsg service group to sym4 and retry. Can all three service groups be
brought online using just one command? If not, then ensure that all three
109

A9
110
A10
Using a single hagrp command with the propagate option, take the dbsg,
appsg, and websg service groups offline on sym3. Can all three service
groups be brought offline using just one command? If not, then ensure that all
three service groups are brought offline on sym3.
Using a single hagrp command with the propagate option, bring the dbsg,
appsg, and websg service groups online on sym3.

failover?
Clear the appip resource fault and use a single hagrp command with the
propagate option to bring the appsg service group back online on sym3.
Take the testappsg service group offline on sym3 and clear the dbip resource
fault. Then, use a single command to bring the dbsg, websg, appsg, and
testappsg service groups offline on any system they may be online.
Unlink the service group dependencies between the testappsg and appsg
service groups, between the dbsg and appsg service groups and between the
websg and appsg service groups. Save, but do not close the VCS
configuration.


In this exercise, you configure a typical multi-child service group dependency so
that one service group has two child service group dependencies causing a service
group to have preference being online on startup and failover.
sym3
From sym3:terminal1, create dependencies between the websg and appsg

service groups and websg and dbsg service groups as indicated below. Save but
do not close the VCS configuration.

111
Bring the testappsg service group online on sym4. Using a single hagrp
command with the propagate option, bring the dbsg, appsg, and websg
service groups online on sym3. Were you able to bring the websg, dbsg, and
appsg service groups online with the online propagate option?
Using a single hagrp command with the propagate option, take the websg,
dbsg, and appsg and service groups offline on sym3. Can all three service

A11
Fault the websg service group by removing the /var/tmp/webvip file

followed by a probe of the webvip resource on sym3. Do any service groups
failover?
Clear the webvip resource fault and bring the websg service group back online
on sym3.
Take the appsg service group offline on sym3. Then, bring the websg, appsg,
and dbsg service groups online on sym4. Account for the testappsg service
group.
Clear the dbip resource fault.
service groups, between the dbsg and appsg service groups, and between the
websg and appsg service groups. Save and close the VCS configuration.
112
11 From sym3:terminal2, terminate the hastatus command.
End of lab
A12

Lab 2: Merging clusters

In this lab, you will merge the east cluster to the west cluster forming one
four-node west cluster.
Exercise 2: Merge cluster preparation
Two clusters are prepared for merging.
Exercise 3: Merging cluster infrastructure

The systems from one cluster are added to another cluster.
113
Exercise 4: Adding service groups and resources

The service groups and resources from one of the original are added to the merged
cluster.
Exercise 5: Merged cluster verification
The merged cluster is verified.
A13

1
mgt
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas
sym3
Account: root
Password: veritas
sym4
Account: root
Password: veritas
114
opened.

terminal2.
A14

power on and log into sym1 and sym2.
115
A15

In this exercise, you prepare two clusters for merging.
sym1
Note: In this lab there are no iSCSI LUNs that are visible to all four systems, so
merging enabled I/O fencing using disks cannot be part of the cluster
merge lab. Therefore, for lab purposes, I/O fencing is re-configured to
disabled mode on both clusters.
From sym1:terminal1, navigate to the /opt/VRTS/install directory and

run the installsfha script with the fencing option using the following
information.
Reconfigure fencing in disabled mode.
Stop VCS and apply the fencing changes.
Do not view the summary file.
Use the gabconfig and service vxfen status commands to confirm

that I/O fencing is configured. Use the vxfenadm command to confirm that
I/O fencing is running in disabled mode.
sym3
116

information.
A16


Note: In this lab, there is a conflicting duplicate named appsg service group in
both the west and east clusters. This would cause errors during a merge.
The appsg service group in the west cluster would have precedence so the
appsg service group in the east cluster must be renamed or, as performed
in this lab, deleted from the configuration.
5
Take the appsg service group offline and list its resources and dependencies.
Open the VCS configuration for update and delete each resource in the appsg
service group in resource dependency order. Then, delete the appsg service
group and save and close the VCS configuration.
Navigate to the /etc/VRTSvcs/conf/config directory and use the

hacf -cftocmd command to create a main.cmd file. Copy the
main.cmd file to eastmain.cmd.
Edit the eastmain.cmd file to remove all commands other than those
related to the websg, testappsg, dbsg service groups, their resources, and their
resource dependencies.
Note: Alternatively, copy the

/student/labs/vcs/vcs60/maincf/CM/eastmain.cmd
file to /etc/VRTSvcs/conf/config/eastmain.cmd.
117
After you have made the edits the contents of the file should be:
hagrp -add dbsg
hagrp -modify dbsg SystemList sym3 0 sym4 1
hagrp -modify dbsg AutoStartList sym3 sym4
hagrp -modify dbsg SourceFile "./main.cf"
hares -add dbdg FileOnOff dbsg
hares -modify dbdg PathName "/var/tmp/dbdg"
hares -modify dbdg Enabled 1
hares -add dbip FileOnOff dbsg
hares -modify dbip PathName "/var/tmp/dbip"
hares -modify dbip Enabled 1
hares -add dblistener FileOnOff dbsg
hares -modify dblistener PathName "/var/tmp/
dblistener"
hares -modify dblistener Enabled 1
hares -add dbmnt FileOnOff dbsg
hares -modify dbmnt PathName "/var/tmp/dbmnt"
A17
hares -modify dbmnt Enabled 1

hares -add dbnic FileOnOff dbsg
hares -modify dbnic PathName "/var/tmp/dbnic"
hares -modify dbnic Enabled 1
hares -add dboracle FileOnOff dbsg
hares -modify dboracle PathName "/var/tmp/
dboracle"
hares -modify dboracle Enabled 1
hares -add dbvol FileOnOff dbsg
hares -modify dbvol PathName "/var/tmp/dbvol"
hares -modify dbvol Enabled 1
hagrp -add testappsg
hagrp -modify testappsg SystemList sym3 0 sym4 1
hagrp -modify testappsg AutoStartList sym3 sym4
hagrp -modify testappsg Operators oper
hagrp -modify testappsg SourceFile "./main.cf"
hares -add testappdg FileOnOff testappsg
hares -modify testappdg PathName "/var/tmp/
testappdg"
hares -modify testappdg Enabled 1
hares -add testappmnt FileOnOff testappsg
hares -modify testappmnt PathName "/var/tmp/
testappmnt"
hares -modify testappmnt Enabled 1
hares -add testappnic FileOnOff testappsg
hares -modify testappnic PathName "/var/tmp/
testappnic"
hares -modify testappnic Enabled 1
hares -add testappproc FileOnOff testappsg
hares -modify testappproc PathName "/var/tmp/
testappproc"
hares -modify testappproc Enabled 1
hares -add testappip FileOnOff testappsg
hares -modify testappip PathName "/var/tmp/
testappip"
hares -modify testappip Enabled 1
hares -add testappvol FileOnOff testappsg
hares -modify testappvol PathName "/var/tmp/
testappvol"
hares -modify testappvol Enabled 1
118
A18

119
hagrp -add websg

hagrp -modify websg SystemList sym3 0 sym4 1
hagrp -modify websg AutoStartList sym3 sym4
hagrp -modify websg Operators oper
hagrp -modify websg SourceFile "./main.cf"
hares -add webapache FileOnOff websg
hares -modify webapache PathName "/var/tmp/
webapache"
hares -modify webapache Enabled 1
hares -add webdg FileOnOff websg
hares -modify webdg PathName "/var/tmp/webdg"
hares -modify webdg Enabled 1
hares -add webmnt FileOnOff websg
hares -modify webmnt PathName "/var/tmp/webmnt"
hares -modify webmnt Enabled 1
hares -add webnic FileOnOff websg
hares -modify webnic PathName "/var/tmp/webnic"
hares -modify webnic Enabled 1
hares -add webvip FileOnOff websg
hares -modify webvip PathName "/var/tmp/webvip"
hares -modify webvip Enabled 1
hares -add webvol FileOnOff websg
hares -modify webvol PathName "/var/tmp/webvol"
hares -modify webvol Enabled 1
hares -link dbip dbnic
hares -link dblistener dbip
hares -link dblistener dboracle
hares -link dbmnt dbvol
hares -link dboracle dbmnt
hares -link dbvol dbdg
hares -link testappmnt testappvol
hares -link testappproc testappmnt
hares -link testappproc testappip
hares -link testappip testappnic
hares -link testappvol testappdg
hares -link webapache webmnt
hares -link webapache webvip
hares -link webmnt webvol
hares -link webvip webnic
hares -link webvol webdg
A19
Shutdown VCS, IO fencing, GAB, and LLT on both nodes of the east cluster.
10 Rename the /etc/VRTSvcs/conf/config/main.cf file to
main.cf.east on sym3 and sym4.
Note: In order for the merge utility to work correctly, the main.cf files on
the east cluster nodes must not exist.
120 A20


In this exercise, you add the systems from one cluster to another cluster.
sym1
1

run the ./installsfha -addnode command taking note of the merge
pre-requisites and using the following information.
Enter sym1 as the name of one node of the west cluster.

Enter sym3 and sym4 as the systems to be added to the west cluster.
Use eth4 and eth5 as the private links and eth0 as the low priority link.
Use eth0 as the public NIC used for the cluster virtual IP.
Record the log file located in the /opt/VRTS/install/logs
directory. Optionally, review the log file.
Log file:
121
Perform a summary status of the cluster noting that none of the former east
cluster service groups have been added.
Use the gabconfig command to check GAB status.
Use the vxfenadm command to check I/O fencing.
Use the lltstat command to check the LLT configured and active status.
Optionally, performed this step on all cluster nodes.
Display the contents of the /etc/llttab, /etc/llthosts, and

/etc/gabtab files. Optionally, perform this step on all cluster nodes.
A21
Display the /etc/VRTSvcs/conf/config/main.cf file noticing the

following elements.
\
The cluster name is west.

The cluster virtual ip is 10.10.2.51.
All four systems are defined as cluster members.
The SystemList and AutoStartList service group attributes for the
ClusterService service group configure all four cluster systems.
The appsg, nfssg, and orasg service groups from the original west cluster
are configured and remain restricted to sym1 and sym2.
The websg, dbsg, and testappsg from the east cluster are not configured.
122 A22


In this exercise, you add the service groups and resources from one of the original
clusters to the merged cluster.
sym3
1
From sym3:terminal1, navigate to the /etc/VRTSvcs/conf/config

directory and open the VCS configuration for update
Run the sh -x ./eastmain.cmd command and observe the output taking

notice of the VCS warning when failing to add the VCS user named oper as an
Operator for the testappsg and websg service groups.
Note: The oper user add is not part of the eastmain.cmd file and would
need to be re-added and re-configured later on.
Save and close the VCS configuration.
123
A23

In this exercise, you verify the merged cluster.
sym3
1
From sym3:terminal1, perform a summary status on the cluster taking note of

the service groups that are not online.
Display /etc/VRTSvcs/conf/config/main.cf file and notice the

following.
The oper user from the old east cluster is not defined nor is the Operator
service group attribute for the testappsg and websg service groups set to
the oper user.
ClusterService service group have been modified for all four cluster
systems.
The SystemList and AutoStartList service group attributes for the appsg,
nfssg, and orasg service groups remain set to sym1 and sym2.
Note: These service groups cannot be online on sym3 and sym4 due to
shared LUN restrictions.
The SystemList and AutoStartList service group attributes for the websg,
testappsg, and dbsg service groups remain set to sym3 and sym4.
Note: These service groups could be online on sym1 and sym2 and could
be modified to accommodate that.
Note: Consideration needs to be given to activated triggers configured at any

service group and/or resource level if the service group is going to be
re-configured to run on a new cluster system.
End of lab
124 A24

Lab 3: Failover policies

In this lab, you use the Veritas Cluster Server Simulator with a preconfigured
main.cf file to show how to configure and test failover policies using the Veritas
Cluster Manager Java Console.
Exercise 2: Starting the simulator

The simulation used in the lab is examined and the Veritas VCS Simulator is
started.
125
Exercise 3: Testing priority failover policy

The default priority failover policy is examined.
Exercise 4: Testing load failover policy
The load failover policy is examined.
Exercise 5: Testing prerequisites and limits
Prerequisites and limits are examined and added to the load failover policy
settings.
Exercise 6: Stopping the simulator
terminated.
A25

1
opened.
mgt
winclient
Account: Administrator
Password: veritas
terminal2.
126 A26
shutdown sym1, sym2, sym3 and sym4, and power on and log into
winclient.


In this exercise, you examine the simulation used in the lab and start the Veritas
VCS Simulator.
winclient
1
From the winclient desktop, use Windows Explorer to navigate to the

C:\Progam Files (x86)\VERITAS\VCS Simulator\wlm\
conf\config folder.
Open the main.cf file using the WordPad application.
Without making any changes, review the contents of the main.cf file. Then,
close the Wordpad and Windows Explorer windows.
Start the Veritas VCS Simulator.
Note: The VCS Simulator may take a moment to display as it discovers any
running simulations even thought there will not be any.
127
Verify the wlm simulation.
Start the wlm simulation.

Note: The status of the cluster name will have a green check mark upon
successful startup.
Launch the VCS Java Console and log in using the following credentials.
User name: admin
Password: password
A27
From the Java Console, verify that the Member Systems and their Priority
and their AutoStartList attribute are set to the values shown in the following
table.
Service Group
Member Systems (Priority)
AutoStartList
A1
S1 (1), S2 (2), S3 (3), S4 (4)
S1
A2
S1 (1), S2 (2), S3 (3), S4 (4)
S1
B1
S1 (4), S2 (1), S3 (2), S4 (3)
S2
B2
S1 (4), S2 (1), S3 (2), S4 (3)
S2
C1
S1 (3), S2 (4), S3 (1), S4 (2)
S3
C2
S1 (3), S2 (4), S3 (1), S4 (2)
S3
D1
S1 (2), S2 (3), S3 (4), S4 (1)
S4
D2
S1 (2), S2 (3), S3 (4), S4 (1)
S4
128 A28


In this exercise, you examine the default priority failover policy.
winclient
1
From the Java Console, verify that the failover policy of all service groups is
Priority.
Verify that all service groups are online as shown in the following table.
Service Group
System S1
A1
Online
A2
Online
System S2
B1
Online
B2
Online
System S3
C1
Online
C2
Online
System S4
D1
Online
D2
Online
If it faults, where will the A1 service group fail over to? Verify this failover by
faulting a critical resource in the A1 service group.
Service Group A1 will fail over to:
129
If the A1 service group faults again without clearing the previous fault, where
should it fail over? Verify the failover by faulting a critical resource in the A1
service group.
Clear the faults in the A1 service group. Where will the service group fail over
to now? Verify the failover by faulting a critical resource in the A1 service
group.
Clear the faults in the A1 service group.
A29

In this exercise, you examine the load failover policy.
winclient
1
From the Java Console, open the cluster configuration for update.
Set the FailOverPolicy attribute to Load for the eight service groups.
Set the Load attribute for each service group as shown in the following table.
130 A30
Service Group
Load
A1
75
A2
75
B1
75
B2
75
C1
50
C2
50
D1
50
D2
50
Set the S1 and S2 systems Capacity attributes to 200.

Note: The S3 and S4 systems Capacity attribute will not be changed and
defaults to 100.
Verify that the systems AvailableCapacity attribute is as shown in the

following table.
Service Group
AvailableCapacity
S1
50
S2
50
S3
S4
0
Save the configuration changes.
Service Group
System S1
A1
Online
A2
Online
System S2
B1
Online
B2
Online
System S3
C1
Online
C2
Online
System S4
D1
Online
D2
Online

following table.
Service Group
AvailableCapacity
S1
125
S2
-25
S3
S4
10 If the S2 system fails, where will the A1, B1 and B2 service groups that are
131
Online on S2 fail over? Verify this failover by powering off the S2 system in
Cluster Manager.
The A1 service group will fail over to:
The B1 service group will fail over to:
A31
11 Verify that the systems AvailableCapacity attribute is as shown in the
following table.
Service Group
AvailableCapacity
S1
-25
S2
200
S3
-75
S4
12 Power up the S2 system, clear all faults, and return the service groups to their
startup locations.
following table.
Service Group
AvailableCapacity
S1
50
S2
50
S3
S4
14 Save and close the configuration changes.
Note: Saving changes to the configuration is not necessary for purposes of

this lab. However, if you want to view the modifications to the
main.cf file then save the changes.
132 A32


In this exercise, you examine the prerequisites and limits and add them to the load
failover policy settings.
winclient
1
Set the Limits attribute for each system to the key ABGroup with a value of 3.
Set the Prerequisites attribute for the A1, A2, B1 and B2 service groups to be
ABGroup with a value of 1.
CAUTION Do not modify the Prerequisites attribute for the C1, C2, D1,
and D2 service groups.
If the S1 system fails, where will the A1 and A2 service groups that are Online
on S1 fail over? Verify this failover by powering off the S1 system in Cluster
Manager. Explain the failover actions that occurred.
133
A33
If the S2 system fails, where will the A1, B1 and B2 service groups that are
Cluster Manager. Explain the failover actions that occurred.
If the S3 system fails, where will the A2, B1, C1 and C2 service groups that
are Online on S3 fail over? Verify this failover by powering off the S3 system
in Cluster Manager. Explain the failover actions that occurred.
The C1 service group will fail over to:
134 A34
S4
Save and close the configuration changes.



In this exercise, you terminate the simulation and the Veritas VCS Simulator.
winclient
1
Log out from Cluster Explorer.
From the Simulator Java Console, stop the wlm cluster.
Verify the configuration.

Note: If you intend to move this configuration in to a running cluster, you
must ensure the main.cf syntax is correct. You can verify the
configuration only in the Simulator GUI after the cluster is stopped.
Close the Simulator Java Console.

Note: The Veritas Cluster Manager Java Console should close as well. If it
does not then close it manually.
End of lab
135
A35
136 A36

Lab 4: Creating a parallel network service group

In this lab application networking is reconfigured to be more efficient.
Exercise 2: Configuring a common parallel network service group
A common parallel network service group and resources is configured.
Exercise 3: Replacing NIC resources with Proxy resources

The pre-existing resources of type NIC supporting application VIPs in each
application service group are replaced with resources of type Proxy.
137

A37

138 A38
opened.
mgt
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas
sym3
Account: root
Password: veritas
sym4
Account: root
Password: veritas
terminal2.

shutdown winclient, and power on and log into sym1, sym2, sym3
and sym4.
139

A39

In this exercise, you configure a common parallel network service group and
resources.
sym1
140 A40
From sym1:terminal1, use the haconf command to open the cluster

configuration for update.
Use the hagrp command to create a service group named netsg.
Modify the SystemList to allow the netsg service group to run on the sym1,
sym2, sym3, and sym4 in that order.
Modify the AutoStartList attribute to allow the service group to start on all
systems.
Modify the Parallel service group to configure the netsg service group as a
parallel service group.
Display the service group attributes for the netsg service group to confirm your
input and key default service group attribute values.
Save the VCS configuration, but do not close it.
Add a resource of type NIC named netnic to the netsg service group.
Modify the Device and Critical resource attributes for the netnic resource
using the following information.
Device is eth0
Critical set to 0 (zero)
10 Display the resource attribute values for the netnic resource to confirm your
input and key default resource attribute values.

11 Enable the netnic resource and verify that it is enabled. Display the state of the
netnic resource to ensure that it is online on both cluster systems.

Note: It is not necessary to bring a resource of type NIC online.
12 Display the state of the netsg service group.
Note: The service group state is offline even though the NIC resource state is
online. This is because VCS does not consider persistent resources
when determining the state of a service group.
13 Save the VCS configuration, but do not close it.
14 Add a resource of type Phantom named netphantom to the netsg service
group.
15 Set the Critical resource attribute to 0 (zero) for the netphantom resource.
16 Display the resource attribute values for the netphantom resource to confirm
your input and key default resource attribute values.

17 Enable the netphantom resource and verify that it is enabled. Display the state
of the netphantom resource to ensure that it is online on both cluster systems.

Note: It is not necessary to bring a resource of type Phantom online.
141

A41

In this exercise, you replace the pre-existing resources of type NIC that supporting
application VIPs in each application service group with resources of type Proxy.
sym1
1
From sym1:terminal1, identify the names of the resources of type NIC that are
configured.
Ignoring the resource of type NIC named netnic, list the resource
dependencies associated with the listed resources of type NIC.
Replace the resource of type NIC named appnic with a resource of type Proxy
named appproxy using the following information.
Replace the resource of type NIC named csgnic with a resource of type Proxy
named csgproxy using the following information.
142 A42
Set the Critical attribute to 0 (zero).

Set the TargetResName attribute to netnic.
Set Enabled to 1 (one).
Link to the appip resource.

Link to the notifier and webip resources.
Replace the resource of type NIC named nfsnic with a resource of type Proxy
named nfsproxy using the following information.

Link to the nfsip resource.

Replace the resource of type NIC named oranic with a resource of type Proxy
named oraproxy using the following information.

Link to the oraip resource.
End of lab
143 Lab 4: Creating a parallel network service group
A43
144 A44

Appendix B
Lab Solutions
145
146 B2


In this lab, you will examine a variety of service group dependency configurations
and behaviors. Online and offline service group propagation is also examined.

The service groups and resources in the east cluster are examined.
A typical service group dependency is configured so that one service group will
have multiple parents causing one service group to have preference being online
on startup and failover over another service group.
A typical service group dependency is configured with a three level dependency
causing one service group to have preference being online on startup and failover
over another service group.
A typical multi-child service group dependency is configured so that one service
group has two child service group dependencies causing a service group to have
preference being online on startup and failover.
147 Lab 1: Service group dependencies
B3

1
opened.
mgt
sym3
Account: root
Password: veritas
sym4
Account: root
Password: veritas

terminal2.
148 B4
shutdown cps, sym1 and sym2, and power on and log into sym3 and
sym4.


In this exercise, you examine the service groups and resources in the east cluster.
sym3
Note: The appsg and testappsg service groups were added in an early Install and
Configure lab using the Veritas Cluster Manager Java Console. If those
lab exercises were skipped, then consult with your instructor on how to
copy in a replacement main.cf file located at:
/student/labs/vcs/vcs60/maincf/IC/
main.cf.east.lab05
1
From sym3:terminal1, perform a summary status on the cluster by:

Listing the state of each service group.
Verifying that all service groups are online on the sym3 system and
switching any that need to be switched.
Verifying that there are no service group dependencies.
Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym3
sym4
-- GROUP STATE
-- Group
System
Probed
AutoDisabled
State
B
B
B
B
B
B
B
B
B
B
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
N
N
N
N
N
N
N
N
N
N
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
hagrp -dep
VCS WARNING V-16-1-50035 No Group dependencies are configured
End of Solution
149 Lab 1: Service group dependencies
B5
Display the state of all resources by listing the resources for each service group
and the dependencies between each resource in each service group.
Solution
150 B6
hares -state
#Resource
appdg
appdg
appmnt
appmnt
appnic
appnic
appproc
appproc
appip
appip
appvol
appvol
csgnic
csgnic
dbdg
dbdg
dbip
dbip
dblistener
dblistener
dbmnt
dbmnt
dbnic
dbnic
dboracle
dboracle
dbvol
dbvol
notifier
notifier
testappdg
testappdg
testappmnt
testappmnt
testappnic
testappnic
testappproc
testappproc
testappip
testappip
testappvol
testappvol
webapache
webapache
webdg
webdg
webip
webip
webmnt
webmnt
webnic
webnic
webvip
webvip
webvol
webvol
Attribute
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
ONLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE

hagrp -resources ClusterService

webip
csgnic
notifier
hares -dep | grep ClusterService

ClusterService notifier
ClusterService webip
csgnic
csgnic
hagrp -resources appsg

appproc
appdg
appmnt
appnic
appip
appvol
hares -dep | grep "^appsg"

appsg
appsg
appsg
appsg
appsg
appmnt
appproc
appproc
appip
appvol
appvol
appip
appmnt
appnic
appdg
hagrp -resources dbsg

dbdg
dbip
dblistener
dbmnt
dbnic
dboracle
dbvol
151
hares -dep | grep dbsg

dbsg
dbsg
dbsg
dbsg
dbsg
dbsg
dbip
dblistener
dblistener
dbmnt
dboracle
dbvol
dbnic
dboracle
dbip
dbvol
dbmnt
dbdg
hagrp -resources testappsg

testappproc
testappdg
testappmnt
testappnic
testappip
testappvol

B7
hares -dep | grep testappsg

testappsg
testappsg
testappsg
testappsg
testappsg
testappmnt
testappproc
testappproc
testappip
testappvol
testappvol
testappip
testappmnt
testappnic
testappdg
hagrp -resources websg

webapache
webdg
webmnt
webnic
webvip
webvol
hares -dep | grep websg

websg
websg
websg
websg
websg
webapache
webapache
webmnt
webvip
webvol
webvip
webmnt
webvol
webnic
webdg
End of Solution
Determine if all resources in all service groups other than the ClusterService
service group are set to critical. If any resources are set to non critical then
modify them. Open, save and close the cluster configuration as appropriate.
Solution
152 B8
hares -list Critical=0 Group=appsg
hares -list Critical=0 Group=dbsg
hares -list Critical=0 Group=testappsg
hares -list Critical=0 Group=websg
End of Solution
Confirm that the FaultPropagation and ManageFaults service group

attributes are set to 1 and ALL respectively for each service group. Modify to
their default value if necessary. Open, save and close the cluster configuration
as appropriate.

Solution
hagrp -display -attribute ManageFaults

FaultPropagation
#Group
ClusterService
ClusterService
#
appsg
appsg
#
dbsg
dbsg
#
testappsg
testappsg
#
websg
websg
Attribute
FaultPropagation
ManageFaults
System
global
global
Value
1
ALL
FaultPropagation
ManageFaults
global
global
1
ALL
FaultPropagation
ManageFaults
global
global
1
ALL
FaultPropagation
ManageFaults
global
global
1
ALL
FaultPropagation
ManageFaults
global
global
1
ALL
End of Solution
Open the cluster configuration for update.

Solution
haconf -makerw
End of Solution
From sym3:terminal2, run the hastatus command in order to observe

cluster changes as they happen in subsequent exercises.
Solution
153
hastatus
testappproc
sym4
OFFLINE
testappip
sym3
ONLINE
testappip
sym4
OFFLINE
------------------------------------------------------------------------testappvol
sym3
ONLINE
testappvol
sym4
OFFLINE
webapache
sym3
OFFLINE
webapache
sym4
ONLINE
webdg
sym3
OFFLINE
------------------------------------------------------------------------webdg
sym4
ONLINE
group
resource
system
message
--------------- -------------------- -------------------- -------------------

B9
webmnt
sym3
OFFLINE
webmnt
sym4
ONLINE
webnic
sym3
OFFLINE
webnic
sym4
ONLINE
------------------------------------------------------------------------webvip
sym3
OFFLINE
webvip
sym4
ONLINE
webvol
sym3
OFFLINE
webvol
sym4
ONLINE
End of Solution
154 B10


In this exercise, you configure a typical service group dependency so that one
service group will have multiple parents causing one service group to have
preference being online on startup and failover over another service group.
sym3
1
From sym3:terminal1, take all of the service groups except the

ClusterService service group offline on all cluster systems.
Solution
hagrp -offline dbsg -any

VCS NOTICE V-16-1-50733 Attempting to offline group on system sym3
hagrp -offline appsg -any

hagrp -offline websg -any

155
hagrp -offline testappsg -any

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
End of Solution

B11
Create and confirm dependencies between the appsg and dbsg and between
the websg and dbsg service groups as indicated below. Save but do not close
Solution
hagrp -link appsg dbsg online local firm
hagrp -link websg dbsg online local firm
hagrp -dep
#Parent
appsg
websg
Child
dbsg
dbsg
Relationship
online local firm
online local firm
haconf -dump
End of Solution
Use a single hagrp command with the propagate option to bring the dbsg,
appsg, and websg service groups online on sym3. Can all three service groups
be brought online using just one command? If not, then ensure that all three
Both top level parent service groups are independent so only one of the parents and
the shared child dbsg service group can be brought online using the propagate option.
Solution
156 B12
hagrp -online -propagate websg -sys sym3

VCS NOTICE V-16-1-40147 Attempting to online group(s) websg dbsg on system
sym3.
hagrp -wait websg State ONLINE -sys sym3
hagrp -state
#Group
ClusterService
ClusterService
appsg
Attribute
State
State
State
System
sym3
sym4
sym3
Value
|ONLINE|
|OFFLINE|
|OFFLINE|

appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
State
State
State
State
State
State
State
sym4
sym3
sym4
sym3
sym4
sym3
sym4
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
hagrp -online -propagate appsg -sys sym3

Note: The propagate option is not required nor is it invalid. The results are
the same as hagrp -online appsg -sys sym3.
VCS NOTICE V-16-1-40147 Attempting to online group(s) appsg on system sym3.
hagrp -wait appsg State ONLINE -sys sym3
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
End of Solution
157
Use a single hagrp command to take the dbsg, appsg, and websg service
groups offline on sym3. Can all three service groups be brought offline using
just one command? If not, then ensure that all three service groups are brought
offline on sym3.
All three service groups can be taken offline with one command.
Solution
hagrp -offline -propagate dbsg -sys sym3

VCS NOTICE V-16-1-40158 Attempting to offline group(s) dbsg websg appsg on
system sym3.
hagrp -wait dbsg State OFFLINE -sys sym3

B13
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
End of Solution
Using a single hagrp command with the propagate option, bring the dbsg
and websg service groups online on sym3. Attempt to bring the appsg service
group online on sym4. Were you successful? If not successful, then bring the
appsg service group online on sym3.
The appsg service group cannot be brought online on sym4 due to service group
dependencies.
Solution

sym3.
158 B14
hagrp -online appsg -sys sym4

VCS WARNING V-16-1-10163 Group dependency is not met if group appsg goes
online on system sym4
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
Attribute
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|

testappsg
testappsg
websg
websg
State
State
State
State
sym3
sym4
sym3
sym4
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
End of Solution

failover?
Only appsg faults. Neither the dbsg nor websg failover or change state.
Solution
rm /var/tmp/appip
hares -probe appip -sys sym3
hagrp -wait appsg State "OFFLINE|FAULTED" -sys

sym3
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|OFFLINE|FAULTED|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
End of Solution
159
Clear the appip resource fault and bring the appsg service group back online
on sym3.
Solution
hares -clear appip -sys sym3

B15
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
End of Solution
The dbsg service group faults. The appsg and websg service groups are taken offline
on sym3. All three service groups failover to sym4.
Solution
160 B16
rm /var/tmp/dbip
hares -probe dbip -sys sym3
hagrp -wait dbsg State 'OFFLINE|FAULTED' -sys sym3
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|STOPPING|
|OFFLINE|
|OFFLINE|FAULTED|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|STOPPING|
|OFFLINE|
hagrp -state
#Group
ClusterService
ClusterService
appsg
Attribute
State
State
State
System
sym3
sym4
sym3
Value
|ONLINE|
|OFFLINE|
|OFFLINE|

appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
State
State
State
State
State
State
State
sym4
sym3
sym4
sym3
sym4
sym3
sym4
|ONLINE|
|OFFLINE|FAULTED|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
End of Solution
Clear the dbip resource fault and take the dbsg, websg, appsg, and testappsg
service groups offline on any system they may be online.
Solution
hares -clear dbip -sys sym3
hagrp -offline -propagate dbsg -any

system sym4.
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
End of Solution
161
10 Link the testappsg service group as a parent of the appsg service group with
an offline local dependency. Save, but do not close the VCS configuration.
Note: This will configure a preference for the appsg service group running on
a system over the testappsg service group and force the testappsg to
switch upon a successful failover of the appsg service group.
Solution
hagrp -link testappsg appsg offline local

B17
hagrp -dep
#Parent
appsg
testappsg
websg
haconf -dump
Child
dbsg
appsg
dbsg
Relationship
online local firm
offline local
online local firm
haconf -dump
End of Solution
11 Bring the dgsg, appsg, and websg service groups online on sym3. Attempt to
bring the testappsg service group online on sym3. Were you successful? If
not, attempt to bring the testappsg service group online on sym4.
The testappsg service group cannot be brought online on the same system as the
appsg service group.
Solution

sym3.
162 B18
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
hagrp -online testappsg -sys sym3

VCS WARNING V-16-1-10163 Group dependency is not met if group testappsg goes
online on system sym3

hagrp -wait testappsg State ONLINE -sys sym4
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|
End of Solution
12 Fault the dbsg service group by removing the /var/tmp/dbip file followed

The dbsg service group faults. The appsg and websg service groups are taken offline
on sym3. The testappsg service group is switched from sym4 to sym3. The dbsg,
appsg, and websg service groups failover to sym4.
Solution
163
rm /var/tmp/dbip
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|PARTIAL|STOPPING|
|OFFLINE|
|OFFLINE|FAULTED|
|OFFLINE|
|OFFLINE|
|ONLINE|
|PARTIAL|STOPPING|
|OFFLINE|
Note: Depending on your timing, appsg may show OFFLINE on both

systems.

B19
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|FAULTED|
|ONLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
End of Solution
13 Take the testappsg service group offline on sym3 and clear the dbip resource
fault. Then, use a single command to take the dbsg, websg, appsg, and
testappsg service groups offline on any system where they may be online.
Solution
hagrp -offline testappsg -sys sym3

system sym4.
164 B20
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
End of Solution

service groups and between the dbsg, websg and appsg service groups. Save,
but do not close the VCS configuration.
Solution
hagrp -unlink testappsg appsg
hagrp -unlink websg dbsg
hagrp -unlink appsg dbsg
hagrp -dep
haconf -dump
End of Solution
165

B21

In this exercise, you configure a typical service group dependency with a three
level dependency causing one service group to have preference being online on
startup and failover over another service group.
sym3
From sym3:terminal1, create dependencies between the appsg and dbsg

service groups, between the websg and appsg service groups and between the
testappsg and appsg service groups as indicated below. Save but do not close
Solution
166 B22
hagrp -link appsg dbsg online local firm
hagrp -link websg appsg online local firm
hagrp -dep
#Parent
appsg
testappsg
websg
Child
dbsg
appsg
appsg
Relationship
online local firm
offline local
online local firm
haconf -dump
End of Solution

Bring the testappsg service group online on sym3. Use a single hagrp
command with the propagate option to bring the dbsg, appsg, and websg
service groups online on sym3. Were you successful? If not, then switch the
testappsg service group to sym4 and retry. Can all three service groups be
brought online using just one command? If not, then ensure that all three
The testappsg is already online on sym3 through normal mechanisms so the appsg
service group cannot come online there. It could, under the proper circumstances, fail
over to sym3 preferentially, but it cannot just come online there with the testappsg
already online.
All three service groups can be brought online using a single command. The dbsg
service group is brought online, followed by the appsg service group and finally the
websg service group.
Solution
VCS WARNING V-16-1-40148 Cannot online group appsg on system sym3.
167
hagrp -switch testappsg -to sym4

VCS NOTICE V-16-1-40147 Attempting to online group(s) websg appsg dbsg on
system sym3.
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
Attribute
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3

Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
B23
dbsg
testappsg
testappsg
websg
websg
State
State
State
State
State
sym4
sym3
sym4
sym3
sym4
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|
End of Solution
Using a single hagrp command with the propagate option, take the dbsg,
appsg, and websg service groups offline on sym3. Can all three service
All three service groups can be taken offline with one command. The testappsg
service group remains online.
Solution
VCS NOTICE V-16-1-40158 Attempting to offline group(s) dbsg appsg websg on

system sym3.
168 B24
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
End of Solution
Solution

system sym3.

hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|
End of Solution

failover?
The appsg faults. The websg is taken offline. No service groups fail over.
Solution
Solution
169
rm /var/tmp/appip
hares -probe appip -sys sym3
hagrp -wait appsg State "OFFLINE|FAULTED" -sys

sym3
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|OFFLINE|FAULTED|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|PARTIAL|STOPPING|
|OFFLINE|
End of Solution

B25
Clear the appip resource fault and use a single hagrp command with the
propagate option to bring the appsg service group back online on sym3.
Solution
hares -clear appip -sys sym3
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|
End of Solution
The dbsg service group faults. The appsg and websg service groups are brought
offline on sym3. The testappsg service group is switched from sym4 to sym3. The
dbsg, appsg, and websg service groups failover to sym4.
170 B26
Solution
rm /var/tmp/dbip
hagrp -state
#Group
Attribute
ClusterService State
System
sym3
sym4
Value
|ONLINE|
|OFFLINE|

appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
State
State
State
State
State
State
State
State
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
|ONLINE|STOPPING|
|OFFLINE|
|OFFLINE|FAULTED|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|FAULTED|
|ONLINE|
|PARTIAL|STARTING|
|OFFLINE|
|OFFLINE|
|ONLINE|
End of Solution
Take the testappsg service group offline on sym3 and clear the dbip resource
fault. Then, use a single command to bring the dbsg, websg, appsg, and
testappsg service groups offline on any system they may be online.
Solution
171
hagrp -offline testappsg -sys sym3

VCS NOTICE V-16-1-40173 Attempting to offline group(s) dbsg appsg on system
sym4.
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
Attribute
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4

Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
B27
websg
websg
State
State
sym3
sym4
|OFFLINE|
|OFFLINE|
End of Solution
Unlink the service group dependencies between the testappsg and appsg
service groups, between the dbsg and appsg service groups and between the
websg and appsg service groups. Save, but do not close the VCS
configuration.
Solution
hagrp -unlink websg appsg
hagrp -unlink appsg dbsg
hagrp -dep
haconf -dump
End of Solution
172 B28


In this exercise, you configure a typical multi-child service group dependency so
that one service group has two child service group dependencies causing a service
group to have preference being online on startup and failover.
sym3
From sym3:terminal1, create dependencies between the websg and appsg

service groups and websg and dbsg service groups as indicated below. Save but
do not close the VCS configuration.
Solution
173
hagrp -link websg appsg online local firm
hagrp -link websg dbsg online local firm
hagrp -dep
#Parent
testappsg
websg
websg
Child
appsg
appsg
dbsg
Relationship
offline local
online local firm
online local firm
haconf -dump
End of Solution

B29
Bring the testappsg service group online on sym4. Using a single hagrp
command with the propagate option, bring the dbsg, appsg, and websg
service groups online on sym3. Were you able to bring the websg, dbsg, and
appsg service groups online with the online propagate option?
Yes, the online propagate option worked as expected.
Solution

system sym3.
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|
End of Solution
174 B30
Using a single hagrp command with the propagate option, take the websg,
dbsg, and appsg and service groups offline on sym3. Can all three service
No, all three service groups cannot be taken offline with one command. The appsg
service group needs to be taken offline explicitly in this case.

Solution

VCS NOTICE V-16-1-40158 Attempting to offline group(s) dbsg websg on system
sym3.
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
hagrp -offline appsg -sys sym3
hagrp -wait appsg State OFFLINE -sys sym3
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
End of Solution
175
Solution

system sym3.

B31
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|
End of Solution
Fault the websg service group by removing the /var/tmp/webvip file

followed by a probe of the webvip resource on sym3. Do any service groups
failover?
The websg faults. No service groups change state or fail over.
Solution
176 B32
rm /var/tmp/webvip
hares -probe webvip -sys sym3
hagrp -wait websg State "OFFLINE|FAULTED" -sys

sym3
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|FAULTED|
|OFFLINE|
End of Solution

Clear the webvip resource fault and bring the websg service group back online
on sym3.
Solution
hares -clear webvip -sys sym3
hagrp -online websg -sys sym3
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|
End of Solution
The dbsg service group faults on sym3. The websg service group is taken offline on
sym3. The appsg service group is unaffected and no failover occurs. The testappsg
service group remains online on sym4.
177
Solution
rm /var/tmp/dbip
hagrp -state
#Group
Attribute
System
sym3
sym4

Value
|ONLINE|
|OFFLINE|
B33
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
State
State
State
State
State
State
State
State
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
|ONLINE|
|OFFLINE|
|OFFLINE|FAULTED|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
End of Solution
Take the appsg service group offline on sym3. Then, bring the websg, appsg,
and dbsg service groups online on sym4. Account for the testappsg service
group.
Solution
hagrp -offline appsg -sys sym3
hagrp -switch testappsg -to sym3
hagrp -wait testappsg State OFFLINE -sys sym3
VCS NOTICE V-16-1-40147 Attempting to online group(s) websg appsg on system

sym4.
178 B34
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|FAULTED|
|ONLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
End of Solution

Clear the dbip resource fault.

Solution
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|ONLINE|
|OFFLINE|
|OFFLINE|
|ONLINE|
End of Solution
service groups, between the dbsg and appsg service groups, and between the
websg and appsg service groups. Save and close the VCS configuration.
Solution
hagrp -unlink websg appsg
hagrp -unlink websg dbsg
hagrp -dep
179
haconf -dump -makero
End of Solution

B35
11 From sym3:terminal2, terminate the hastatus command.

Solution
From sym3:terminal2, press Ctrl-C.

End of Solution
End of lab
180 B36


In this lab, you will merge the east cluster to the west cluster forming one
four-node west cluster.
Two clusters are prepared for merging.

The systems from one cluster are added to another cluster.
181

The service groups and resources from one of the original are added to the merged
cluster.
The merged cluster is verified.
B37

182 B38
opened.
mgt
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas
sym3
Account: root
Password: veritas
sym4
Account: root
Password: veritas
terminal2.

power on and log into sym1 and sym2.
183
B39

In this exercise, you prepare two clusters for merging.
sym1
Note: In this lab there are no iSCSI LUNs that are visible to all four systems, so
merging enabled I/O fencing using disks cannot be part of the cluster
merge lab. Therefore, for lab purposes, I/O fencing is re-configured to
disabled mode on both clusters.

information.
Solution
cd /opt/VRTS/install
./installsfha -fencing
Checking communication on sym1
Checking release compatibility on sym1
Checking VCS installation on sym1
Cluster information verification:
184 B40
Cluster Name: west

Cluster ID Number: 5
Systems: sym1 sym2
Would you like to configure I/O fencing on the cluster? [y,n,q]
y
Checking
Done
Checking
Checking
Checking
communication on sym1 ....................................

release compatibility on sym1 ............................ Done
VCS installation on sym1 .................. Version 6.0.000.000

Done
Checking release compatibility on sym2 ............................ Done
Checking VCS installation on sym2 .................. Version 6.0.000.000
Fencing is already started in enabled mode, do you want to reconfigure it?
[y,n,q] (y)
y
Fencing configuration
1) Configure Coordination Point client based fencing
2) Configure disk based fencing
3) Configure fencing in disabled mode
4) Online fencing migration
Select the fencing mechanism to be configured in this Application Cluster:
3
Installer will stop VCS before applying fencing configuration. To make sure
VCS shuts down successfully, unfreeze any frozen service group and unmount
the mounted file systems in the cluster.
Are you ready to stop VCS and apply fencing configuration on all nodes at
this time? [y,n,q,b] (y)
y
Deleting Coordination Point Agent Service Group westfensg ......... Done
Stopping VCS on sym1 ..............................................
Done
Stopping Fencing on sym1 ..........................................
Done
Stopping VCS on sym2 ..............................................
Done
Stopping Fencing on sym2 ..........................................
Done
Configuring fencing in disabled mode on sym1 ...................... Done
Configuring fencing in disabled mode on sym2 ...................... Done
Updating main.cf without fencing ..................................
Done
Starting VCS on sym1 ..............................................
Done
Starting VCS on sym2 ..............................................
Done
I/O Fencing configuration .........................................
185
Done
I/O Fencing configuration completed successfully
...
installsfha log files, summary file, and response file are saved at:
/opt/VRTS/install/logs/installsfha-201112081226AJb
Would you like to view the summary file? [y,n,q] (n)
End of Solution
B41

Solution
gabconfig -a
GAB Port Memberships
===============================================================
Port a gen
1b5701 membership 01
Port b gen
Port h gen
1b570a membership 01
service vxfen status

running
vxfenadm -d
I/O Fencing Cluster Information:
================================
Fencing Protocol Version: 201
Fencing Mode: Disabled
Cluster Members:
* 0 (sym1)
1 (sym2)
RFSM State Information:
node
0 in state 8 (running)
node
End of Solution
sym3
186 B42

information.
Solution

./installsfha -fencing
Checking communication on sym3
Checking release compatibility on sym3
Checking VCS installation on sym3
Cluster information verification:
Cluster Name: east
Cluster ID Number: 777
Systems: sym3 sym4
Would you like to configure I/O fencing on the cluster? [y,n,q]
y
Checking
Done
Checking
Checking
Checking
Done
Checking
Checking

Fencing is already started in enabled mode, do you want to reconfigure it?

[y,n,q] (y)
y
Fencing configuration
1) Configure Coordination Point client based fencing
2) Configure disk based fencing
3) Configure fencing in disabled mode
4) Online fencing migration
Select the fencing mechanism to be configured in this Application Cluster:
187
3
Installer will stop VCS before applying fencing configuration. To make sure
VCS shuts down successfully, unfreeze any frozen service group and unmount
the mounted file systems in the cluster.
Are you ready to stop VCS and apply fencing configuration on all nodes at
this time? [y,n,q,b] (y)
y
Stopping VCS on sym3 .............................................
Stopping Fencing on sym3 .........................................
Stopping VCS on sym4 .............................................
Stopping Fencing on sym4 .........................................
Configuring fencing in disabled mode on sym3 ......................
Configuring fencing in disabled mode on sym4 ......................
Updating main.cf without fencing .................................
Starting VCS on sym3 .............................................
Starting VCS on sym4 .............................................
Done
Done
Done
Done
Done
Done
Done
Done
Done
I/O Fencing configuration ........................................ Done
B43
I/O Fencing configuration completed successfully

...
/opt/VRTS/install/logs/installsfha-201112081226Zat
Would you like to view the summary file? [y,n,q] (n)
End of Solution

Solution
gabconfig -a
===============================================================
Port a gen
Port b gen
Port h gen
1b570a membership 01
service vxfen status

running
188 B44
vxfenadm -d
================================
Cluster Members:
* 0 (sym3)
1 (sym4)
node
node
End of Solution
Note: In this lab, there is a conflicting duplicate named appsg service group in
both the west and east clusters. This would cause errors during a merge.
The appsg service group in the west cluster would have precedence so the
appsg service group in the east cluster must be renamed or, as performed
in this lab, deleted from the configuration.

Take the appsg service group offline and list its resources and dependencies.
Solution
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
testappsg
testappsg
websg
websg
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
hagrp -dep
hagrp -offline appsg -any

Attempting to offline group on system sym3
hagrp -state appsg

#Group
appsg
appsg
189
Attribute
State
State
System
sym3
sym4
Value
|OFFLINE|
|OFFLINE|

appdg
appmnt
appnic
appproc
appip
appvo
hares -dep | grep "^appsg"

appsg
appsg
appsg
appsg
appsg
appmnt
appproc
appproc
appip
appvol
appvol
appip
appmnt
appnic
appdg
End of Solution
B45
Open the VCS configuration for update and delete each resource in the appsg
service group in resource dependency order. Then, delete the appsg service
group and save and close the VCS configuration.
Solution
haconf -makerw
hares -delete appproc
hares -delete appmnt
hares -delete appvol
hares -delete appdg
hares -delete appip
hares -delete appnic
hagrp -delete appsg
hagrp -state
#Group
ClusterService
ClusterService
dbsg
dbsg
testappsg
testappsg
websg
websg
190 B46
Attribute
State
State
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
End of Solution
Navigate to the /etc/VRTSvcs/conf/config directory and use the

hacf -cftocmd command to create a main.cmd file. Copy the
main.cmd file to eastmain.cmd.
Solution
cd /etc/VRTSvcs/conf/config
hacf -cftocmd .
cp -p main.cmd eastmain.cmd
ls *.cmd
eastmain.cmd
main.cmd
End of Solution
Edit the eastmain.cmd file to remove all commands other than those
related to the websg, testappsg, dbsg service groups, their resources, and their
resource dependencies.
Note: Alternatively, copy the

/student/labs/vcs/vcs60/maincf/CM/eastmain.cmd
file to /etc/VRTSvcs/conf/config/eastmain.cmd.
191
After you have made the edits the contents of the file should be:
hagrp -add dbsg
hagrp -modify dbsg SystemList sym3 0 sym4 1
hagrp -modify dbsg AutoStartList sym3 sym4
hagrp -modify dbsg SourceFile "./main.cf"
hares -add dbdg FileOnOff dbsg
hares -modify dbdg PathName "/var/tmp/dbdg"
hares -modify dbdg Enabled 1
hares -add dbip FileOnOff dbsg
hares -modify dbip PathName "/var/tmp/dbip"
hares -modify dbip Enabled 1
hares -add dblistener FileOnOff dbsg
hares -modify dblistener PathName "/var/tmp/
dblistener"
hares -modify dblistener Enabled 1
hares -add dbmnt FileOnOff dbsg
hares -modify dbmnt PathName "/var/tmp/dbmnt"
hares -modify dbmnt Enabled 1
hares -add dbnic FileOnOff dbsg
hares -modify dbnic PathName "/var/tmp/dbnic"
hares -modify dbnic Enabled 1
hares -add dboracle FileOnOff dbsg
hares -modify dboracle PathName "/var/tmp/
dboracle"
hares -modify dboracle Enabled 1
hares -add dbvol FileOnOff dbsg
B47
192 B48
hares -modify dbvol PathName "/var/tmp/dbvol"

hares -modify dbvol Enabled 1
hagrp -add testappsg
hagrp -modify testappsg SystemList sym3 0 sym4 1
hagrp -modify testappsg AutoStartList sym3 sym4
hagrp -modify testappsg Operators oper
hagrp -modify testappsg SourceFile "./main.cf"
hares -add testappdg FileOnOff testappsg
hares -modify testappdg PathName "/var/tmp/
testappdg"
hares -modify testappdg Enabled 1
hares -add testappmnt FileOnOff testappsg
hares -modify testappmnt PathName "/var/tmp/
testappmnt"
hares -modify testappmnt Enabled 1
hares -add testappnic FileOnOff testappsg
hares -modify testappnic PathName "/var/tmp/
testappnic"
hares -modify testappnic Enabled 1
hares -add testappproc FileOnOff testappsg
hares -modify testappproc PathName "/var/tmp/
testappproc"
hares -modify testappproc Enabled 1
hares -add testappip FileOnOff testappsg
hares -modify testappip PathName "/var/tmp/
testappip"
hares -modify testappip Enabled 1
hares -add testappvol FileOnOff testappsg
hares -modify testappvol PathName "/var/tmp/
testappvol"
hares -modify testappvol Enabled 1
hagrp -add websg
hagrp -modify websg SystemList sym3 0 sym4 1
hagrp -modify websg AutoStartList sym3 sym4
hagrp -modify websg Operators oper
hagrp -modify websg SourceFile "./main.cf"
hares -add webapache FileOnOff websg
hares -modify webapache PathName "/var/tmp/
webapache"
hares -modify webapache Enabled 1

193
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
hares
-add webdg FileOnOff websg

-modify webdg PathName "/var/tmp/webdg"
-modify webdg Enabled 1
-add webmnt FileOnOff websg
-modify webmnt PathName "/var/tmp/webmnt"
-modify webmnt Enabled 1
-add webnic FileOnOff websg
-modify webnic PathName "/var/tmp/webnic"
-modify webnic Enabled 1
-add webvip FileOnOff websg
-modify webvip PathName "/var/tmp/webvip"
-modify webvip Enabled 1
-add webvol FileOnOff websg
-modify webvol PathName "/var/tmp/webvol"
-modify webvol Enabled 1
-link dbip dbnic
-link dblistener dbip
-link dblistener dboracle
-link dbmnt dbvol
-link dboracle dbmnt
-link dbvol dbdg
-link testappmnt testappvol
-link testappproc testappmnt
-link testappproc testappip
-link testappip testappnic
-link testappvol testappdg
-link webapache webmnt
-link webapache webvip
-link webmnt webvol
-link webvip webnic
-link webvol webdg
Solution
vi eastmain.cmd
Remove the appropriate lines.
B49
From the vi editor in command mode, type: :wq
End of Solution
Shutdown VCS, IO fencing, GAB, and LLT on both nodes of the east cluster.
Solution
hastop -all
gabconfig -a
===============================================================
Port a gen
a9e001 membership 01
Port b gen
a9e007 membership 01
service vxfen stop

Stopping vxfen..
Stopping vxfen.. Done
ssh sym4 service vxfen stop

Stopping vxfen..
Stopping vxfen.. Done
service gab stop

Stopping GAB:
ssh sym4 service gab stop

Stopping GAB:
194 B50
service llt stop

Stopping LLT:
ssh sym4 service llt stop

Stopping LLT:
gabconfig -a
GAB gabconfig ERROR V-15-2-25022 unknown error
ssh sym4 gabconfig -a

GAB gabconfig ERROR V-15-2-25022 unknown error

lltstat -nvv configured

LLT lltstat ERROR V-14-2-15000 open /dev/llt failed: No such file or
directory
ssh sym4 lltstat -nvv configured

LLT lltstat ERROR V-14-2-15000 open /dev/llt failed: No such file or
directory
End of Solution
10 Rename the /etc/VRTSvcs/conf/config/main.cf file to
main.cf.east on sym3 and sym4.

Note: In order for the merge utility to work correctly, the main.cf files on
the east cluster nodes must not exist.
Solution
mv main.cf main.cf.east
ls -l main.cf
ls: main.cf No such file or directory
ssh sym4 mv /etc/VRTSvcs/conf/config/main.cf

/etc/VRTSvcs/conf/config/main.cf.east
ssh sym4 ls -l /etc/VRTSvcs/conf/config/main.cf

main.cf: No such file or directory
End of Solution
195
B51

In this exercise, you add the systems from one cluster to another cluster.
sym1
1

run the ./installsfha -addnode command taking note of the merge
pre-requisites and using the following information.
Enter sym1 as the name of one node of the west cluster.

Enter sym3 and sym4 as the systems to be added to the west cluster.
Use eth4 and eth5 as the private links and eth0 as the low priority link.
Use eth0 as the public NIC used for the cluster virtual IP.
Record the log file located in the /opt/VRTS/install/logs
directory. Optionally, review the log file.
Log file:
The log file will be in the form of
installsfha-yyyymmddhhmmxxx
Solution
./installsfha -addnode
Logs are being written to /var/tmp/installsfha-201112081249IHO while
installsfha
is in progress.
Following are the prerequisites to add a node to the cluster:
196 B52
* The cluster to which you want to add the node must have all required
SFHA rpms installed
* New node must have all required SFHA rpms installed
* SFHA must be running on the cluster to which you want to add a node
* There should be no prior running SFHA processes on the new node
* New node must have the same SFHA version as that of the existing
cluster nodes
Refer to the SFHA Installation Guide for more details
Press [Enter] to continue:

Press Enter.
Enter one node of the SFHA cluster to which you would like to add one or
more new nodes:
sym1
Checking communication on sym1 ....................................
Done
Checking release compatibility on sym1 ............................ Done
Following cluster information detected:
Cluster Name: west
Cluster ID: 5
Systems: sym1 sym2
Is this information correct? [y,n,q] (y)
y
Checking communication on sym2 ....................................
Done
Checking VCS running state on sym1 ................................ Done
Checking VCS running state on sym2 ................................ Done
Enter the system names separated by spaces to add to the cluster:
sym3 sym4
Checking
Done
Checking
Checking
Done
Checking

Do you want to add the system(s) sym3 sym4 to the cluster west? [y,n,q] (y)
197
y
Checking
Checking
Checking
Checking
Checking
installed
installed
installed
installed
installed
product on cluster west ............ SFHA 6.0.000.000

product on sym3 .................... SFHA 6.0.000.000
packages on sym3 ............................... Done
product on sym4 .................... SFHA 6.0.000.000
packages on sym4 ............................... Done
Each node to be added to the cluster should have the same LLT heartbeat
configuration as the existing node(s).
To connect to the existing node(s) properly, each new node is required to be
configured with 2 private and 1 low-priority heartbeat links.
Discovering NICs on sym3 ...... Discovered eth0 eth1 eth2 eth3 eth4 eth5
Enter the NIC for the first private heartbeat link on sym3: [q,?] (eth4)
eth4
Enter the NIC for the second private heartbeat link on sym3: [b,q,?] (eth5)
B53
eth5
Enter the NIC for the low-priority heartbeat link on sym3: [b,q,?] (eth0)
eth0
Are you using the same NICs for private heartbeat links on all systems?
[y,n,q,b,?]
y
Checking
Checking
Checking
Checking
Checking
Checking
media
media
media
media
media
media
speed
speed
speed
speed
speed
speed
Private Heartbeat NICs

link1=eth4
link2=eth5
Low-Priority Heartbeat
link-lowpri1=eth0
Private Heartbeat NICs
link1=eth4
link2=eth5
Low-Priority Heartbeat
link-lowpri1=eth0
for
for
for
for
for
for
eth4
eth5
eth4
eth5
eth4
eth5
on
on
on
on
on
on
sym3
sym3
sym4
sym4
sym1
sym1
.........................
.........................
.........................
.........................
.........................
.........................
1000Mb/s
1000Mb/s
1000Mb/s
1000Mb/s
1000Mb/s
1000Mb/s
for sym3:
NIC for sym3:

for sym4:
NIC for sym4:
Is this information correct? [y,n,q] (y)
y
Cluster Virtual IP is configured on the cluster west
A public NIC device is required by following services on each of the newly
added nodes:
Cluster Virtual IP
Active NIC devices discovered on sym3: eth0 eth1 eth2 eth3 eth4 eth0
Enter the NIC for the VCS to use on sym3: (eth0)
198 B54
m eth0
Is eth0 the public NIC used by other new nodes? [y,n,q] (y)
y
Starting llt
Starting
Starting
Starting
Done
Checking
Updating
Updating
Starting
Starting
on sym3 .............................................. Done

llt on sym4 ............................................. Done
gab on sym3 ............................................. Done
gab on sym4 ..............................................
GAB control port seed .......... Checking GAB control port
csgnic resource .........................................
webip resource ..........................................
vxfen on sym3 ...........................................
vxfen on sym4 ...........................................
seed
Done
Done
Done
Done

Starting
Starting
Starting
Starting
amf
had
amf
had
on
on
on
on
sym3
sym3
sym4
sym4
.............................................
.............................................
.............................................
.............................................
Done
Done
Done
Done
Addnode completed successfully

installsfha log files, summary file, and response file are saved at:
/opt/VRTS/install/logs/installsfha-201112081249IHO
End of Solution
Perform a summary status of the cluster noting that none of the former east
cluster service groups have been added.
Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
A
A
RUNNING
RUNNING
RUNNING
RUNNING
0
0
0
0
sym1
sym2
sym3
sym4
-- GROUP STATE
-- Group
System
B
B
B
B
B
B
B
B
B
B
sym1
sym2
sym3
sym4
sym1
sym2
sym1
sym2
sym1
sym2
ClusterService
ClusterService
ClusterService
ClusterService
appsg
appsg
nfssg
nfssg
orasg
orasg
Probed
Y
Y
Y
Y
Y
Y
Y
Y
Y
AutoDisabled
N
N
N
N
N
N
N
N
N
State
ONLINE
OFFLINE
OFFLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
End of Solution
199
Use the gabconfig command to check GAB status.

Solution
gabconfig -a
===============================================================
Port a gen
Port b gen
Port h gen
1b570c membership 0123
End of Solution
B55
Use the vxfenadm command to check I/O fencing.

Solution
vxfenadm -d
================================
Cluster Members:
* 0
1
2
3
(sym1)
(sym2)
(sym3)
(sym4)

node
node
node
node
End of Solution
Use the lltstat command to check the LLT configured and active status.
Optionally, performed this step on all cluster nodes.
Solution
lltstat -nvv configured

LLT node information:
Node
* 0 sym1
1 sym2
200 B56
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
eth4
eth5
eth0
UP
UP
UP
00:0C:29:97:FB:5D
00:0C:29:97:FB:67
00:0C:29:97:FB:35
eth4
eth5
eth0
UP
UP
UP
00:0C:29:C4:1D:0A
00:0C:29:C4:1D:14
00:0C:29:C4:1D:E2
OPEN
2 sym3
OPEN
3 sym4
State
OPEN
OPEN
lltstat -nvv active

LLT node information:
Node
* 0 sym1
State
OPEN
Link
eth4
eth5
Status
UP
UP
Address
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20

1 sym2
eth0
UP
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
eth4
eth5
eth0
UP
UP
UP
00:0C:29:97:FB:5D
00:0C:29:97:FB:67
00:0C:29:97:FB:35
eth4
eth5
eth0
UP
UP
UP
00:0C:29:C4:1D:0A
00:0C:29:C4:1D:14
00:0C:29:C4:1D:E2
OPEN
2 sym3
OPEN
3 sym4
OPEN
End of Solution
Display the contents of the /etc/llttab, /etc/llthosts, and

/etc/gabtab files. Optionally, perform this step on all cluster nodes.
Solution
cat /etc/llthosts
0
1
2
3
sym1
sym2
sym3
sym4
cat /etc/llttab
set-node sym3
set-cluster 5
link eth4 eth-00:0c:29:97:fb:5d - ether - link eth5 eth-00:0c:29:97:fb:67 - ether - link-lowpri eth0 eth-00:0c:29:97:fb:35 - ether - -
cat /etc/gabtab
/sbin/gabconfig -c -n4
End of Solution
201
Display the /etc/VRTSvcs/conf/config/main.cf file noticing the

following elements.
The cluster name is west.

The cluster virtual ip is 10.10.2.51.
All four systems are defined as cluster members.
ClusterService service group configure all four cluster systems.
The appsg, nfssg, and orasg service groups from the original west cluster
are configured and remain restricted to sym1 and sym2.
The websg, dbsg, and testappsg from the east cluster are not configured.
B57
Solution
Note: Only the main.cf file though the ClusterService service group
definition is shown.
more /etc/VRTSvcs/conf/config/main.cf
...
include
include
include
include
include
"OracleASMTypes.cf"
"types.cf"
"Db2udbTypes.cf"
"OracleTypes.cf"
"SybaseTypes.cf"
cluster west (
UserNames = { admin = bQRjQLqNRmRRpZRlQO }
ClusterAddress = "10.10.2.51"
Administrators = { admin }
)
system sym1 (
)
system sym2 (
)
system sym3 (
)
system sym4 (
)
group ClusterService (
SystemList = { sym1 = 0, sym2 = 1, sym3 = 2, sym4 = 3 }
AutoStartList = { sym1, sym2, sym3, sym4 }
OnlineRetryLimit = 3
OnlineRetryInterval = 120
)
202 B58
IP webip (
Device = eth0
Address = "10.10.2.51"
NetMask = "255.255.255.0"
)
NIC csgnic (
Device = eth0
)
NotifierMngr notifier (
SmtpServer = "mgt.example.com"
SmtpServerVrfyOff = 1
SmtpRecipients = { student = Warning }
)
notifier requires csgnic
webip requires csgnic
...
End of Solution


In this exercise, you add the service groups and resources from one of the original
clusters to the merged cluster.
sym3
1
From sym3:terminal1, navigate to the /etc/VRTSvcs/conf/config

directory and open the VCS configuration for update
Solution
cd /etc/VRTSvcs/conf/config
haconf -makerw
End of Solution
Run the sh -x ./eastmain.cmd command and observe the output taking

notice of the VCS warning when failing to add the VCS user named oper as an
Operator for the testappsg and websg service groups.
Note: The oper user add is not part of the eastmain.cmd file and would
need to be re-added and re-configured later on.
Solution
sh -x ./eastmain.cmd
+ hagrp -add dbsg
VCS NOTICE V-16-1-10136 Group added; populating SystemList and setting the
Parallel attribute recommended before adding resources
+ hagrp -modify dbsg SystemList sym3 0 sym4 1
+ hagrp -modify dbsg AutoStartList sym3 sym4
+ hagrp -modify dbsg SourceFile ./main.cf
+ hares -add dbdg FileOnOff dbsg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
+ hares -modify dbdg PathName /var/tmp/dbdg
+ hares -modify dbdg Enabled 1
+ hares -add dbip FileOnOff dbsg
agent monitors
+ hares -modify dbip PathName /var/tmp/dbip
203 Lab 2: Merging clusters
B59
204 B60
+ hares -modify dbip Enabled 1

+ hares -add dblistener FileOnOff dbsg
agent monitors
+ hares -modify dblistener PathName /var/tmp/dblistener
+ hares -modify dblistener Enabled 1
+ hares -add dbmnt FileOnOff dbsg
agent monitors
+ hares -modify dbmnt PathName /var/tmp/dbmnt
+ hares -modify dbmnt Enabled 1
+ hares -add dbnic FileOnOff dbsg
agent monitors
+ hares -modify dbnic PathName /var/tmp/dbnic
+ hares -modify dbnic Enabled 1
+ hares -add dboracle FileOnOff dbsg
agent monitors
+ hares -modify dboracle PathName /var/tmp/dboracle
+ hares -modify dboracle Enabled 1
+ hares -add dbvol FileOnOff dbsg
agent monitors
+ hares -modify dbvol PathName /var/tmp/dbvol
+ hares -modify dbvol Enabled 1
+ hagrp -add testappsg
+ hagrp -modify testappsg SystemList sym3 0 sym4 1
+ hagrp -modify testappsg AutoStartList sym3 sym4
+ hagrp -modify testappsg Operators oper
VCS WARNING V-16-1-50109 User oper is not a valid Cluster user
+ hagrp -modify testappsg SourceFile ./main.cf
+ hares -add testappdg FileOnOff testappsg
agent monitors
+ hares -modify testappdg PathName /var/tmp/testappdg
+ hares -modify testappdg Enabled 1
+ hares -add testappmnt FileOnOff testappsg
agent monitors
+ hares -modify testappmnt PathName /var/tmp/testappmnt
+ hares -modify testappmnt Enabled 1
+ hares -add testappnic FileOnOff testappsg
agent monitors
+ hares -modify testappnic PathName /var/tmp/testappnic
+ hares -modify testappnic Enabled 1
+ hares -add testappproc FileOnOff testappsg
agent monitors
+ hares -modify testappproc PathName /var/tmp/testappproc
+ hares -modify testappproc Enabled 1
+ hares -add testappip FileOnOff testappsg
agent monitors
+ hares -modify testappip PathName /var/tmp/testappip
+ hares -modify testappip Enabled 1
+ hares -add testappvol FileOnOff testappsg
agent monitors
+ hares -modify testappvol PathName /var/tmp/testappvol
+ hares -modify testappvol Enabled 1
+ hagrp -add websg


+ hagrp -modify websg SystemList sym3 0 sym4 1
+ hagrp -modify websg AutoStartList sym3 sym4
+ hagrp -modify websg Operators oper
VCS WARNING V-16-1-50109 User oper is not a valid Cluster user
+ hagrp -modify websg SourceFile ./main.cf
+ hares -add webapache FileOnOff websg
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be
agent monitors
+ hares -modify webapache PathName /var/tmp/webapache
+ hares -modify webapache Enabled 1
+ hares -add webdg FileOnOff websg
agent monitors
+ hares -modify webdg PathName /var/tmp/webdg
+ hares -modify webdg Enabled 1
+ hares -add webmnt FileOnOff websg
agent monitors
+ hares -modify webmnt PathName /var/tmp/webmnt
+ hares -modify webmnt Enabled 1
+ hares -add webnic FileOnOff websg
agent monitors
+ hares -modify webnic PathName /var/tmp/webnic
+ hares -modify webnic Enabled 1
+ hares -add webvip FileOnOff websg
agent monitors
+ hares -modify webvip PathName /var/tmp/webvip
+ hares -modify webvip Enabled 1
+ hares -add webvol FileOnOff websg
agent monitors
+ hares -modify webvol PathName /var/tmp/webvol
+ hares -modify webvol Enabled 1
+ hares -link dbip dbnic
+ hares -link dblistener dbip
+ hares -link dblistener dboracle
+ hares -link dbmnt dbvol
+ hares -link dboracle dbmnt
+ hares -link dbvol dbdg
+ hares -link testappmnt testappvol
+ hares -link testappproc testappmnt
+ hares -link testappproc testappip
+ hares -link testappip testappnic
+ hares -link testappvol testappdg
+ hares -link webapache webmnt
+ hares -link webapache webvip
+ hares -link webmnt webvol
+ hares -link webvip webnic
+ hares -link webvol webdg
set before
set before
set before
set before
set before
set before
End of Solution

Solution

End of Solution
B61

In this exercise, you verify the merged cluster.
sym3
1
From sym3:terminal1, perform a summary status on the cluster taking note of

the service groups that are not online.
Solution
hastatus -sum
206 B62
-- SYSTEM STATE
-- System
State
Frozen
A
A
A
A
RUNNING
RUNNING
RUNNING
RUNNING
0
0
0
0
sym1
sym2
sym3
sym4
-- GROUP STATE
-- Group
System
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
sym1
sym2
sym3
sym4
sym1
sym2
sym3
sym4
sym1
sym2
sym1
sym2
sym3
sym4
sym3
sym4
ClusterService
ClusterService
ClusterService
ClusterService
appsg
appsg
dbsg
dbsg
nfssg
nfssg
orasg
orasg
testappsg
testappsg
websg
websg
Probed
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
AutoDisabled
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
State
ONLINE
OFFLINE
OFFLINE
OFFLINE
ONLINE
OFFLINE
OFFLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
OFFLINE
OFFLINE
OFFLINE
OFFLINE
End of Solution
Display /etc/VRTSvcs/conf/config/main.cf file and notice the

following.
The oper user from the old east cluster is not defined nor is the Operator
service group attribute for the testappsg and websg service groups set to
the oper user.


ClusterService service group have been modified for all four cluster
systems.
The SystemList and AutoStartList service group attributes for the appsg,
nfssg, and orasg service groups remain set to sym1 and sym2.
Note: These service groups cannot be online on sym3 and sym4 due to
shared LUN restrictions.
The SystemList and AutoStartList service group attributes for the websg,
testappsg, and dbsg service groups remain set to sym3 and sym4.
Note: These service groups could be online on sym1 and sym2 and could
be modified to accommodate that.
Note: Consideration needs to be given to activated triggers configured at any

service group and/or resource level if the service group is going to be
re-configured to run on a new cluster system.
Solution
more /etc/VRTSvcs/conf/config/main.cf
...
cluster west (
UserNames = { admin = bQRjQLqNRmRRpZRlQO }
ClusterAddress = "10.10.2.51"
Administrators = { admin }
)
...
group ClusterService (
SystemList = { sym1 = 0, sym2 = 1, sym3 = 2, sym4 = 3 }
AutoStartList = { sym1, sym2, sym3, sym4 }
OnlineRetryLimit = 3
OnlineRetryInterval = 120
)
...
group appsg (
SystemList = { sym1 = 0, sym2 = 1 }
AutoStartList = { sym1, sym2 }
TriggersEnabled @sym1 = { NOFAILOVER, RESFAULT }
TriggersEnabled @sym2 = { NOFAILOVER, RESFAULT }
)
...
group nfssg (
)
...
group orasg (
B63
)
...
group websg (
)
...
group testappsg (
)
...
group dbsg (
...
End of Solution
End of lab
208 B64


In this lab, you use the Veritas Cluster Server Simulator with a preconfigured
main.cf file to show how to configure and test failover policies using the Veritas
Cluster Manager Java Console.

started.
The default priority failover policy is examined.
The load failover policy is examined.
Prerequisites and limits are examined and added to the load failover policy
settings.
terminated.
209 Lab 3: Failover policies
B65

1
opened.
mgt
winclient
Account: Administrator
Password: veritas
terminal2.
210 B66
shutdown sym1, sym2, sym3 and sym4, and power on and log into
winclient.


In this exercise, you examine the simulation used in the lab and start the Veritas
VCS Simulator.
winclient
1
From the winclient desktop, use Windows Explorer to navigate to the

C:\Progam Files (x86)\VERITAS\VCS Simulator\wlm\
conf\config folder.
Solution
From the desktop, double-click the Computer icon.
From the left pane, expand and select Computer > C: >
Program Files (x86) > VERITAS > VCS Simulator > wlm > conf >
config.
End of Solution
Open the main.cf file using the WordPad application.
Solution
211
Right-click the main.cf file and select Open.
From the pop-up window, select Select a program from a list of installed
programs.
Click OK.
Clear Always use the selected program to open this kind of file.
Select WordPad.
Click OK.
End of Solution
B67
Without making any changes, review the contents of the main.cf file. Then,
close the Wordpad and Windows Explorer windows.
Solution
From the Wordpad window, review the contents of the main.cf file.
Click the Close button that appears as an X in the upper right hand corner
of the window.
From the Windows Explorer window, click the Close button that appears
as an X in the upper right hand corner of the window.
End of Solution
Start the Veritas VCS Simulator.

Note: The VCS Simulator may take a moment to display as it discovers any
running simulations even thought there will not be any.
Solution
From the desktop, double-click the Veritas VCS Simulator Java

Console icon.
Note: Alternately, from the start bar, select Start > All programs >
Symantec > Veritas VCS Simulator Java Console.
End of Solution
212 B68
Verify the wlm simulation.

Solution
From the Symantec Veritas Cluster Server Simulation window, from the
left pane list, select wlm.
From the task (right) pane, click Verify Configuration.
From the pop-up window, click OK.
End of Solution

Start the wlm simulation.

Note: The status of the cluster name will have a green check mark upon
successful startup.
Solution
From the left pane list, select wlm.
From the task (right) pane, Click Start Cluster.
End of Solution
Launch the VCS Java Console and log in using the following credentials.
User name: admin
Password: password
Solution
From the task (right) pane, Click Launch Console.
In the User name field, type: admin
In the Password field, type: password
Click OK.
End of Solution
213
B69
From the Java Console, verify that the Member Systems and their Priority
and their AutoStartList attribute are set to the values shown in the following
table.
Service Group
Member Systems (Priority)
AutoStartList
A1
S1 (1), S2 (2), S3 (3), S4 (4)
S1
A2
S1 (1), S2 (2), S3 (3), S4 (4)
S1
B1
S1 (4), S2 (1), S3 (2), S4 (3)
S2
B2
S1 (4), S2 (1), S3 (2), S4 (3)
S2
C1
S1 (3), S2 (4), S3 (1), S4 (2)
S3
C2
S1 (3), S2 (4), S3 (1), S4 (2)
S3
D1
S1 (2), S2 (3), S3 (4), S4 (1)
S4
D2
S1 (2), S2 (3), S3 (4), S4 (1)
S4
Solution
From the left pane, select wlm > A1.
From the right pane, select the Properties tab.
Verify the AutoStartList attribute.
From the System Details section, verify Member Systems and their
Priority.
Repeat steps a through d for the A2, B1, B2, C1, C2, D1 and D2 service
groups.
End of Solution
214 B70


In this exercise, you examine the default priority failover policy.
winclient
1
From the Java Console, verify that the failover policy of all service groups is
Priority.
Solution
From the right pane, verify that the FailOverPolicy attribute is set to
Priority.
Repeat steps a and b for the A2, B1, B2, C1, C2, D1 and D2 service
groups.
End of Solution
215
Service Group
System S1
A1
Online
A2
Online
System S2
B1
Online
B2
Online
System S3
C1
Online
C2
Online
System S4
D1
Online
D2
Online
Solution
From the left pane, select wlm.
B71
From the right pane, select the Status tab.

Note: This view is referred to as the Cluster Status view.
Verify the service groups are online as shown in the above table.
End of Solution
S2
Solution
From left pane, expand wlm > A1 > FileOnOff > A1File.
Right-click wlm > A1 > FileOnOff > A1File and select

Fault Resource > S1.
From the pop-up window, click Yes.
From the right pane, notice that the A1 service group shows faulted on S1,
but Online on S2.
End of Solution
216 B72
If the A1 service group faults again without clearing the previous fault, where
should it fail over? Verify the failover by faulting a critical resource in the A1
service group.
S3
Solution
From the left pane, right-click wlm > A1> FileOnOff > A1File and select

and S2 but Online on S3.
End of Solution
Clear the faults in the A1 service group. Where will the service group fail over
to now? Verify the failover by faulting a critical resource in the A1 service
group.
S1
Solution
From the left pane, right-click wlm > A1 > FileOnOff > A1File and select
Clear Fault > Auto.

but Online on S1.
End of Solution
Clear the faults in the A1 service group.

Solution
From the left pane, right-click wlm > A1 > FileOnOff > A1File and select
Clear Fault > Auto.
217
End of Solution
B73

In this exercise, you examine the load failover policy.
winclient
1
Solution
From the menu, select File > Open Configuration.

End of Solution
Set the FailOverPolicy attribute to Load for the eight service groups.
Solution
218 B74
To the right of the FailOverPolicy attribute click the Edit button.
From the Edit Attribute window, from the Scaler Value drop-down menu,
select Load.
Note: The drop-down menu will appear when you click in the Scaler
Value field.
Click OK.
Repeat steps a through e for the A2, B1, B2, C1, C2, D1 and D2 service
groups.
End of Solution

Set the Load attribute for each service group as shown in the following table.
Service Group
Load
A1
75
A2
75
B1
75
B2
75
C1
50
C2
50
D1
50
D2
50
Solution
219
Click Show all attributes.
Scroll to the Load attribute and click the Edit icon.
From the Edit Attribute window, in the Scaler Value field, type: 75
Click OK.
Close the Attributes View window.
Repeat steps a through g for the A2, B1, B2, C1, C2, D1 and D2 service
groups using the Load value shown in the table above.
End of Solution
Set the S1 and S2 systems Capacity attributes to 200.

Note: The S3 and S4 systems Capacity attribute will not be changed and
defaults to 100.
Solution
From the left pane, select the Systems tab.
B75
Select wlm > S1.
Scroll to the Capacity attribute and click the Edit icon.
From the pop-up window, in the Scaler Value field, type: 200
Click OK.
Repeat steps a through h for the S2 system.
End of Solution

following table.
Service Group
AvailableCapacity
S1
50
S2
50
S3
S4
Solution
220 B76
From the left pane, select wlm > S1.
Verify that the AvaliableCapacity is set to the value shown in the table
above.
Repeat steps a through e for the S2, S3 and S4 systems.
End of Solution


Solution
From the menu, select File >Save Configuration.

End of Solution
Service Group
System S1
A1
Online
A2
Online
System S2
B1
Online
B2
Online
System S3
C1
Online
C2
Online
System S4
D1
Online
D2
Online
Solution
221
From the left pane, select the Service Groups tab.
Select wlm.
Verify the service groups are online as shown in the above table.
End of Solution
S2
Solution
From left pane, expand wlm > A1 > FileOnOff > A1File.
B77

but Online on S2.
End of Solution

following table.
Service Group
AvailableCapacity
S1
125
S2
-25
S3
S4
Solution
222 B78
Select wlm > S1.
above.
Repeat steps b through e for the S2, S3 and S4 systems.
End of Solution

10 If the S2 system fails, where will the A1, B1 and B2 service groups that are
Cluster Manager.
S3
S1
S1
Solution
From the left pane, right-click wlm > S2 and select on Power Off.
Select wlm.
Notice that the A1 service group shows faulted on S1, but Online on S3.
Notice that the B1 and B2 service groups show Online on S1.
End of Solution
following table.
Service Group
AvailableCapacity
S1
-25
S2
200
S3
-75
S4
Solution
From the left pane, select wlm > S1.
above.
B79
Repeat steps b through f for the S2, S3 and S4 systems.
End of Solution
12 Power up the S2 system, clear all faults, and return the service groups to their
startup locations.
Solution
From the left pane, right-click wlm > S2 and select Up > S2.
Select the Service Groups tab.
Right-click wlm > A1 and select Clear Fault > Auto.
Right-click wlm > A1 and select Switch To > S1.
Right-click wlm > B1 and select Switch To > S2.
Right-click wlm > B2 and select Switch To > S2.
Solution
following table.
224 B80
Service Group
AvailableCapacity
S1
50
S2
50
S3
S4
Solution
Select wlm > S1.

above.
Repeat steps b through f for the S2, S3 and S4 systems.
End of Solution
14 Save and close the configuration changes.

Solution
From the menu, select File >Close Configuration.
End of Solution
B81

In this exercise, you examine the prerequisites and limits and add them to the load
failover policy settings.
winclient
1
Solution
From the menu, select File > Open Configuration.

End of Solution
Set the Limits attribute for each system to the key ABGroup with a value of 3.
Solution
226 B82
Select wlm > S1.
Click Show all Attributes.
Scroll to the Limits attribute and click the Edit icon.
From the Edit Attribute window, click the add an element button.
In the Key field, type: ABGroup
In the Value field, type: 3
Click OK.
Repeat steps b through j for the S2, S3 and S4 systems.
End of Solution

Set the Prerequisites attribute for the A1, A2, B1 and B2 service groups to be
ABGroup with a value of 1.
CAUTION Do not modify the Prerequisites attribute for the C1, C2, D1,
and D2 service groups.
Solution
From the left pane, select the Service Groups tab.
Select wlm > A1.
Click Show all Attributes.
Scroll to the Prerequisites attribute and click the Edit icon.
From the Edit Attribute window, click the add an element button.
In the Key field, type: ABGroup
In the Value field, type: 1
Click OK.
Repeat steps b through j for the A2, B1 and B2 service groups.
End of Solution

Solution
From the menu, select File >Save Configuration.

End of Solution
B83
If the S1 system fails, where will the A1 and A2 service groups that are Online
on S1 fail over? Verify this failover by powering off the S1 system in Cluster
Manager. Explain the failover actions that occurred.
S2
S3
A1 failed over to S2, but A2 failed over to S3 because the limit was reached on S2.
Solution
Select wlm.
Select wlm.
Notice that the A1 service group shows Online on S2.
Notice that the A2 service group show Online on S3.
End of Solution
228 B84
If the S2 system fails, where will the A1, B1 and B2 service groups that are
Cluster Manager. Explain the failover actions that occurred.
S4
S3
S4
A1 failed over to S4, B1 failed over to S3, and B2 failed over to S4 due to the values
of the Load attribute on each system.

Solution
Select wlm.
Notice that the B1 service group show Online on S3.
Notice that the B2 service group show Online on S4.
End of Solution
If the S3 system fails, where will the A2, B1, C1 and C2 service groups that
are Online on S3 fail over? Verify this failover by powering off the S3 system
in Cluster Manager. Explain the failover actions that occurred.
S4
B1 does not fail over and stays offline
S4
S4
All service groups fail over to S4 except B1. B1 is the last group to attempt to fail over
to S4. Since B1 has a prerequisite that cannot be met on S4, it cannot fail over. A1,
A2, B1, and B2 can run on the same system. B1 stays offline.
Solution
Select wlm.
Notice that the B1 service group shows Offline on all systems.
B85
Notice that the C1 service group show Online on S4.
Notice that the C2 service group show Online on S4.
End of Solution
Save and close the configuration changes.

Solution
From the menu, select File >Close Configuration.
End of Solution
230 B86


In this exercise, you terminate the simulation and the Veritas VCS Simulator.
winclient
1
Log out from Cluster Explorer.

Solution
From the menu, select File >Log Out.

End of Solution
From the Simulator Java Console, stop the wlm cluster.

Solution
From the task (right) pane, click Stop Cluster.
End of Solution
231
Verify the configuration.

Note: If you intend to move this configuration in to a running cluster, you
must ensure the main.cf syntax is correct. You can verify the
configuration only in the Simulator GUI after the cluster is stopped.
Solution
From the task (right) pane, click Verify Configuration.
From the pop-up window, click OK.
End of Solution
B87
Close the Simulator Java Console.

Note: The Veritas Cluster Manager Java Console should close as well. If it
does not then close it manually.
Solution
Click the Close button that appears as an X in the upper right hand corner
of the window.
End of Solution
End of lab
232 B88


In this lab application networking is reconfigured to be more efficient.
A common parallel network service group and resources is configured.

The pre-existing resources of type NIC supporting application VIPs in each
application service group are replaced with resources of type Proxy.
B89

234 B90
opened.
mgt
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas
sym3
Account: root
Password: veritas
sym4
Account: root
Password: veritas
terminal2.

shutdown winclient, and power on and log into sym1, sym2, sym3
and sym4.
B91

In this exercise, you configure a common parallel network service group and
resources.
sym1
1
From sym1:terminal1, use the haconf command to open the cluster

configuration for update.
Solution
haconf -makerw
End of Solution
Use the hagrp command to create a service group named netsg.

Solution
hagrp -add netsg

Solution
236 B92
Modify the SystemList to allow the netsg service group to run on the sym1,
sym2, sym3, and sym4 in that order.
Solution
hagrp -modify netsg SystemList sym1 0 sym2 1

sym3 2 sym4 3
End of Solution

Modify the AutoStartList attribute to allow the service group to start on all
systems.
Solution
hagrp -modify netsg AutoStartList sym1 sym2 sym3

sym4
End of Solution
Modify the Parallel service group to configure the netsg service group as a
parallel service group.
Solution
hagrp -modify netsg Parallel 1

End of Solution
Display the service group attributes for the netsg service group to confirm your
input and key default service group attribute values.
Solution
hagrp -display netsg | more

#Group
netsg
netsg
netsg
netsg
netsg
netsg
netsg
netsg
netsg
netsg
...
netsg
...
netsg
sym3
...
Attribute
AdministratorGroups
Administrators
Authority
AutoFailOver
AutoRestart
AutoStart
AutoStartIfPartial
AutoStartList
AutoStartPolicy
ClusterFailOverPolicy
System
global
global
global
global
global
global
global
global
global
global
Value
Parallel
global
SystemList
sym4
global
0
1
1
1
1
sym1
Order
Manual
sym1
sym2
sym3
sym4
sym2
End of Solution
B93
Save the VCS configuration, but do not close it.

Solution
haconf -dump
End of Solution
Add a resource of type NIC named netnic to the netsg service group.
Solution
hares -add netnic NIC netsg

agent monitors
End of Solution
Modify the Device and Critical resource attributes for the netnic resource
using the following information.
Device is eth0
Critical set to 0 (zero)
Solution
hares -modify netnic Device eth0
hares -modify netnic Critical 0
End of Solution
10 Display the resource attribute values for the netnic resource to confirm your
input and key default resource attribute values.
238 B94
Solution
hares -display netnic | more

#Resource
netnic
netnic
netnic
netnic
netnic
...
netnic
...
Attribute
Group
Type
AutoStart
Critical
Enabled
System
global
global
global
global
global
Value
netsg
NIC
1
0
0
Device
global
eth0
End of Solution

11 Enable the netnic resource and verify that it is enabled. Display the state of the
netnic resource to ensure that it is online on both cluster systems.

Note: It is not necessary to bring a resource of type NIC online.
Solution
hares -modify netnic Enabled 1
hares -value netnic Enabled

1
hares -state netnic

#Resource
netnic
netnic
netnic
netnic
Attribute
State
State
State
State
System
sym1
sym2
sym3
sym4
Value
ONLINE
ONLINE
ONLINE
ONLINE
End of Solution
Solution
hagrp -state netsg

#Group
netsg
netsg
netsg
netsg
Attribute
State
State
State
State
System
sym1
sym2
sym3
sym4
Value
|OFFLINE|
|OFFLINE|
|OFFLINE|
|OFFLINE|
End of Solution

Solution
haconf -dump
End of Solution
B95
14 Add a resource of type Phantom named netphantom to the netsg service
group.
Solution
hares -add netphantom Phantom netsg

agent monitors
End of Solution
15 Set the Critical resource attribute to 0 (zero) for the netphantom resource.
Solution
hares -modify netphantom Critical 0

End of Solution
16 Display the resource attribute values for the netphantom resource to confirm
your input and key default resource attribute values.

Solution
hares -display netphantom | more

#Resource
netphantom
netphantom
netphantom
netphantom
netphantom
...
Attribute
Group
Type
AutoStart
Critical
Enabled
System
global
global
global
global
global
Value
netsg
Phantom
1
0
0
End of Solution
240 B96
17 Enable the netphantom resource and verify that it is enabled. Display the state
of the netphantom resource to ensure that it is online on both cluster systems.

Note: It is not necessary to bring a resource of type Phantom online.
Solution
hares -modify netphantom Enabled 1
hares -value netphantom Enabled

1

hares -state netphantom

#Resource
netphantom
netphantom
netphantom
netphantom
Attribute
State
State
State
State
System
sym1
sym2
sym3
sym4
Value
ONLINE
ONLINE
ONLINE
ONLINE
End of Solution
Solution
hagrp -state netsg

#Group
netsg
netsg
netsg
netsg
Attribute
State
State
State
State
System
sym1
sym2
sym3
sym4
Value
|ONLINE|
|ONLINE|
|ONLINE|
|ONLINE|
End of Solution

Solution
haconf -dump
End of Solution
B97

In this exercise, you replace the pre-existing resources of type NIC that supporting
application VIPs in each application service group with resources of type Proxy.
sym1
1
From sym1:terminal1, identify the names of the resources of type NIC that are
configured.
Solution
hares -list Type=NIC

appnic
appnic
csgnic
csgnic
csgnic
csgnic
netnic
netnic
netnic
netnic
nfsnic
nfsnic
oranic
oranic
sym1
sym2
sym1
sym2
sym3
sym4
sym1
sym2
sym3
sym4
sym1
sym2
sym1
sym2
End of Solution
242 B98
Ignoring the resource of type NIC named netnic, list the resource
dependencies associated with the listed resources of type NIC.
Solution
hares -dep appnic csgnic nfsnic oranic

#Group
ClusterService
ClusterService
appsg
nfssg
orasg
Parent
notifier
webip
appip
nfsip
oraip
Child
csgnic
csgnic
appnic
nfsnic
oranic
End of Solution

Replace the resource of type NIC named appnic with a resource of type Proxy
named appproxy using the following information.

Link to the appip resource.
Solution
hares -delete appnic
hares -add appproxy Proxy appsg

agent monitors
hares -modify appproxy Critical 0
hares -modify appproxy TargetResName netnic
hares -display appproxy | more

#Resource
appproxy
appproxy
appproxy
appproxy
appproxy
...
appproxy
...
Attribute
Group
Type
AutoStart
Critical
Enabled
System
global
global
global
global
global
Value
appsg
Proxy
1
0
0
TargetResName
global
netnic
hares -modify appproxy Enabled 1
hares -value appproxy Enabled
hares -state appproxy

#Resource
appproxy
appproxy
Attribute
State
State
System
sym1
sym2
hares -link appip appproxy
hares -dep | grep appproxy

appsg
appip
Value
ONLINE
ONLINE
appproxy
End of Solution
B99
Replace the resource of type NIC named csgnic with a resource of type Proxy
named csgproxy using the following information.

Link to the notifier and webip resources.
Solution
hares -delete csgnic
hares -add csgproxy Proxy ClusterService

agent monitors
hares -modify csgproxy Critical 0
hares -modify csgproxy TargetResName netnic
hares -display csgproxy | more

#Resource
csgproxy
csgproxy
csgproxy
csgproxy
csgproxy
...
csgproxy
...
Attribute
Group
Type
AutoStart
Critical
Enabled
System
global
global
global
global
global
Value
ClusterService
Proxy
1
0
0
TargetResName
global
netnic
hares -modify csgproxy Enabled 1
hares -value csgproxy Enabled
244B100
hares -state csgproxy

#Resource
csgproxy
csgproxy
csgproxy
csgproxy
Attribute
State
State
State
State
System
sym1
sym2
sym3
sym4
hares -link webip csgproxy
hares -link notifier csgproxy
Value
ONLINE
ONLINE
ONLINE
ONLINE

hares -dep | grep csgproxy

ClusterService notifier
ClusterService webip
csgproxy
csgproxy
End of Solution
Replace the resource of type NIC named nfsnic with a resource of type Proxy
named nfsproxy using the following information.

Link to the nfsip resource.
Solution
hares -delete nfsnic
hares -add nfsproxy Proxy nfssg
agent monitors
hares -modify nfsproxy Critical 0
hares -modify nfsproxy TargetResName netnic
hares -display nfsproxy | more

#Resource
nfsproxy
nfsproxy
nfsproxy
nfsproxy
nfsproxy
...
nfsproxy
...
Attribute
Group
Type
AutoStart
Critical
Enabled
System
global
global
global
global
global
Value
nfssg
Proxy
1
0
0
TargetResName
global
netnic
hares -modify nfsproxy Enabled 1
hares -value nfsproxy Enabled

1
hares -state nfsproxy

#Resource
nfsproxy
nfsproxy
Attribute
State
State
System
sym1
sym2
Value
ONLINE
ONLINE
B101
hares -link nfsip nfsproxy
hares -dep | grep nfsproxy

nfssg
nfsip
nfsproxy
End of Solution
Replace the resource of type NIC named oranic with a resource of type Proxy
named oraproxy using the following information.

Link to the oraip resource.
Solution
hares -delete oranic
hares -add oraproxy Proxy orasg
agent monitors
246 B102
hares -modify oraproxy Critical 0
hares -modify oraproxy TargetResName netnic
hares -display oraproxy | more

#Resource
oraproxy
oraproxy
oraproxy
oraproxy
oraproxy
...
oraproxy
...
Attribute
Group
Type
AutoStart
Critical
Enabled
System
global
global
global
global
global
Value
orasg
Proxy
1
0
0
TargetResName
global
netnic
hares -modify oraproxy Enabled 1
hares -value oraproxy Enabled

1
hares -state oraproxy

#Resource
oraproxy
oraproxy
Attribute
State
State
System
sym1
sym2
Value
ONLINE
ONLINE

hares -link oraip oraproxy
hares -dep | grep oraproxy

orasg
oraip
oraproxy
End of Solution

Solution

End of Solution
End of lab
B103
248 B104

Appendix C
Supplemental Content
249
Service group dependenciesFailover process
250 C2

251
The following steps describe what happens when a service group in a service
group dependency relationship is faulted due to a critical resource fault:
1 The entire service group is taken offline due to the critical resource fault
together with any of its parent service groups that have an online firm or hard
dependency (online local firm, online global firm, online remote firm, or
online local hard).
2 Then a failover target is chosen from the SystemList of the service group based
on the failover policy and the restrictions brought by the service group
dependencies. Note that if the faulted service group is also the parent service
group in a service group dependency relationship, the service group
dependency has an impact on the choice of a target system. For example, if the
faulted service group has an online local (firm or soft) dependency with a child
service group that is online only on that system, no failover targets are
available.
3 If there are no other systems to which the service group can fail over, both the
child service group and all of the parents that were already taken offline remain
offline.
4 If there is a failover target then VCS takes any child service group with an
online local hard dependency offline.
5 VCS then checks if there are any conflicting parent service groups that are
already online on the target system. These service groups can be parent service
groups that are linked with an offline local dependency or online remote soft
dependency. In either case, the parent service group is taken offline to enable
the child service group to start on that system.
6 If there is any child service group with an online local hard dependency, first
the child service group and then the service group that initiated the failover are
brought online.
7 After the service group is brought online successfully on the target system,
VCS takes any parent service groups offline that have an online local soft
dependency to the failed-over child.
8 Finally, VCS selects a failover target for any parent service groups that may
have been taken offline during steps 1, 5, or 7 and brings the parent service
group online on an available system.
9 If there are no target systems available to fail over the parent service group that
has been taken offline, the parent service group remains offline.
Appendix C Supplemental Content

C3
252 C4


68-Veritas Cluster Server 6.0 For UNIX Cluster Management

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

68-Veritas Cluster Server 6.0 For UNIX Cluster Management

Hochgeladen von

Copyright:

Verfügbare Formate

Veritas Cluster

Server 6.0 for UNIX:

LEAD SUBJECT MATTER

Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Lesson 5: High Availability in the Enterprise

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Cluster Server 6.0 for UNIX: Cluster Management

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Cluster Server curriculum path

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Cluster Server 6.0 for UNIX: Cluster Management

Lab design for the course

Copyright 2012 Symantec Corporation. All rights reserved.

The cluster consists of:

Additional complexity is added to the design to illustrate certain aspects of cluster

Copyright 2012 Symantec Corporation. All rights reserved.

This training provides comprehensive instruction on the deployment of advanced

Veritas Cluster Server 6.0 for UNIX: Cluster Management

Copyright 2012 Symantec Corporation. All rights reserved.

Service Group Dependencies

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Cluster Server 6.0 for UNIX: Cluster Management

Common application relationships

Copyright 2012 Symantec Corporation. All rights reserved.

The example in the slide shows a typical three-tier model where:

Lesson 1 Service Group Dependencies

Example 1: Online on the same system

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Cluster Server 6.0 for UNIX: Cluster Management

Example 2: Online on different systems

Copyright 2012 Symantec Corporation. All rights reserved.

Lesson 1 Service Group Dependencies

Example 3: Offline on the same system

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Cluster Server 6.0 for UNIX: Cluster Management

Service group dependencies

Copyright 2012 Symantec Corporation. All rights reserved.

Service group dependency definitions

Lesson 1 Service Group Dependencies

Veritas Cluster Server 6.0 for UNIX: Cluster Management

Service group dependency examples

Copyright 2012 Symantec Corporation. All rights reserved.

Online local firm

Lesson 1 Service Group Dependencies

Online remote example

Veritas Cluster Server 6.0 for UNIX: Cluster Management

Offline local example

Copyright 2012 Symantec Corporation. All rights reserved.

Lesson 1 Service Group Dependencies

Configuring service group dependencies

Copyright 2012 Symantec Corporation. All rights reserved.

You can use service group dependencies to implement parent/child relationships

Veritas Cluster Server 6.0 for UNIX: Cluster Management

Creating service group dependencies

Copyright 2012 Symantec Corporation. All rights reserved.

Lesson 1 Service Group Dependencies

Modeling service group dependencies

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Cluster Server 6.0 for UNIX: Cluster Management

Online and offline of linked service groups

Copyright 2012 Symantec Corporation. All rights reserved.