Appendixes B and C

Veritas Cluster
Server 6.0 for UNIX:

Install and Configure
Appendixes B and C
100-002685-C
COURSE DEVELOPERS
Bilge Gerrits
Steve Hoffer
Siobhan Seeger
Pete Toemmes
LEAD SUBJECT MATTER

EXPERTS
Graeme Gofton
Sean Nockles
Brad Willer
TECHNICAL
CONTRIBUTORS AND
REVIEWERS
Copyright 2012 Symantec Corporation. All rights reserved.
Geoff Bergren
Kelli Cameron
Tomer Gurantz
Anthony Herr
James Kenney
Gene Henriksen
Bob Lucas
Paul Johnston
Rod Pixley
Clifford Barcliff
Danny Yonkers
Antonio Antonucci
Satoko Saito
Feng Liu

Symantec, the Symantec Logo, and VERITAS are trademarks or
registered trademarks of Symantec Corporation or its affiliates in
the U.S. and other countries. Other names may be trademarks of
their respective owners.
THIS PUBLICATION IS PROVIDED AS IS AND ALL
EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS
AND WARRANTIES, INCLUDING ANY IMPLIED
WARRANTY OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE
DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH
DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.
SYMANTEC CORPORATION SHALL NOT BE LIABLE FOR
INCIDENTAL OR CONSEQUENTIAL DAMAGES IN
CONNECTION WITH THE FURNISHING, PERFORMANCE,
OR USE OF THIS PUBLICATION. THE INFORMATION
CONTAINED HEREIN IS SUBJECT TO CHANGE WITHOUT
NOTICE.
No part of the contents of this book may be reproduced or
transmitted in any form or by any means without the written
permission of the publisher.
Veritas Cluster Server 6.0 for UNIX: Install and Configure
Symantec Corporation
World Headquarters
350 Ellis Street
Mountain View, CA 94043
United States
http://www.symantec.com
Table of Contents
Appendix B: Lab Solutions

Lab 2: Intro Lab A: Using VMware Workstation in a classroom .................. B-8
Exercise 1: Starting virtual machines (VMware Workstation) ............... B-12
Exercise 2: Logging on to virtual machines (VMware Workstation)...... B-17
Exercise 3: Adjusting the VMware view (VMware Workstation) ........... B-20
Exercise 4: Running basic commands (VMware Workstation) ............. B-22
Exercise 5: Shutting down virtual machines (VMware Workstation)..... B-31
Lab 2: Intro Lab B: Hatsize introduction ..................................................... B-32
Exercise 1: Connecting to the lab environment (Hatsize) ..................... B-35
Exercise 2: Connecting to additional virtual machines (Hatsize) .......... B-38
Exercise 3: Running basic commands (Hatsize) .................................. B-41
Lab 3: Validating site preparation............................................................... B-51
Exercise 1: Checking lab prerequisites................................................. B-52
Exercise 2: Verifying the network configuration .................................... B-53
Exercise 3: Performing a CPI pre-installation verification check........... B-56
Exercise 4: Performing a SORT pre-installation verification check....... B-59
Lab 4: Installing Storage Foundation HA 6.0.............................................. B-69
Exercise 1: Checking lab prerequisites (part a) .................................... B-71
Exercise 2: Installing SFHA using the CPI............................................ B-73
Exercise 3: Running a post installation check....................................... B-83
Exercise 4: Adding a cluster to VOM .................................................... B-85
Exercise 5: Checking lab prerequisites (part b) .................................... B-88
Exercise 6: Upgrading SFHA using the CPI ......................................... B-89
Exercise 7: Examining the VCS configuration .................................... B-103
Lab 5: Performing common VCS operations............................................ B-107
Exercise 1: Checking lab prerequisites............................................... B-109
Exercise 2: Orientation........................................................................ B-110
Exercise 3: Viewing status and attributes ........................................... B-113
Exercise 4: Manipulating service groups ............................................ B-118
Exercise 5: Manipulating resources .................................................... B-122
Exercise 6: Modifying resource attributes ........................................... B-128
Exercise 7: Adding a service group .................................................... B-130
Exercise 8: Copying a service group .................................................. B-136
Lab 6: Starting and stopping VCS ............................................................ B-141
Exercise 2: VCS configuration state and stopping VCS ..................... B-143
Exercise 3: Configuring automatic backup of the VCS configuration.. B-146
Exercise 4: Setting non default VCS stop options .............................. B-148
Lab 7: Preparing application services ...................................................... B-157
Exercise 2: Examining storage for the service.................................... B-159
Exercise 3: Examining the application ................................................ B-161
Exercise 4: Manually starting and stopping the application ................ B-163
Lab 8: Online configuration of a service group......................................... B-167
Table of Contents
i
Exercise 1: Checking lab prerequisites .............................................. B-168

Exercise 2: Creating a service group for the loopy application .......... B-169
Exercise 3: Configuring resources for the loopy application .............. B-171
Exercise 4: Performing a virtual fire drill on the service group ........... B-184
Exercise 5: Testing the service group ................................................ B-187
Exercise 6: Setting resources to critical ............................................. B-189
Exercise 7: (Optional) Examining Veritas File System locking........... B-192
Lab 9: Offline configuration...................................................................... B-199
Exercise 2: Editing a copy of the main.cf file using a system editor... B-201
Exercise 3: Stopping VCS .................................................................. B-205
Exercise 4: Restarting VCS using the edited main.cf file ................... B-207
Lab 10: Configuring notification ............................................................... B-215
Exercise 2: Preparing the email notification reader............................ B-218
Exercise 3: Configuring and testing the notifier using VOM ............... B-219
Exercise 4: Configuring trigger scripts................................................ B-227
Lab 11: Configuring resource fault behavior............................................ B-239
Exercise 2: Observing non-critical resource faults ............................. B-241
Exercise 3: Observing critical resource faults .................................... B-248
Exercise 4: (Optional) Observing faults in frozen service groups....... B-256
Exercise 5: (Optional) Observe ManageFaults behavior.................... B-265
Exercise 6: (Optional) Observing restart limit behavior ...................... B-274
Lab 12: IMF and AMF .............................................................................. B-285
Exercise 2: Examining IMF monitoring on a resource........................ B-287
Exercise 3: (Optional) Examining the IMF default configuration ........ B-294
Lab 13: Cluster communications ............................................................. B-299
Exercise 2: Reconfiguring LLT ........................................................... B-301
Exercise 3: Jeopardy membership ..................................................... B-313
Lab 14: Configuring SCSI3 disk-based I/O fencing ................................. B-327
Exercise 2: Fencing configuration pre-checks.................................... B-329
Exercise 3: Configuring and testing disks for I/O fencing................... B-332
Exercise 4: Configuring VCS for I/O fencing ...................................... B-338
Exercise 5: I/O fencing configuration verification ............................... B-342
Exercise 6: Verifying data disks for I/O fencing.................................. B-348
Exercise 7: Testing communication failures ....................................... B-352
Lab 15: Configuring CPS-based I/O fencing............................................ B-387
Exercise 2: Configuring a single node CP server............................... B-390
Exercise 3: Verifying a CP server configuration ................................. B-398
Exercise 4: Configuring for I/O fencing with a CP server .................... B-411
Exercise 5: Testing CP server communication failure........................ B-432
Exercise 6: Testing CP server caching .............................................. B-440
ii

Exercise 7: Configuring back to SCSI3 disk-based only I/O fencing .. B-448
Appendix C: Supplemental Content

VCS Simulator configuration ....................................................................... C-2
Configuring IP addresses ............................................................................ C-5
Communication configuration file examples............................................... C-10
Troubleshooting VCS ................................................................................. C-19
Table of Contents
iii
iv

Appendix B
Lab Solutions
B2

Lab environment
Whether you are taking the Veritas Cluster Server 6.0 for UNIX: Administration
course in a physical classroom environment where you are provided a physical
computer to perform the labs or you are taking the course in a virtual classroom
where you are connected via the internet to a virtual lab environment, the labs are
performed within a virtual environment consisting of eight virtual machines
(VMs). This virtual lab environment has been developed specifically for this
course and is detailed in the next sections.
The eight virtual machines are as follows:

mgt: A RedHat Enterprise Linux 5.5 (RHEL55) VM with 1 GB RAM and one
CPU which provides DNS and NTP services for the VM configuration. This
VM also hosts the Symantec software and lab files needed to perform the
remaining labs. It also provides 2 GB SCSI3 PGR compliant iSCSI based emc
simulated LUNS as shared storage for the cluster VMs. There are
preconfigured Storage Foundation (SF) storage objects for the sym1-sym2
cluster systems which will be identified and explained in a later lab. Also, the
"student" user is hosted on this server. This user receives email notifications
from VCS in a later lab.
winclient: A Windows 2008 R2 Enterprise (W2K8) VM with 1 GB RAM and
one CPU to provide the Windows version of the Veritas Cluster Manager
Java Console for VCS 6.0.
vom: A RHEL55 VM with 1 GB RAM and one CPU which will be configured
as the Veritas Operations Manager 4.1 (vom41) Management Server.
B3
The cluster systems and target cluster systems have all required patches installed
for Storage Foundation HA 6.0. Additionally, required patches and OS
configuration has been done on the sym1, sym2, sym3, and sym4 systems for
Oracle 11.2.0.2.
10
cps: A RHEL55 VM with 1 GB RAM and one CPU, which has a

preconfigured single node VCS 6.0 cluster and which will further be
configured in later labs as a Coordination Point (CP) server.
sym1 and sym2: RHEL55 VMs with 2 GB RAM and two CPUs which serve
as one set of two-node cluster systems to be configured in a later lab. Each
cluster system has:
One 25 GB local boot disk
A second 25 GB internal non shared disk
Twelve 2 GB emc LUNs shared between cluster systems to:
Experiment with various storage configurations in later labs
Accommodate configuring I/O fencing.
Network interfaces:
Two private network interfaces for the cluster interconnect
Two public network interfaces
Other network interfaces used for dual access to the emc LUNs
sym3 and sym4: RHEL55 VMs with 2 GB RAM and two CPUs which serve
as another set of two-node cluster systems that have been pre-configured as a
SFHA5.1SP1 (VCS51.SP1) cluster. Each cluster system has:
One 25 GB local boot disk
A second 25 GB internal non shared disk
Twelve 2 GB emc LUNs shared between cluster systems to:
Experiment with various storage configurations in later labs
Accommodate when I/O fencing is pre-configured and enabled
Network interfaces:
Two private network interfaces for the cluster interconnect
Two public network interfaces
Other network interfaces used for dual access to the emc LUNs
Note: Note: In the context of the labs for this course, the term virtual machine or
VM is used when a specific VMware action is taken, such as turning the
power on or taking a snapshot of a VM. The terms system, node and host
are used to refer to activity being performed on the host operating system
within the VM, such as navigating to a terminal window, installing
software, checking status, and so on. While most labs adhere to this
convention, the terms VM, system, node, and host can be considered
interchangeable.
B4
Veritas Cluster Server 6.0 for UNIX: Administration

Virtual machine configuration
The labs in the Veritas Cluster Server 6.0 for UNIX: Administration course use
eight virtual machines. The introductory page of each lab shows which virtual
machines will be used. The lab exercises contain icons like those shown in this
slide, to indicate which steps are performed on which virtual machines.
11
System Name
Description
Fully Qualified Host Name
mgt
mgt.example.com
winclient
Veritas Cluster Manager Java Console
winclient.example.com
vom
Veritas Operations
Manager CMS
vom.example.com
cps
Coordination Point
Server
cps.example.com
NFS Server
DNS Server
iSCSI Array
NTP Server
Note: In the lab exercises, virtual machines are identified by the system names
found in this table. The first Fully Qualified Host Name and Hostname
system names are bound to the first IP address listed in both the DNS and
the local hosts file.
B5
Virtual machine configuration (continued)

Description
Fully Qualified Host Name
sym1
SFHA 5.1 SP1 (VCS 5.1

SP1) cluster
sym1.example.com
sym2

SP1) cluster
sym2.example.com
sym3

SP1) cluster
sym3.example.com
sym4

SP1) cluster
sym4.example.com
Note: In the lab exercises, virtual machines are identified by the system names
found in this table. The first Fully Qualified Host Name and Hostname
system names are bound to the first IP address listed in both the DNS and
the local hosts file.
12
System Name
B6

Virtual machine account information

Refer to the following table for the log in credentials to the various virtual
machines:
System
Logon Information
mgt
User: student
Password: veritas
winclient
User: Administrator
Password: veritas
vom
User: root
Password: veritas
cps
User: root
Password: veritas
sym1
User: root
Password: veritas
sym2
User: root
Password: veritas
sym3
User: root
Password: veritas
sym4
User: root
Password: veritas
Note: Access to the mgt system by the root user is restricted. Notify your
instructor if you require root access.
Accessing virtual machines
13
There are two introductory labs that follow, Intro Lab A and Intro Lab B. You will
complete only one of them depending on how and where you attend this course.
Your instructor will direct you to the appropriate introductory lab.
If you are attending this course in a physical classroom, continue to Intro Lab
A: Using VMware Workstation in a physical classroom.
If you are attending this course in a virtual environment where you are
connected via the internet, skip to Intro Lab B: Using Hatsize.
B7
Lab 2: Intro Lab A: Using VMware Workstation in a classroom

In this lab, you become familiar with the VMware Workstation lab environment
used in a physical classroom environment. You will become familiar with the
virtual machines used with the Veritas Cluster Server 6.0 for UNIX:
Administration course, as well as the method for accessing these virtual machines.
The hands-on portion of this lab enables you to perform basic operations on virtual
machines. Adopting the best practice guidelines provided in this lab enables you to
perform the remaining labs more effectively.
Exercise 1: Starting virtual machines (VMware Workstation)
The virtual machines are started and the existing snapshots for each virtual
machine are examined.
Exercise 2: Logging on to virtual machines (VMware Workstation)

You log in to each virtual machine to become familiar with the log in procedures
and account credentials.
14
Exercise 3: Adjusting the VMware view (VMware Workstation)

VMware view controls are used to view the virtual machines in Quick Switch and
Full Screen modes and to resize the virtual machines display.
Exercise 4: Running basic commands (VMware Workstation)
Basic commands are used to determine virtual network connectivity.
B8

Note: These exercises are to be used only in a physical classroom using the
VMware Workstation environments. Exercises for the Hatsize environment
are in Intro Lab B.
Two parallel versions of the labs enable you to choose the level of detail that suits
your experience level:
Appendix A provides step-by-step lab instructions.
Appendix B provides complete lab instructions and solutions.
15

B9
VMware Workstation in a physical classroom environment

In the physical classroom environment, virtual machines are often referred to as
guest systems, which run their own guest operating systems. The physical
computer running the VMware Workstation application is referred to as the host
system or host computer. In many classrooms, the host computer will be named
PCtrainxx where xx is replaced with a number.
The eight virtual machines communicate using the several virtual networks
residing within the host.
16
B10

VMware Workstation interface
VMware Workstation software runs on the host system and provides an interface
to access the virtual machines. This interface has menu items and a tool bar that is
used to control the virtual machines or guest systems. Virtual machines are
accessed by clicking on the tab with the appropriate system name.
17

B11
Exercise 1: Starting virtual machines (VMware Workstation)

In this exercise, you review the settings on a few of the virtual machines and start
them.
1
If VMware is not already open, start VMware Workstation and ensure that the
following tabs are present:
mgt
winclient
vom
cps
sym1
sym2
sym3
sym4
CAUTION If any of the tabs are not present, notify your instructor.
Solution
On the desktop of your host system, double-click the Load Environment

icon.
Ensure that VMWare Workstation opens and that the following tabs are
present - mgt, winclient, vom, cps, sym1, sym2, sym3 and sym4. If these
tabs are not present, notify your instructor.
End of Solution
18
Review the current settings, including the memory and virtual network
adapters, for each virtual machine. Ensure the memory for each virtual
machine is set to the following:
B12
mgt: 1 GB
winclient: 1 GB
vom: 1 GB
cps: 1 GB
sym1: 2 GB
sym2: 2 GB
sym3: 2 GB
sym4: 2 GB
CAUTION If the memory settings are different that listed above, notify your
instructor.
Solution
From the VMware Workstation application, choose a virtual machine by

clicking its tab.
From the Summary view, locate the Devices pane and review the
information showing the virtual machine configuration.
Notice which virtual network (VMnetx) to which each network adapter is

connected.
Click each of the remaining tabs and review the Devices pane information
for each virtual machine.
End of Solution
19

B13
mgt
3
Start the mgt virtual machine.

CAUTION Wait for the mgt virtual machine to fully start and display a log
in prompt before proceeding to the next step. If these virtual
machines or any other virtual machines do not start and display a
log in prompt, notify your instructor.
Note: The mgt virtual machine is required for all lab exercises. Failure to
start it will result in missing files and missing shared LUNs.
Solution
From the VMware Workstation application, click the mgt tab.
From the toolbar, click the green Power On button.
End of Solution
vom
4
Note: For best performance, do not start another virtual machine until the
previous virtual machine is fully started and displaying a log in prompt.
20
Start the vom virtual machine.
Note: Generally, each lab requires no more than six virtual machines to be
started. For best performance, you may shut down and re-start virtual
machines to match those required by each lab. The virtual machines
needed for each lab is identified in Exercise 1 of each lab.
B14

Solution
From the VMware Workstation application, click the vom tab.
Wait until the login window is displayed.
End of Solution
sym1
5
Start the sym1 virtual machine.

Solution
From the VMware Workstation application, click the sym1 tab.
End of Solution
21
sym2
6
Start the sy2 virtual machine.

Solution

B15
End of Solution
winclient
7
Start the winclient virtual machine.

Solution
From the VMware Workstation application, click the winclient tab.
End of Solution
22
B16

Exercise 2: Logging on to virtual machines (VMware Workstation)

In this exercise, you log in to a few of the virtual machine to become familiar with
the log in procedures for each system type.
vom
1
Log into vom using:

Account: root
Password: veritas
Solution
From the VMware Workstation application, click the vom tab.
From the vom server desktop, click the mouse to establish keyboard
control.
In the Username field, type: root
Press the Enter key.
In the Password field, type: veritas
End of Solution
23

B17
sym1
2
Log into sym1 using:

Account: root
Password: veritas
Solution
From the sym1 computer desktop, click the mouse to establish keyboard
control.
In the Password field, type: Password: veritas
End of Solution
sym2
3
Account: root
Password: veritas
24
Log into sym2 using:
Solution
B18
control.

End of Solution
winclient
4
Log onto the winclient computer with the following credentials.

Account: Administrator
Password: veritas
Note: Due to the differences in host computer hardware, you may be
prompted to restart the winclient computer to update driver software.
At the prompt, click Restart Now and log on again when prompted.
Solution
From the VMware Workstation application, click the winclient tab.
From the winclient computer desktop, click the mouse to establish

keyboard control.
From the VMware Workstation menu, select

VM > Send CTRL+Alt+Del.
Note: Alternately, press Ctrl+Alt+Ins.
25
From the winclient computer desktop, in the Password field,

type: veritas
End of Solution

B19
Exercise 3: Adjusting the VMware view (VMware Workstation)
In this exercise, VMware Workstation controls are used to view the virtual
machines in Quick Switch and Full Screen modes and to resize the virtual
machines displays.
1
From the VMware Workstation menu, select View > Quick Switch. How is
the Quick Switch view different from the default view?
The workspace, including the virtual machine tabs, is enlarged to fit the host
computers display.
2
Press F11 to return to the default view.

Note: If the view does not change, press Ctrl+Alt to release control from the
virtual machine and press F11 again.
Note: When working through the lab exercises, use the F11 shortcut key to
toggle between the default and Quick Switch views, as desired.
From the VMware Workstation menu, select View > Full Screen. How is the
Full Screen view different from the default view?
The workspace is enlarged to fit the host computers display. A toolbar (which can be
hidden) is included that can be used to control the virtual machine and to switch
26
between virtual machines.

4
Press Ctrl+Alt+Enter to return to the default window.

Note: When working through the lab exercises, use the Ctrl+Alt+Enter
shortcut key to toggle between the default and Full Screen views, as
desired.
B20
From the VMware Workstation menu, select View > Autofit Guest.
Resize the VMware Workstation window, or toggle between Quick Switch

and Full Screen modes. How does Autofit Guest mode affect the display?
In Autofit Guest mode, the screen resolution of the guest system will automatically
change to match the size of the VMware Workstation window.
27

B21
Exercise 4: Running basic commands (VMware Workstation)

In this exercise, you use basic commands to determine whether the virtual
machines can communicate by way of TCP/IP on the virtual network.
Note: For simplicity, the instructions found in the solutions to enter a command
in a terminal window will only show the command. to be entered. These
commands will be appear in a bolded font that is different than the normal
text.
As an example, the instructions for step d will be shown as:
d ip addr show
rather than:
d In the terminal window, type: ip addr show
After typing each command, press Enter.
Sample output that is to be expected from each command will be shown
directly after the command in a smaller font.
vom
1
From vom, open a terminal window if none is already open.

Solution
If a terminal window is not already open, from the desktop, right-click and
select Konsole.
28
End of Solution
B22
Use the ip addr show command and record the IP addresses assigned to
this system.
vom IP address - eth0:
10.10.2.6
10.10.3.6

Solution
ip addr show
Note: This is a command to be entered in the terminal window. In the
terminal window, type: ip addr show
The output from this command follows.
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
1000
link/ether 00:0c:29:06:14:ff brd ff:ff:ff:ff:ff:ff
inet 10.10.2.6/24 brd 10.10.2.255 scope global eth0
inet6 fe80::20c:29ff:fe06:14ff/64 scope link
1000
link/ether 00:0c:29:06:14:09 brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe06:1409/64 scope link
1000
1000
link/ether 00:0c:29:06:14:1d brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe06:141d/64 scope link
6: sit0: <NOARP> mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0
29
Locate the entries for the eth0 and eth1 interfaces.
Record the IP addresses.
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
End of Solution
Use the ping -c <ip address> command to ping default gateways.

Solution
ping -c 3 10.10.2.1
PING 10.10.2.1 (10.10.2.1) 56(84) bytes of data.
64 bytes from 10.10.2.1: icmp_seq=1 ttl=128 time=1.62 ms

B23

--- 10.10.2.1 ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.100/0.621/1.627/0.711 ms
ping -c 3 10.10.3.1
End of Solution
Use the nslookup command to discover the fully qualified host name of
sym1.
Solution
nslookup sym1
Server:
Address:
10.10.2.3
10.10.2.3#53
Name:
sym1.example.com
Address: 10.10.2.11
Notice that the fully qualified host name is: sym1.example.com
End of Solution
sym1
30
From sym1, open a terminal window if none are already open.

Solution
select Konsole.
End of Solution
B24

this system.
sym1 IP address - eth0:
10.10.2.11
10.10.3.11
10.10.4.11
10.10.5.11
Solution
31
ip addr show
1000
link/ether 00:0c:29:5a:b1:ee brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe5a:b1ee/64 scope link
1000
link/ether 00:0c:29:5a:b1:f8 brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe5a:b1f8/64 scope link
1000
link/ether 00:0c:29:5a:b1:02 brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe5a:b102/64 scope link
1000
link/ether 00:0c:29:5a:b1:0c brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe5a:b10c/64 scope link
1000
1000
link/sit 0.0.0.0 brd 0.0.0.0
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
Locate the entries for the eth0, eth1, eth2 and eth3 interfaces.

B25
End of Solution
sym2.
Solution
nslookup sym2
Server:
Address:
10.10.2.3
10.10.2.3#53
Name:
sym2.example.com
Address: 10.10.2.12
End of Solution
Use the fdisk -l command to ensure that the iSCSI LUNs are available.
Note: The mgt virtual machine must be running to have access to the iSCSI
LUNs.
Solution
fdisk -l
Device Flag
/dev/sdd3 u
/dev/sdd8 u
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
Disk /dev/sdd3 (Sun disk label): 128 heads, 32 sectors, 1022 cylinders
Units = cylinders of 4096 * 512 bytes
32
Start
0
0
Device Flag
/dev/sdd3p3 u
/dev/sdd3p8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
Device Flag
Start
End
Blocks
Id System
/dev/sdd8p3 u
0
1022
2093056
5 Whole disk
/dev/sdd8p8 u
0
1022
2093056
f Unknown
...
Disk /dev/sdz (Sun disk label): 128 heads, 32 sectors, 1022 cylinders
B26

Device Flag
/dev/sdz3 u
/dev/sdz8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
Disk /dev/sdz3 (Sun disk label): 128 heads, 32 sectors, 1022 cylinders
Device Flag
/dev/sdz3p3 u
/dev/sdz3p8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
Device Flag
/dev/sdz8p3 u
/dev/sdz8p8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
fdisk -l | grep "^Disk" | wc -l

74
End of Solution
sym2
9

Solution
select Konsole.
End of Solution
33
10 Use the ip addr show command and record the IP addresses assigned to
this system.
10.10.2.12
10.10.3.12
10.10.4.12
10.10.5.12

B27
Solution
ip addr show

1000
link/ether 00:0c:29:2d:77:72 brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe2d:7772/64 scope link
1000
link/ether 00:0c:29:2d:77:7c brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe2d:777c/64 scope link
1000
1000
1000
link/ether 00:0c:29:2d:77:9a brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe2d:779a/64 scope link
1000
link/ether 00:0c:29:2d:77:a4 brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe2d:77a4/64 scope link
link/sit 0.0.0.0 brd 0.0.0.0
34
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
End of Solution
B28

11 Use the nslookup command to discover the fully qualified host name of
vom.
Solution
nslookup vom
Server:
Address:
10.10.2.3
10.10.2.3#53
Name:
vom.example.com
Address: 10.10.2.6
Notice that the fully qualified host name is: vom.example.com
End of Solution
12 Use the fdisk -l command to ensure that the iSCSI LUNs are available.
LUNs.
Solution
fdisk -l
Disk /dev/sde3 (Sun disk label): 128 heads, 32 sectors, 1022 cylinders
Device Flag
/dev/sde3p3 u
/dev/sde3p8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
35
Device Flag
/dev/sde8p3 u
/dev/sde8p8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
Disk /dev/sdf (Sun disk label): 128 heads, 32 sectors, 1022 cylinders
Device Flag
Start
End
Blocks
Id System
/dev/sdf3 u
0
1022
2093056
5 Whole disk
/dev/sdf8 u
0
1022
2093056
f Unknown
...
Device Flag
/dev/sdz3 u
/dev/sdz8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f

System
Whole disk
Unknown
B29
Device Flag
/dev/sdz3p3 u
/dev/sdz3p8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
Device Flag
/dev/sdz8p3 u
/dev/sdz8p8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown

74
End of Solution
36
B30

Exercise 5: Shutting down virtual machines (VMware Workstation)

In this exercise, you shut down the vom and winclient virtual machines.
vom
1
From vom, from the terminal window, use the shutdown -h now
command to shutdown and power off the virtual machines until it is needed in
a future lab.
Solution
shutdown -h now
End of Solution
winclient
2
From winclient, terminal window, use the Start menu to shutdown and
power off the virtual machines until it is needed in a future lab.
Solution
37
From the Start bar, select Start.
To the right of the Log off button, click the right-arrow menu and select
Shutdown.
End of Solution
End of lab
B31
Lab 2: Intro Lab B: Hatsize introduction

In this lab, you become familiar with the Hatsize lab environment accessed
through the internet. You will become familiar with the virtual machines used with
the Veritas Cluster Server 6.0 for UNIX: Administration course, as well as the
method for accessing these virtual machines.
The hands-on portion of this lab enables you to perform basic operations on virtual
machines. Adopting the best practice guidelines provided in this lab enables you to
perform the remaining labs more effectively.
This lab contains the following exercises:
Exercise 1: Connecting to the lab environment (Hatsize)
You log on to Hatsize, then connect to and log on to the primary system.
Exercise 2: Connecting to additional virtual machines (Hatsize)

You connect to additional virtual machines and become familiar with switching
between systems.
38
Exercise 3: Running basic commands (Hatsize)

Basic console commands are used to determine virtual network connectivity.
Note: These exercises are to be used only if the class is using the hosted Hatsize
platform to access the lab environment. Exercises for the VMware
Workstation environment are in Intro Lab A.
B32

Hatsize lab environment
The following table provides a translation of virtual machine system names

referred to in the lab guide to the corresponding system names in the Hatsize
interface. Each system name is prefixed with S# (Student number) in Hatsize.
39
System Name
Hatsize Name
mgt
S#.mgt
winclient
S#.winclient
vom
S#.vom
cps
S#.cps
sym1
S#sym1
sym2
S#.sym2
sym3
S#.sym3
sym4
S#.sym4

B33
Hatsize interface
The Hatsize interface is used to access the virtual machines. Instead of using tabs
like VMware Workstation, the Hatsize virtual machines are accessed from the
Operations and Windows menus. Other key interface elements include:
Title Bar: Indicates the currently connected machine and whether you have
control of the machine or are in view-only mode
Operations: Used to connect to additional virtual machines
Windows: Used to switch between active virtual machines
System Control: Used to start, stop and restart virtual machines
Time Remaining: Indicates the time remaining in the session
40
B34

Exercise 1: Connecting to the lab environment (Hatsize)

In this exercise, you log on to Hatsize and connect to the first system. For each lab
environment in Hatsize, a particular virtual machine is marked as a primary
machine. All other machines are marked as secondary machines. When you
connect to the Hatsize interface, you are initially connected to the primary virtual
machine.
1
Locate the Hatsize portal URL and login credentials from your registration
e-mail. Record your credentials here:
Hatsize username:
Hatsize password:
2
Your student number is the number at the end of your Hatsize username
recorded in the previous step. Record your student number here:
Note: When you use the Hatsize environment, all of the virtual machines assigned
to you are prefixed with a letter and your student number. For example, if
your student number is 8, the virtual machine named mgt is named k8-mgt,
s8-mgt or in a similar fashion. Since the prefix is different for each student,
the lab exercises refer only to the system name without the prefix.
41

B35
In Internet Explorer, open the Hatsize portal URL and log on with your
assigned user name and password recorded in a previous step.
After logging in, find your class in the Current Classes table and click Enter.
Note: The name of your class will be different than the sample shown here.
42
B36
In the Agenda for your class, find the Day 1 lab session and click OPEN.

Confirm that you see the primary machine connecting.

Note: The machine name will be different than the sample shown here.
Wait for the first console window to open.
43

B37
Exercise 2: Connecting to additional virtual machines (Hatsize)

In this exercise, you connect to additional virtual machines and become familiar
with switching between systems.
CAUTION
In the Hatsize environment, all eight virtual machines will be

booted and ready for login. Exercise 1 in every lab instructs you to
shutdown the virtual machines that are not needed for that lab. Skip
these exercises. Do NOT shutdown and start the virtual machines
unless directed to do so by your instructor.
Note: The menus you encounter in the hatsize interface will be slightly different
than the menu samples shown in this exercise.
44
From the Hatsize menu, select Operations > Connect to Secondary to

connect to the sym1 system.
sym1
2
Log into the sym1 server using the following credentials.

Account: root
B38

Password: veritas
Solution
control.
End of Solution
Use the Window > mgt menu item to switch to the mgt system.
Use the Window > sys1 menu item to switch back to the sys1 system.
Use the Operations > Disconnect menu item to disconnect the sys1 system.
45

B39
From the Disconnect Confirmation window, click Yes.
Use the Window menu item to verify that the console window for the sys1
system is closed.
Note: Notice that the sys1 entry is no longer available from the Window
menu item.
Note: In further labs, use the steps you practiced in this exercise to navigate
between different virtual lab systems in the Hatsize environment. The lab
solutions will no longer display the steps needed to navigate from one
virtual machine to another.
46
B40

Exercise 3: Running basic commands (Hatsize)

In this exercise, you use basic commands to determine whether the virtual
machines can communicate by way of TCP/IP on the virtual network.
Note: Use the steps in Exercise 2 to connect and navigate to the additional
machines as needed.
Note: For simplicity, the instructions found in the solutions to enter a command
in a terminal window will only show the command. to be entered. These
commands will be appear in a bolded font that is different than the normal
text.
As an example, the instructions for step d will be shown as:
d ip addr show
rather than:
d In the terminal window, type: ip addr show
After typing each command, press Enter.
Sample output that is to be expected from each command will be shown
directly after the command in a smaller font.
vom
47
From vom, open a terminal window if none is already open.

Solution
select Konsole.
End of Solution

B41
this system.
10.10.2.6
10.10.3.6
Solution
ip addr show
Note: This is a command to be entered in the terminal window. In the
terminal window, type: ip addr show
The output from this command follows.

1000
link/ether 00:0c:29:06:14:ff brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe06:14ff/64 scope link
1000
1000
1000
link/ether 00:0c:29:06:14:1d brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe06:141d/64 scope link
link/sit 0.0.0.0 brd 0.0.0.0
48
Locate the entries for the eth0 and eth1 interfaces.
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
End of Solution
B42

Use the ping -c <ip address> command to ping default gateways.

Solution
ping -c 3 10.10.2.1
ping -c 3 10.10.3.1
End of Solution
sym1.
Solution
nslookup sym1
Server:
Address:
10.10.2.3
10.10.2.3#53
Name:
sym1.example.com
Address: 10.10.2.11
49
End of Solution

B43
sym1
5

Solution
select Konsole.
End of Solution
this system.
10.10.2.11
10.10.3.11
10.10.4.11
10.10.5.11
Solution

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
1000
1000
1000
50
ip addr show
B44


1000
1000
1000
link/sit 0.0.0.0 brd 0.0.0.0
End of Solution
sym2.
Solution
nslookup sym2
Server:
Address:
10.10.2.3
10.10.2.3#53
Name:
sym2.example.com
Address: 10.10.2.12
51
End of Solution
Use the fdisk -l command to ensure that the iSCSI LUNs are available.
LUNs.

B45
Solution
fdisk -l
Device Flag
/dev/sdd3 u
/dev/sdd8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
Device Flag
/dev/sdd3p3 u
/dev/sdd3p8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
Device Flag
Start
End
Blocks
Id System
/dev/sdd8p3 u
0
1022
2093056
5 Whole disk
/dev/sdd8p8 u
0
1022
2093056
f Unknown
...
Device Flag
/dev/sdz3 u
/dev/sdz8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
Device Flag
/dev/sdz3p3 u
/dev/sdz3p8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
Device Flag
/dev/sdz8p3 u
/dev/sdz8p8 u
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown

74
52
Start
End of Solution
sym2
B46


Solution
select Konsole.
End of Solution
10 Use the ip addr show command and record the IP addresses assigned to
this system.
10.10.2.12
10.10.3.12
10.10.4.12
10.10.5.12
Solution
53
ip addr show
1000
1000
link/ether 00:0c:29:2d:77:7c brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe2d:777c/64 scope link
1000
1000
1000

pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
pfifo_fast qlen
B47
link/ether 00:0c:29:2d:77:9a brd ff:ff:ff:ff:ff:ff

inet6 fe80::20c:29ff:fe2d:779a/64 scope link
1000
link/ether 00:0c:29:2d:77:a4 brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe2d:77a4/64 scope link
link/sit 0.0.0.0 brd 0.0.0.0
End of Solution
11 Use the nslookup command to discover the fully qualified host name of
vom.
Solution
nslookup vom
Server:
Address:
10.10.2.3
10.10.2.3#53
Name:
vom.example.com
Address: 10.10.2.6
Notice that the fully qualified host name is: vom.example.com
End of Solution
12 Use the fdisk -l command to ensure that the iSCSI LUNs are available.
LUNs.
54
Solution
fdisk -l
Device Flag
/dev/sde3p3 u
/dev/sde3p8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
B48

Device Flag
/dev/sde8p3 u
/dev/sde8p8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
Disk /dev/sdf (Sun disk label): 128 heads, 32 sectors, 1022 cylinders
Device Flag
Start
End
Blocks
Id System
/dev/sdf3 u
0
1022
2093056
5 Whole disk
/dev/sdf8 u
0
1022
2093056
f Unknown
...
Device Flag
/dev/sdz3 u
/dev/sdz8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
Device Flag
/dev/sdz3p3 u
/dev/sdz3p8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown
Device Flag
/dev/sdz8p3 u
/dev/sdz8p8 u
Start
0
0
End
1022
1022
Blocks
2093056
2093056
Id
5
f
System
Whole disk
Unknown

74
End of Solution
End of lab
55

B49
56
B50

Lab 3: Validating site preparation

In this lab, you will perform a pre installation verification of the sym1 and sym2
systems in preparation for the installation and configuration of Storage Foundation
High Availability (SFHA) 6.0 in a later lab.
Exercise 1: Checking lab prerequisites
A verification that the virtual machines needed for this lab are powered on and
functioning is performed.
Exercise 2: Verifying the network configuration

The network configuration of the lab environment including host and virtual host
names, IP and virtual IP addresses, and the Network Time Protocol (NTP) server
are verified.
57
Exercise 3: Performing a CPI pre-installation verification check

A CPI based pre-installation verification check for SFHA 6.0 is performed.
Exercise 4: Performing a SORT pre-installation verification check
A Symantec Operational Readiness Tool (SORT) based pre-installation
verification check for SFHA 6.0 is performed.

B51

In this exercise, you verify that the virtual machines needed for this lab are
powered on and functioning, that you are logged in using the proper account and
that any needed terminal windows are opened.
1
Verify that the following virtual machines shown are powered on, that you are
logged using the indicated account and that the indicated terminal windows are
opened.
mgt
This virtual machine should be powered on, but there is no need to log
into it at this time.
sym1
Log in with credentials:
Account: root
Password: veritas
Two terminal windows should be active on this server.
sym2
Note: The terminal windows are referred to as hostname:terminal#
throughout the labs. For example: sym1:terminal1, sym1:terminal2,
sym2:terminal1, and sym2:terminal2. It is not necessary to label the
terminal windows and you may decide which is terminal1 and which is
terminal2.
58
If you have virtual machines running that are not used in this lab, shut down
the operating system and power the machines off.
Note: If you are completing the lab exercises in order, you will not need to
shutdown or power on any virtual machines.
B52

Exercise 2: Verifying the network configuration

In this exercise, you verify the network configuration of the lab environment
including host and virtual host names, IP and virtual IP addresses, and the Network
Time Protocol (NTP) server.
sym1
1
From sym1:terminal1, display the /etc/hosts file and review the

hostnames and their associated IP addresses.
Note: The hostnames of mgt, winclient, vom, cps, sym1, sym2, sym3,
and sym4 are systems whose IP address is configured on the eth0:
interface.
The hostnames of west, east, and cpsclus are virtual host names
and cluster names whose IP addresses are or will be configured on the
eth0: interface.
The hostnames of loopyvip, nfsvip, and symdb are virtual host
names for lab applications whose IP addresses will be configured on
the eth0: interface.
The hostnames of cpsvipa and cpsvipb are virtual names for use
in the Coordination Point (CP) Server lab.
Solution
59
cat /etc/hosts
127.0.0.1
localhost.localdomain
localhost
10.10.2.3
10.10.2.4
10.10.2.5
10.10.2.6
10.10.2.7
mgt.example.com
3par.example.com
winclient.example.com
vom.example.com
cps.example.com
mgt
3par
winclient
vom
cps
10.10.2.11
10.10.2.12
10.10.2.13
10.10.2.14
sym1.example.com
sym2.example.com
sym3.example.com
sym4.example.com
sym1
sym2
sym3
sym4
10.10.2.51
10.10.2.52
west.example.com
east.example.com
west
east

B53
10.10.2.61
10.10.2.71
10.10.2.90
10.10.2.91
10.10.3.91
loopyvip.example.com
nfsvip.example.com
cpsclus.example.com
cpsvipa.example.com
cpsvipb.example.com
loopyvip
nfsvip
cpsclus
cpsvipa
cpsvipb
10.10.2.101
symdb.example.com
symdb
End of Solution
Review the IP addresses assigned to the eth0:, eth1:, eth2:, and eth3:
interfaces using the ip addr show command.
Note: The eth4: and eth5: interfaces will be used in the VCS private network
lab and do not have an IP address configured at this time.
Solution

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...
link/sit 0.0.0.0 brd 0.0.0.0
60
ip addr show
End of Solution
B54

Use the nslookup command to verify the following IP addresses or host

names on the DNS server running on mgt are configured the same as in the /
etc/hosts file.
sym2
symdb
10.10.3.91
Note: The mgt server has been configured as a DNS and NTP server and
must be powered on and functioning.
Solution
nslookup sym2
Server:
Address:
10.10.2.3
10.10.2.3#53
Name:
sym2.example.com
Address: 10.10.2.12
nslookup symdb
Name:
symdb.example.com
Address: 10.10.2.101
nslookup 10.10.3.91
91.3.10.10.in-addr.arpa name = cpsvipb.example.com.
End of Solution
61
Use the ntpq -p command to verify the Network Time Protocol (NTP)
server for sym1 (a NTP client).
Solution
ntpq -p
remote
refid
st t when poll reach
delay
offset
======================================================================
*mgt.example.com LOCAL(0)
11 u
58
64
77
0.248 -85.517
End of Solution

B55
Exercise 3: Performing a CPI pre-installation verification check

In this lab, you perform a CPI based pre-installation verification check for
SFHA 6.0.
sym1
1
From sym1:terminal1, navigate to the

/student/software/sf/sf60 directory and verify its contents by
displaying a long listing.
Solution
cd /student/software/sf/sf60
ls -l
total 1172
-rwxr-xr-x 1 root root 657308 Nov 29 00:01 3rdpartyattributions.pdf
drwxrwxr-x 4 root root
4096 Nov 29 02:54 cluster_server
-rw-r--r-- 1 root root
860 Nov 29 02:54 copyright
4096 Nov 29 02:54 dynamic_multipathing
4096 Nov 29 02:54 file_system
-rwxr-xr-x 1 root root 415782 Nov 29 00:01 getting_started.pdf
-rwxr-xr-x 1 root root
5292 Nov 28 22:38 installer
4096 Jun 7 2011 perl
-rw-r--r-- 1 root root 23957 Nov 29 00:01 readme_first.txt
4096 Nov 29 02:54 rpms
4096 Nov 29 02:54 scripts
4096 Nov 29 02:54 storage_foundation
4096 Nov 29 02:54
storage_foundation_cluster_file_system_ha
4096 Nov 29 02:54 storage_foundation_for_oracle_rac
4096 Nov 29 02:54
storage_foundation_high_availability
4096 Nov 29 02:54 virtualstore
4096 Nov 29 02:54 volume_manager
-rwxr-xr-x 1 root root 18014 Nov 28 22:38 webinstaller
4096 Nov 29 02:54 windows
4096 Nov 28 22:38 xprtl
62
End of Solution
B56
Start the installer script and perform a pre-installation verification check for
SFHA 6.0 using the following information and record the directory within the
/opt/VRTS/install/logs directory that contains the installer log files,
summary file and response file.

Task: Perform a Pre-Installation Check

Product: Veritas Storage Foundation and High Availability (SFHA)
RHEL system names: sym1 sym2
Do not install SFHA
View the summary file
Directory within the /opt/VRTS/install/logs directory that contains the

log files, summary files and response file:
The directory will be in the form installer-yyyymmddhhmmxxx
Solution
./installer
To Perform a Pre-Installation Check, type: P
To perform the Veritas Storage Foundation and High Availability

(SFHA) check, type: 4
To specify the RHEL system names to be checked, type: sym1 sym2

Note: Separate the names using a space.
63
Observe the following system verification checks:
System communication
Release compatibility
Installed product
Prerequisite patches and rpms
Platform version
File system free space
Product licensing
Product prechecks
Verify the pre-check was successful and the following messages were
received:
Precheck report completed
System verification checks completed successfully
No issues found in prechecks

B57
At the prompt to install SFHA on sym1 and sym2, type: n
Notice that the installer log files, summary file, and response file are
saved at:
/opt/VRTS/install/logs/installer-yyyymmddhhmmxxx
To view the summary file, type: y

installer Summary
System Verification checks completed successfully
Note: The CPI ends at this point and control is returned to the login shell.
End of Solution
Navigate to the directory containing the log file recorded above, list the
directory contents, and examine the *.log0 log file and any other files you
wish to view.
Solution
cd /opt/VRTS/install/logs/
installer-yyyymmddhhmmxxx
Note: Use the directory you recorded in the previous step in place of
installer-yyyymmddhhmmxxx.
installer-201201111027yIr.log0
64
ls
installer-201201111027yIr.response
installer-201201111027yIr.summary
installer-201201111027yIr.tunables
more installer-yyyymmddhhmmxxx.log0
Examine any other files using the more or cat command.
End of Solution
B58

Exercise 4: Performing a SORT pre-installation verification check

In this exercise, you perform a Symantec Operational Readiness Tool (SORT)
based pre-installation verification check for SFHA 6.0.
sym1
1

/student/software/sf/sort60 directory and verify its contents by
displaying a long listing.
Solution
cd /student/software/sf/sort
ls -l
total 18664
-rwxrw-rw- 1 root root 19085939 Jan
9 11:31 sort_linux_x64.sh
End of Solution
Copy the sort_linux_x64.sh shar file to the /var/tmp directory. Then,

navigate to the /var/tmp directory and confirm it was copied.
Solution
65
cp sort_linux_x64.sh /var/tmp/.
cd /var/tmp
ls -l
total 18672
drwx------ 2 root
root
4096 May
drwx------ 2 student student
4096 Dec
-rwxr--r-- 1 root
root
19085939 Jan
6 2011 kdecache-root
7 15:21 kdecache-student
9 11:11 sort_linux_x64.sh
End of Solution

B59
Execute the sort_linux_x64.sh shar file to extract its contents and begin
the SORT.
Solution
./sort_linux_x64.sh
Uncompressing
Uncompressing
Uncompressing
Uncompressing
...
Uncompressing
SORT
SORT
SORT
SORT
data
data
data
data
collector:
collector:
collector:
collector:
0% |*
1% |*
2% |*
|
|
|
SORT data collector: 100% |********************************|
End of Solution
Run the data collector and accept the terms and conditions.
Note: Access to the sort.symantec.com web site to check for the latest
version will fail in these labs.
Note: In the future, you can start the data collector by running:
cd sort
./sortdc
Solution
Would you like to run the data collector now? [y,n] (y)
===========================================================================
| Checking available driver (sym1) resources
===========================================================================
Working directory file system type: ext3
Open files limit: 1024
Virtual memory limit: unlimited (KB)
Free swap: 4094 (MB)
66
Loading Data Collector...

Symantec Operations Readiness Tools
Copyright (c) 2011 Symantec Corporation. All rights reserved. Symantec and
the Symantec logo are trademarks or registered trademarks of Symantec
Corporation or its affiliates in the U.S. and other countries. Other names
may be trademarks of their respective owners.
The Licensed Software and Documentation are deemed to be "commercial
computer software" and "commercial computer software documentation" as
defined in FAR Sections 12.212 and DFARS Section 227.7202.
The data collector writes logs to /var/tmp/sortdc-201201091116OfJ.
B60

Starting the data collector [Build: 3.5.20111219]...

Press [Return] to indicate your acceptance of the terms and
conditions as indicated in the /var/tmp/sort/advanced/terms.txt
file, or q to decline: (y)
y
Connecting to Symantec to check for the latest data collector version
... Failed
Skipped getting the latest version of the data collector. To check for the
latest version manually, go to: https://sort.symantec.com
End of Solution
Continue with the SORT using the following information.
Task: Installation/Upgrade report

Product family: Storage Foundation and HA Solutions
Systems to report: one or more remote systems
System names: sym1 and sym2
User name to access the remote system(s): root
Note: Atypically, passwordless ssh for the root user has been configured on
the lab systems.
Solution
Press [Return] to continue:
Press Enter.
67
Main Menu:
What task do you want to accomplish?
1) Installation/Upgrade report (NetBackup, Storage Foundation and HA
Solutions)
2) Risk Assessment report (Storage Foundation and HA Solutions)
3) License/Deployment report (Storage Foundation and HA Solutions)
4) VxExplorer report
5) NetBackup Support Utility (NBSU) report
6) Other tasks
Choose your option (separate multiple selections with commas): [1-6,q] (1)
1
Main Menu->Installation/Upgrade report:

B61
Choose the Symantec enterprise product family:

1) NetBackup
2) Storage Foundation and HA Solutions
b) Back to previous menu
Choose your option: [1-2,b,q] (1)
2
Main Menu->Installation/Upgrade report->SFProductFamily:
On which systems do you want to run and report?
1) This system only(<sym1>)
2) One or more remote systems
Choose your option: [1-2,b,q] (1)
2
Enter one or more system names separated by space, or the full-qualified
path to a file containing a list of systems names: [b,q,?]
sym1 sym2
Systems Discovery and Authentication
Enter a user name to access the remote system(s): [q,?] (root)
root
Discovering transport (ssh,rsh) on sym1 ................ localhost
Discovering transport (ssh,rsh) on sym2 ......... ssh passwordless
Collected basic system information from sym2 ................ Done
Authenticating system(s)
Number of systems requiring authentication ..................... 0
Summary:
2 of 2 hosts are accessible via ssh, rsh.

Logs are being written to /var/tmp/sortdc-201201091116OfJ while sortdc is
inprogress
68
Partial clusters check: 0%

Estimated time remaining:
Checking for partial clusters among input systems
0 of 2

Estimated time remaining: 00:00:01
Checking for partial clusters among input
1 of 2

2 of 2

2 of 2
No partial cluster found
B62


inprogress
Collecting basic information: 0%
1. Basic data collection in progress
0 of 2

1 of 2

2 of 2

2 of 2
1. Basic data collection in progress .............................. Done
Collecting detailed information: 0%
0 of 2
2. Detailed data collection in progress
1 of 2
2 of 2
2 of 2
2. Detailed data collection in progress ........................... Done
Summary:
Data collection completed successfully for all the systems.
Total collection time:
0 hrs
1 mins 17 secs
End of Solution
69
Continue the SORT using the following information and notice the txt and xml
report files that are created. Then exit the data collector and record the
directory within the /var/tmp/sort/log directory that contains the
SORT log files.
Product to install or upgrade to: Veritas Storage Foundation HA
Product version: 6.0

B63
Note: The utility cannot access the sort.symantec.com web site for xml
report file upload in the this lab.
Directory within the /var/tmp/sort/log directory that contains the log files
The directory will be in the form sortdc-yyyymmddhhmmxxx
Solution
Choose the Symantec enterprise product you want to install or upgrade to. If
you are installing or upgrading multiple products, run the data collector
for each one.
1)
2)
3)
4)
5)
6)
7)
8)
9)
10)
11)
b)
Veritas Storage Foundation

Veritas Storage Foundation for Oracle
Veritas Cluster Server
Veritas Storage Foundation HA
Veritas Storage Foundation Cluster File System
Veritas Storage Foundation for Cluster File System/HA
Veritas Storage Foundation for Oracle RAC
Veritas Storage Foundation for Sybase
Veritas Storage Foundation for DB2
Veritas Storage Foundation Cluster File System for Oracle RAC
Veritas Storage Foundation Sybase ASE CE
Back to previous menu
Choose the product: [1-11,b,q] (1)
4
Choose the product version you want to install or upgrade to on the
system(s) in your environment.
Veritas Storage Foundation HA
1)
2)
3)
4)
5)
6)
7)
8)
9)
10)
b)
70
6.0 (AIX, HP-UX, Linux_x86_64, SunOS_sparc, SunOS_x86_64)

5.1SP1 (AIX, HP-UX, Linux_x86_64, SunOS_sparc, SunOS_x86_64)
5.1 (AIX, Linux_x86_64, SunOS_sparc, SunOS_x86_64)
5.0MP4 (Linux_ppc64, Linux_x86_64)
5.0RU4 (Linux_ppc64, Linux_x86_64)
5.0RU1 (Linux_x86_64)
5.0MP3 (AIX, Linux_x86_64, SunOS_sparc, SunOS_x86_64)
5.0MP2 (HP-UX, Linux_x86_64)
5.0MP1 (AIX, HP-UX, Linux_x86_64, SunOS_sparc)
5.0 (AIX, HP-UX, Linux_x86_64, SunOS_sparc, SunOS_x86_64)
Back to previous menu
Choose the product version: [1-10,b,q] (1)
1
Checking that the product and version match...
inprogress
B64

Analyzing systems: 0%
Detecting the server tier
71
0 of 5

1
Detecting the server tier ........................................
1
Detecting the processor tier
2
Detecting the processor tier .....................................
2
Detecting installed Storage Foundation products
3
Detecting installed Storage Foundation products ...................
3
Running a pre-installation assessment Analyzing systems: 80%
4
Running a pre-installation assessment ............................
4
Detecting installed Storage Foundation patches
5
Detecting installed Storage Foundation patches ....................
of 5
Done
of 5
Done
of 5
Done
Done
of 5
Done
Done
of 5
Done
Done
Done
of 5
Done
Done
Done
of 5
Done
Done
Done
Done
of 5
Done
Done
Done
Done
of 5
Done
Done
Done
Done
Done
Generated XML and text files based on the systems and the time you ran the
data collector.
Created /var/tmp/sort/reports/sym1andothers_IAS_20120109_111942.xml
Created /var/tmp/sort/reports/sym1andothers_IAS_20120109_111942.txt
The system cannot access the SORT Web site now, you can manually upload the
XML file to the SORT Web site (https://sort.symantec.com/) to view your
custom server report that contains documentation and links related to your
environment. The text file does not contain this additional information.
If the system can connect to the Internet via a proxy server, set the
https_proxy environment variable.
In sh syntax, set it as follows: export https_proxy=https://
proxy.domain.com:port

B65
In csh or tcsh syntax, set it as follows:

setenv https_proxy=https://proxy.domain.com:port
where proxy is the proxy server name, domain.com is the domain name, and
port is the port number for the proxy server.
Your tasks are completed. Would you like to exit the data collector?
[y,n,q](y)
y
SORT data collector current session log files are saved at:
/var/tmp/sort/log/sortdc-201201091116OfJ
End of Solution
List the directory contents of the log file location. Then, navigate to the reports
directory and display the contents of the txt file report.
Solution
ls -l /var/tmp/sort/log/sortdc-yyyymmddhhmmxxx
sortdc-yyyymmddhhmmxxx.
total 972
-rw-r--r--rw-r--r--rwx------rwx------rwx------rwx------rwx------rw-r--r--rw-r--r--
72
1
1
1
1
1
1
1
1
1
root
root
root
root
root
root
root
root
root
root
51 Jan
root
215 Jan
root
753 Jan
root 40752 Jan
root 11353 Jan
root
5653 Jan
root 19257 Jan
root 444469 Jan
root 443937 Jan
11:20
11:20
11:20
11:20
11:20
11:20
11:20
11:20
11:20
authentication.log
discovery.log
exitfile-201201091116OfJ
sortdc-201201091116OfJ.log
sortdc-201201091116OfJ.log0
sym1.log
sym2.log
ls -l
total 18680
drwx------ 2
drwx------ 2
drwxr-xr-x 6
-rwxr--r-- 1
drwxr-xr-x 2
cd sort
ls -l
root
student
root
root
root
root
4096 May
student
4096 Dec
root
4096 Jan
root
19085939 Jan
root
4096 Jan
total 140
drwxr-xr-x 6 root root
drwxr-xr-x 3 root root
B66
9
9
9
9
9
9
9
9
9
4096 Jan
4096 Jan
6
7
9
9
9
2011
15:21
11:16
11:11
11:12
kdecache-root
kdecache-student
sort
sort_linux_x64.sh
vos
9 11:16 advanced
9 11:19 db

-rw-r--r--rw-r--r-drwxr-xr-x
-rwxr-xr-x
-rw-r--r-drwxr-xr-x
-rwxr-xr-x
-rw-r--r--
1
1
3
1
1
2
1
1
root
root
root
root
root
root
root
root
root
599 Dec 19 18:44
root
492 Dec 19 18:44
root 4096 Jan 9 11:20
root 19767 Dec 19 18:44
root 20707 Dec 19 18:44
root 4096 Jan 9 11:19
root 6535 Dec 19 18:44
root 60812 Dec 19 18:44
group_input_sample.txt
host_input_sample.txt
log
offlinedc.sh
README
reports
sortdc
sortdc.conf
cd reports
ls -l
total 32
-rw-r--r-- 1 root root 5490 Jan 9 11:19
sym1andothers_IAS_20120109_111942.txt
-rw-r--r-- 1 root root 23104 Jan 9 11:19
sym1andothers_IAS_20120109_111942.xml
more *.txt
SORT Installation and Upgrade report
2012/01/09-11:19:42 PST
Server name: sym1
Operating system: Linux
Operating system version: 2.6.18-194.el5
Architecture: x86_64
Server model: VMware, Inc. VMware Virtual Platform
Distribution: Red Hat Enterprise Linux Server release 5.5 (Tikanga)
The following table lists Symantec patches recommended for installation or

upgrade, installed patches and patches with undetermined installation
status.
Patches that are not installed using standard operating system methods have
undetermined status. The patch list may not be up to date. For the latest
patch list, go to https://sort.symantec.com/patch/finder. Note: as Linux
has no package install history, a "Yes" under "Installed?" column on Linux
reports may indicate that newer versions of RPMs are installed.
73
Recommended Symantec patches for SFHA 6.0

-----------------------------------------------------------------------------Release ID Release name
Status
Summary: 94% checks passed, 0% failed, 6% skipped.
End of Solution
End of lab

B67
74
B68

Lab 4: Installing Storage Foundation HA 6.0

In this lab you will install and configure Storage Foundation High Availability
(SFHA) 6.0 on sym1 and sym2 using the Common Product Installer (CPI) and
then upgrade from SFHA 5.1 SP1 to SFHA 6.0 on sym3 and sym4 using the CPI.
The SFHA 6.0 configuration on the sym3 and sym4 systems will be examined.
The west cluster, made up of sym1 and sym2, will be placed under Veritas
Operations Manager (VOM) control. The east cluster, made up of sym3 and
sym4, remains under VOM controls after the upgrade.
Exercise 1: Checking lab prerequisites (part a)

A verification that the virtual machines needed for Exercises 2 through 4 are
powered on and functioning is performed.
75
Exercise 2: Installing SFHA using the CPI

An installation and configuration of SFHA 6.0 is performed.
Exercise 3: Running a post installation check
A post install check on the installation and configuration of SFHA 6.0 is
performed.
Exercise 4: Adding a cluster to VOM
The sym1 and sym2 systems are added to the Veritas Operations Manager.
Exercise 5: Checking lab prerequisites (part b)
A verification that the virtual machines needed for Exercises 6 and 7 are powered
on and functioning is performed.

B69
Exercise 6: Upgrading SFHA using the CPI

A full upgrade of an SFHA 51 SP1 two system cluster to SFHA 6.0 is performed.
Exercise 7: Examining the VCS configuration

The VCS configuration for the newly upgraded east cluster composed of the sym3
and sym4 systems is briefly examined.
76
B70

Exercise 1: Checking lab prerequisites (part a)

In this exercise, you verify that the virtual machines needed for Exercises 2
through 4 are powered on and functioning, that you are logged in using the proper
account and that any needed terminal windows are opened.
1
opened.
mgt
vom
Account: root
Password: veritas
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas
77

terminal2.

B71
Note: If you are completing the lab exercises in order, you will need to power on
and log into vom.
78
B72

Exercise 2: Installing SFHA using the CPI

In this exercise, you install and configure of SFHA 6.0 on sym1 and sym2.
sym1
1
From sym1:terminal1, navigate to and display a long listing of the

/student/software/sf/sf60 directory.
Solution
79
ls -l
total 1172
-rwxr-xr-x 1 root root 657308 Nov 29 00:01 3rdpartyattributions.pdf
4096 Nov 29 02:54 cluster_server
-rw-r--r-- 1 root root
860 Nov 29 02:54 copyright
4096 Nov 29 02:54 dynamic_multipathing
4096 Nov 29 02:54 file_system
-rwxr-xr-x 1 root root 415782 Nov 29 00:01 getting_started.pdf
-rwxr-xr-x 1 root root
5292 Nov 28 22:38 installer
4096 Jun 7 2011 perl
-rw-r--r-- 1 root root 23957 Nov 29 00:01 readme_first.txt
4096 Nov 29 02:54 rpms
4096 Nov 29 02:54 scripts
4096 Nov 29 02:54 storage_foundation
4096 Nov 29 02:54
storage_foundation_cluster_file_system_ha
4096 Nov 29 02:54 storage_foundation_for_oracle_rac
4096 Nov 29 02:54
storage_foundation_high_availability
4096 Nov 29 02:54 virtualstore
4096 Nov 29 02:54 volume_manager
-rwxr-xr-x 1 root root 18014 Nov 28 22:38 webinstaller
4096 Nov 29 02:54 windows
4096 Nov 28 22:38 xprtl
End of Solution
Begin the installation and configuration of SFHA 6.0 on sym1 and sym2
using the following information:

Install all rpms
RHEL5 system names: sym1 and sym2
Enable keyless licensing and complete system licensing later

B73
Product mode to license: SF Enterprise HA

Disable replication
Disable the Global Cluster Option
Solution
./installer
To Install a Product, type: I
To install Veritas Storage Foundation and High Availability (SFHA),

type: 4
To agree with the terms of the EULA, type: y
To Install all rpms, type: 3
To specify the RHEL5 system names, type: sym1 sym2
Observe the list of rpms that will be installed.

Note: Depending on the size of the terminal window, you may need to
press Enter one or more times to observe the complete listing of
rpms.
80
Installed product
Platform version
Product licensing
Product prechecks
Observe the following SFHA rpms being installed.
B74
Performing SFHA preinstall tasks

Installing VRTSvlic rpm
Installing VRTSperl rpm
Installing VRTSspt rpm
Installing VRTSvxvm rpm
Installing VRTSaslapm rpm

Installing VRTSob rpm
Installing VRTSlvmconv rpm
Installing VRTSvxfs rpm
Installing VRTSfsadv rpm
Installing VRTSfssdk rpm
Installing VRTSllt rpm
Installing VRTSgab rpm
Installing VRTSvxfen rpm
Installing VRTSamf rpm
Installing VRTSvcs rpm
Installing VRTScps rpm
Installing VRTSvcsag rpm
Installing VRTSvcsdr rpm
Installing VRTSvcsea rpm
Installing VRTSdbed rpm
Installing VRTSodm rpm
Installing VRTSsfcpi60 rpm
Installing VRTSvbs rpm
Performing SFHA postinstall tasks
Note: The rpm installations may take several minutes to complete. The
VRTSsfmh rpm is installed, but not reported out as such by the
installer.
81
To Enable keyless licensing and complete system licensing later, type: 2
To license SF Enterprise HA, type: 2
To disable replication, type: n
m To disable the Global Cluster Option, type: n

End of Solution

B75
Continue the installation and configuration of SFHA 6.0 by configuring SFHA

on sym1 and sym2 using the following information:
Do NOT configure I/O Fencing in enabled mode
Unique cluster name: west
Configure heartbeat links using LLT over Ethernet
First private heartbeat link: eth4
Second private heartbeat link: eth5
No additional low-priority heartbeat link
Use the same NICs for private heartbeat on all systems
Accept the unique cluster ID
Skip the check to determine if the cluster ID is in use by another cluster
Solution
82
To configure SFHA on sym1 sym2, type: y
To NOT configure I/O Fencing in enabled mode, type: n
To continue, press Enter.
To specify the unique cluster name, type: west
To Configure heartbeat links using LLT over Ethernet, type: 1
To specify the NIC for first private heartbeat link on sym1, type: eth4
To configure a second private heartbeat link, type: y
To specify the NIC for the first private heartbeat link on sym1, type:
eth5
To NOT configure a third private heartbeat link, type: n
To NOT configure an additional low-priority heartbeat link, type: n
To use the same NICs for private heartbeat links on all systems, type: y
To accept the default unique cluster ID number, press Enter.
m To NOT check if the cluster ID is in use by another cluster, type: n

n
To confirm that the information is correct, type: y
End of Solution
B76

Continue the installation and configuration of SFHA 6.0 by configuring the

virtual IP of the cluster, secure mode, users, SMTP notifications and SNMP
notifications and using the following information:
NIC for Virtual IP of the Cluster to use on sym1: eth0

Virtual IP address: 10.10.2.51
Net mask for 10.10.2.51: 255.255.255.0
Do not configure the VCS cluster in secure mode.
Accept the default cluster credentials for the admin account.
Do not add another user.
Do not configure SMTP or SNMP notifications.
Note: For each system, the CPI will attempt to connect to the SORT web site
to determine if there are product updates. Since the lab environment
does not provide this connectivity, the connection to SORT will fail.
Solution
83
To configure the Virtual IP, type: y
To accept the default eth0 as the NIC for Virtual IP of the Cluster to use
on sym1, press Enter.
To specify eth0 to be the public NIC used by all systems, type: y
To specify the Virtual IP address for the Cluster, type: 10.10.2.51
To accept the default 255.255.255.0 as the NetMask for IP 10.10.2.51,

press Enter.
To confirm that the Cluster Virtual IP information is correct, type: y
To NOT configure the VCS cluster in secure mode, type: n
To accept the default cluster credentials of admin/password, type: y
To NOT add another user to the cluster, type: n
To confirm that the VCS User information is correct, type: y
To NOT configure SMTP notification, type: n
To NOT configure SNMP notification, type: n
m Observe the following SFHA elements being started.

B77
Performing SFHA configuration

Starting vxdmp
Starting vxio
Starting vxspec
Starting vxconfigd
Starting vxesd
Starting vxrelocd
Starting vxchached
Starting vxconfigbackupd
Starting vxattached
Starting vxportal
Starting fdd
Starting llt
Starting gab
Starting amf
Starting had
Starting CmdServer
Starting vxdbd
Starting vxodm
Performing SFHA poststart tasks
End of Solution
Review the message about updates to the VRTSasplapm rpm.

Note: In many lab environments the ability to connect to the SORT Web site
is not enabled and a message may be generated indicating as such.
Solution
Veritas Storage Foundation and High Availability Startup completed

successful
84
The updates to VRTSaslapm package are released via the Symantec SORT web
page:
https://sort.symantec.com/asl. To make sure you have the latest version of
VRTSaslapm (for up to date ASLs and APMs), download and install the latest
package from the SORT web page.
Checking online updates for Veritas Storage Foundation and High
Availability 6.0
Attempted to connect to https://sort.symantec.com to check for
product updates, but connection failed.
Please visit https://sort.symantec.com to check for available
B78

product updates and information.
End of Solution
Complete the installation and configuration of SFHA 6.0 by recording the

directory within the /opt/VRTS/install/logs directory that contains
the installer log files and viewing the summary file. Then, navigate to the log
directory, list the directory contents, and examine the *.log0 log file and any
other files you wish to view.
log files, summary file and response file:
Solution
Notice that the installer log files, summary file, and response file are saved
at:
To scroll through the summary file, press Enter multiple times.

installer-yyyymmddhhmmxxx.
85
ls
installer-201111010815Wks.log0
...
install.VRTSvcs.sym2
install.VRTSvlic.sym1
install.VRTSvlic.sym2
install.VRTSvxfen.sym1
install.VRTSvxfen.sym2
grep had installer-yyyymmddhhmmxxx.log0

0 14:19:43 had started successfully on sym1
0 14:19:55 had started successfully on sym2

B79
grep gab *log0

0 03:11:34 cmd /bin/ls /student/software/sf/sf60/rpms | /bin/grep VRTSgab[v0-9] 2>&1
VRTSgab-6.0.000.000-GA_RHEL5.x86_64.rpm
0 03:11:34 cmd /usr/bin/du -sk /student/software/sf/sf60/rpms/VRTSgab6.0.000.000-GA_RHEL5.x86_64.rpm | /bin/awk '{print $1}' 2>&1
0 03:12:51 VRTSgab is not installed on sym1
0 03:12:51 Package VRTSgab will be installed on sym1
0 03:12:51 VRTSgab is not installed on sym2
0 03:12:51 Package VRTSgab will be installed on sym2
0 03:12:51 VRTSgab
Veritas Group Membership and Atomic Broadcast
0 03:15:00 VRTSgab rpm installed successfully on sym1
0 03:15:00 VRTSgab rpm installed successfully on sym2
0 03:37:39 gab started successfully on sym1
0 03:37:39 gab started successfully on sym2
Examine any other files using the more, cat or grep commands.
End of Solution
Use the lltstat -c command on both systems to display Low Latency

Transport (LLT) status and notice the differences in node and name. Then, use
the lltstat -active command only on sym1.
Solution
LLT configuration information:

node: 0
name: sym1
cluster: 43482
Supported Protocol Version(s)
nodes: 0 - 63
max nodes: 64
max ports: 32
links: 2
mtu: 1452
max sdu: 66560
max iovecs: 100
broadcast HB: 0
use LLT ARP: 1
checksum level: 10
response hbs: 4
trace buffer size: 10 KB
trace level: 1
strict source check: 0
max threads: 4
rexmit factor: 10
link fail detect level: 0
86
lltstat -c
: 5.0
ssh sym2 lltstat -c

LLT configuration information:
node: 1
name: sym2
cluster: 43482
B80

Supported Protocol Version(s)

nodes: 0 - 63
max nodes: 64
max ports: 32
links: 2
mtu: 1452
max sdu: 66560
max iovecs: 100
broadcast HB: 0
use LLT ARP: 1
checksum level: 10
response hbs: 4
trace buffer size: 10 KB
trace level: 1
strict source check: 0
max threads: 4
rexmit factor: 10
link fail detect level: 0
: 5.0
lltstat -nvv active

LLT node information:
Node
* 0 sym1
1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
eth4
eth5
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
OPEN
End of Solution
Use the gabconfig -a command to display Group Membership and

Atomic Broadcast (GAB) status.
Solution
gabconfig -a
GAB Port Memberships

===============================================================
Port a gen
b02a01 membership 01
Port h gen
b02a04 membership 01
87
End of Solution
Use the hastatus -sum command to display cluster summary status.

Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2

B81
-- GROUP STATE
-- Group
B
B
ClusterService
ClusterService
System
sym1
sym2
Probed
AutoDisabled
Y
Y
N
N
State
ONLINE
OFFLINE
Note: The ClusterService service group is configured to be able to run on

both systems, but is ONLINE on only one system.
End of Solution
88
B82

Exercise 3: Running a post installation check

In this exercise you perform a post install check on the installation and
configuration of SFHA 6.0 on sym1 and sym2 that was completed in the
preceding exercise.
sym1
From sym1:terminal1, navigate to the /opt/VRTS/install directory and
perform a post installation check on the installed and configured SFHA 6.0
cluster on sym1 and sym2 and review the summary file. To facilitate
reviewing the log file if errors occur, record the directory within the
/opt/VRTS/install/logs directory that contains the installer log files.
Note: The post installation check will NOT complete successfully due to
I/O Fencing being unconfigured. Symantec recommends configuring
I/O Fencing before using a cluster for production. I/O Fencing will be
configured in a later lab.
log files:
Solution
89
cd /opt/VRTS/install
./installsfha -postcheck sym1 sym2
Observe the following system verification checks.
Installed Product
Platform version
Product prechecks
Observe the following post installation checks.

B83
Checking processes and drivers status

Checking lltstat status
Checking llt configuration
Checking installed VRTSllt
Checking llt starting setting
Checking clusterid configuration
Checking gabconfig ports status
Checking installed VRTSgab
Checking GAB starting setting
Checking vxfen status
Checking installed VRTSvxfen
Checking had status
Checking installed VRTSvcs
Checking Volume Manager status
Notice that the post installation check does NOT complete successfully due
to I/O Fencing being unconfigured.
Notice that the installer log files, summary file, and response files are
saved at:
End of Solution
90
B84

Exercise 4: Adding a cluster to VOM

In this exercise, you add sym1 and sym2 to the Veritas Operations Manager.
vom
1
From vom, open the Firefox browser, navigate to the URL,

https://vom.example.com:14161 and log into the Veritas Operations
Manager using the following account.
Username: root
Password: veritas
Domain: vom.example.com(unixpwd)
Note: Use the FQDN name vom.example.com in the URL to avoid a security
certificate warning. There may be cached URLs in the browser history.
Alternately, select the one that says Login.
Solution
91
From the lower left corner of desktop, click the Web Browser icon.
From the browser window, in the address field, type:

https://vom.example.com:14161
Click Login.
End of Solution
Note: The faults and risk shown on the Home tab in the Faults and Risks
Overview section, and elsewhere within VOM, for the cps, sym3, and
sym4 servers can be ignored as those virtual machines are powered off.

B85
Add sym1 and sym2 to the Veritas Operations Manager using the following
information.
For sym1:
Host: sym1.example.com
User Name: root
Password: veritas
For sym2:
Host: sym2.example.com
User Name: root
Password: veritas
Solution
92
From the VOM menu bar, select Settings > Host Management.
From the right side of the Host Management pane, from the Actions dropdown menu, select Add Host(s).
From the Add Hosts(s) window, from the Hosts to be added section, in
the Host field, type: sym1.example.com
In the User Name field, type: root
Click Add entries.
In the new Host field, type: sym2.example.com
In the User Name field, type: root
From the bottom right hand corner of the window, click Next.
Observe the add hosts process.
From the bottom right hand corner of the Add Hosts window, click OK.
Note: Both sym1.example.com and sym2.example.com should show up
in the Host Management list in a Healthy Host State.
End of Solution
B86

Log out of the Veritas Operations Manager and close the browser window.
Solution
From the top right corner of the Veritas Operations Manager console,
from the User (root) drop-down menu, select Logout.
Close the browser window.
End of Solution
93

B87
Exercise 5: Checking lab prerequisites (part b)

In this exercise, you verify that the virtual machines needed for Exercises 6 and 7
are powered on and functioning, that you are logged in using the proper account
and that any needed terminal windows are opened.
1
opened.
mgt
There is no need to log into this virtual machine at this time.
sym3
Account: root
Password: veritas
sym4
Account: root
Password: veritas

terminal2.
94
Note: If you are completing the lab exercises in order, this will require you to
shutdown vom, sym1 and sym2, and power on and log into sym3 and
sym4.
B88

Exercise 6: Upgrading SFHA using the CPI

In this exercise you will perform a full upgrade of an SFHA 51 SP1 two system
cluster to SFHA 6.0. You begin by displaying the cluster elements and performing
a Common Product Installer (CPI) version check and pre-installation check. You
then upgrade the cluster, perform the CPI version check again and run a
post-installation check.
sym3
Note: You may optionally perform steps 1 through 5 on both the sym3 and sym4
cluster nodes, though it is not strictly necessary to do so. These checks are
a limited subset of all the checks that might be performed prior to a cluster
upgrade.
From sym3:terminal1, use the lltstat command to display the Low

Latency Transport (LLT) active status.
Solution
lltstat -nvv active

Node
* 0 sym3
1 sym4
State
OPEN
Link
Status
Address
eth4
eth5
UP
UP
00:0C:29:97:FB:5D
00:0C:29:97:FB:67
eth4
eth5
UP
UP
00:0C:29:C4:1D:0A
00:0C:29:C4:1D:14
OPEN
End of Solution
95
Use the gabconfig command to display the Group Membership and Atomic
Broadcast (GAB) status.
Solution
gabconfig -a
===============================================================
Port a gen
857802 membership 01
Port b gen
Port h gen
End of Solution

B89
Use the hastatus command to display the cluster summary status.

Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym3
sym4
-- GROUP STATE
-- Group
System
Probed
AutoDisabled
State
B
B
B
B
B
B
sym3
sym4
sym3
sym4
sym3
sym4
Y
Y
Y
Y
Y
Y
N
N
N
N
N
N
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ClusterService
ClusterService
dbsg
dbsg
websg
websg
End of Solution
Navigate to the /student/software/sf/sf60 directory and perform a

version check of the SFHA cluster on sym3 and sym4 using the
SFHA 6.0 CPI.
Note: At the end of the version check, the CPI will attempt to connect to the
SORT web site to determine if there are product updates. Since the lab
environment does not provide this connectivity, the connection to
SORT will fail.
Solution
96
./installer -version
To specify the system names for the version check, type: sym3 sym4
Installed product(s) on sym3:
Veritas Storage Foundation and High Availability - 5.1SP1
Product:
Veritas Storage Foundation and High Availability - 5.1SP1
Packages:
Installed Required packages for Veritas Storage Foundation and ...
#PACKAGE
#VERSION
VRTSamf
5.1.100.000
VRTSaslapm
5.1.100.000
VRTSatClient 5.0.32.0
B90

VRTSatServer
VRTSgab
VRTSllt
VRTSperl
VRTSvcs
VRTSvcsag
VRTSvlic
VRTSvxfen
VRTSvxfs
VRTSvxvm
5.0.32.0
5.1.100.000
5.1.100.000
5.10.0.7
5.1.100.000
5.1.100.000
3.02.51.010
5.1.100.000
5.1.100.000
5.1.100.000
Installed Optional packages for Veritas Storage Foundation and ...

#PACKAGE
#VERSION
VRTScps
5.1.100.000
VRTSdbed
5.1.100.000
VRTSfssdk
5.1.100.000
VRTSlvmconv 5.1.100.000
VRTSob
3.4.289
VRTSodm
5.1.100.000
VRTSsfmh
4.1.100.0
VRTSspt
5.5.000.005
VRTSvcsdr
5.1.100.000
VRTSvcsea
5.1.100.000
Summary:
Packages:
13 of 13 required Veritas Storage Foundation and High Availability 5.1SP1
10 of 10 optional Veritas Storage Foundation and High Availability 5.1SP1
Attempted to connect to https://sort.symantec.com to check for product ...
Please visit https://sort.symantec.com to check for available product ...
End of Solution
Note: The following step may be skipped. It is documented here to show that
separate upgrade precheck(s) maybe run. These prechecks are also
performed as part of the actual upgrade as shown in the next exercise.
97
Perform a pre-installation verification check (precheck) on the SFHA 5.1 SP1

cluster on sym3 and sym4 and review the summary file. To facilitate
reviewing the log file if errors occur, record the directory within the /opt/
VRTS/install/logs directory containing the installer log files.
Note: Do not upgrade SFHA on sym3 and sym4 at this time.
log files:
Solution
./installer -precheck

B91
To check Veritas Storage Foundation and High Availability (SFHA),

type: 4
To accept the default sym3 and sym4 as the RHEL5 system names, press
Enter.
Installed product
Platform version
Product licensing
Product prechecks
Observe the important notes discovered on the systems.

Note: These notes do not apply to this cluster upgrade and can be ignored.
System verification checks completed successfully
The following notes were discovered on the systems:
CPI NOTE V-9-40-4790 Notice: This version of VxFS supports only disk layout
version 7 and above. Disk layout version 6 is deprecated and you can only
mount it locally after upgrade. Only the VxFS file systems in the /etc/fstab
file and mounted VxFS file systems were checked. If you have any VxFS file
systems that are using unsupported or deprecated disk layout version,
upgrade them to disk layout Version 7 or above and restart the VxFS
installation.
CPI NOTE V-9-40-3073 Use the vxupgrade(1M) command to upgrade mounted file
systems, or the vxfsconvert(1M) command to upgrade unmounted file systems.
Use the fstyp -v command to determine the VxFS file system version. See the
Veritas Storage Foundation Installation Guide for more information on
upgrading VxFS disk layout versions.
98
CPI NOTE V-9-30-1309 SFHA version 5.1.100.000 will be upgraded to SFHA

version
6.0.000.000 on sym3
CPI NOTE V-9-40-4790 Notice: This version of VxFS supports only disk layout
version 7 and above. Disk layout version 6 is deprecated and you can only
mount it locally after upgrade. Only the VxFS file systems in the /etc/fstab
file and mounted VxFS file systems were checked. If you have any VxFS file
systems that are using unsupported or deprecated disk layout version,
upgrade them to disk layout Version 7 or above and restart the VxFS
installation.
CPI NOTE V-9-40-3073 Use the vxupgrade(1M) command to upgrade mounted file
systems, or the vxfsconvert(1M) command to upgrade unmounted file systems.
Use the fstyp -v command to determine the VxFS file system version. See the
B92

Veritas Storage Foundation Installation Guide for more information on

upgrading VxFS disk layout versions.
CPI NOTE V-9-30-1309 SFHA version 5.1.100.000 will be upgraded to SFHA
version
6.0.000.000 on sym4
No issues found in prechecks
To NOT upgrade SFHA on sym3 and sym4, type: n
saved at:
Note: The CPI ends at this point and control is returned to the login shell. If a
separate precheck run were desired, then the log files could now be
checked. Log file examination is skipped in this stand alone precheck.
End of Solution
Navigate to the /student/software/sf/sf60 directory and perform an

upgrade of the SFHA 5.1 SP1 cluster on sym3 and sym4 to SFHA 6.0 using
the SFHA 6.0 CPI. Use the following information to perform the upgrade.
Record the directory within the /opt/VRTS/install/logs directory that
contains the installer log files.
99
Perform a full upgrade

RHEL system names: sym3 and sym4
Stop the SFHA processes
View the summary file

log files:
Solution
./installer
To Upgrade a Product, type: G

B93
To perform a Full Upgrade: type: 1
To accept the default RHEL5 system names of sym3 and sym4, press
Enter.
Installed product
Platform version
Product licensing
Product prechecks
Note: The system verification checks should complete successfully.

Several notes will be displayed but will require no action on this
cluster.
100 B94
To accept the End User License Agreement, type: y
Observe the list of rpms that will be uninstalled and press Enter.
Observe the list of rpms that will be installed or upgraded and press Enter.
To stop SFHA processes now, type: y
Observe the following processes being stopped.
Performing SFHA prestop tasks

Stopping vxsvc
Stopping vxodm
Stopping vxdbd
Stopping vxcpserv
Stopping had
Stopping CmdServer
Stopping amf
Stopping vxfen
Stopping gab
Stopping llt
Stopping sfmh-discovery

101
Stopping vxdclid
Stopping vxatd
Stopping fdd
Stopping vxportal
Stopping vxfs
Stopping vxconfigbackupd
Stopping vxsited
Stopping vxattachd
Stopping vxcached
Stopping vxrelocd
Stopping vxnotify
Stopping vxesd
Stopping vxconfigd
Stopping vxspec
Stopping vxio
Stopping vxdmp
Observe the following rpms being removed
Performing SFHA preremove tasks

Uninstalling VRTSvcsea
Uninstalling VRTSodm
Uninstalling VRTSdbed
Uninstalling VRTScscm
Uninstalling VRTSvcsdr
Uninstalling VRTSvcsag
Uninstalling VRTScps
Uninstalling VRTSvcs
Uninstalling VRTSamf
Uninstalling VRTSvxfen
Uninstalling VRTSgab
Uninstalling VRTSllt
Uninstalling VRTSfssdk
Uninstalling VRTSlvmconv
Uninstalling VRTSspt
Uninstalling VRTSob
Uninstalling VRTSatServer
Uninstalling VRTSatClient
Uninstalling VRTSvxfs
Uninstalling VRTSvxvm

B95
Uninstalling VRTSperl
Uninstalling VRTSvlic
Performing SFHA postremove tasks
m Observe the following new and updated rpms being installed.
102 B96
Performing SFHA preinstall tasks

Installing VRTSvlic rpm
Installing VRTSperl rpm
Installing VRTSspt rpm
Installing VRTSvxvm rpm
Installing VRTSaslapm rpm
Installing VRTSob rpm
Installing VRTSlvmconv rpm
Installing VRTSvxfs rpm
Installing VRTSfsadv rpm
Installing VRTSfssdk rpm
Installing VRTSllt rpm
Installing VRTSgab rpm
Installing VRTSvxfen rpm
Installing VRTSamf rpm
Installing VRTSvcs rpm
Installing VRTScps rpm
Installing VRTSvcsag rpm
Installing VRTSvcsdr rpm
Installing VRTSvcsea rpm
Installing VRTSdbed rpm
Installing VRTSodm rpm
Installing VRTSsfcpi60 rpm
Installing VRTSvbs
Performing SFHA postinstall tasks
Observe the new SFHA elements being started.
Performing SFHA upgrade configuration

Starting vxdmp
Starting vxio
Starting vxspec
Starting vxconfigd
Starting vxesd

Starting vxrelocd
Starting vxcached
Starting vxconfigbackupd
Starting vxattachd
Starting vxsvc
Starting sfmh-discovery
Starting vxportal
Starting fdd
Starting llt
Starting gab
Starting vxfen
Starting amf
Starting had
Starting CmdServer
Starting vxdbd
Starting vxodm
Performing SFHA poststart tasks
Observe that the CPI attempts to connect to the SORT web site to
determine if there are product updates.
Note: Since the lab environment does not provide this connectivity, the
connection to SORT will fail.
103
Notice that the installer log files, summary file, and response file are saved
at:
Press Enter multiple times to review the summary file.
Note: The the CPI ends at this point and control is returned to the login shell.
End of Solution

B97
Navigate to the log file location. Then navigate to the sub directory named
upgrade and perform a recursive listing of the directory.
Note: Elements of the SFHA51.SP1 configuration were saved to assist in any
rollback of the upgrade.
Solution
104 B98
cd upgrade
ls -lR *
new_conf:
total 180
-rwx------rwx------rwx------rwx------rwx------rwx------rwx------rwx------
1
1
1
1
1
1
1
1
root
root
root
root
root
root
root
root
root
1087 Jan 12 14:01 Db2udbTypes.cf
root
1073 Nov 14 09:32 iSCSItypes.cf
root
122 Jan 12 14:01 main.cf
root 139763 Jan 12 14:01 main.cmd
root
636 Jan 12 14:01 OracleASMTypes.cf
root
1545 Jan 12 14:01 OracleTypes.cf
root
2329 Jan 12 14:01 SybaseTypes.cf
root 12125 Nov 14 09:32 types.cf
old_conf:
total 172
-rwx------rwx------rwx------rwx------rwx------rwx------rwx------
1
1
1
1
1
1
1
root
root
root
root
root
root
root
root
root
122 Jan 12 14:01 main.cf
root
root
root
root 11310 Oct 1 2010 types.cf
old_config:
total 236
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
-rwx------ 1
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
944 Jan 12 13:50 Db2udbTypes.cf.12Jan2012.13.50.56
root
944 Oct 20 16:03 Db2udbTypes.cf.previous
root
3026 Jan 12 13:50 main.cf
root
3026 Jan 12 13:50 main.cf.12Jan2012.13.50.56
root
3026 Dec 8 12:24 main.cf.previous
root
root
632 Jan 12 13:50 OracleASMTypes.cf.12Jan2012...
root
632 Oct 20 16:03 OracleASMTypes.cf.previous
root
root
1434 Jan 12 13:50 OracleTypes.cf.12Jan2012.13.50.56
root
1434 Oct 20 16:03 OracleTypes.cf.previous
root
root
836 Jan 12 13:50 SybaseTypes.cf.12Jan2012.13.50.56
root
836 Oct 20 16:03 SybaseTypes.cf.previous
root 11170 Jan 12 13:50 types.cf
root 11170 Jan 12 13:50 types.cf.12Jan2012.13.50.56
root 11170 Oct 20 16:03 types.cf.previous

result_config:
total 156
-rwx------ 1 root root
4947 Jan 12 14:01 custom.cmd
-rwx------ 1 root root 143796 Jan 12 14:01 main.cmd
End of Solution
Navigate to the /opt/VRTS/install directory and perform a version

check of the SFHA cluster on sym3 and sym4 using the SFHA 6.0 CPI.
Note: At the end of the version check, the CPI will attempt to connect to the
SORT web site to determine if there are product updates. Since the lab
environment does not provide this connectivity, the connection to
SORT will fail.
Solution
./installsfha -version sym3 sym4

Veritas Storage Foundation and High Availability - 6.0
Product:
105
Packages:
Installed Required packages for Veritas Storage Foundation and High
Availability 6.0:
#PACKAGE
#VERSION
VRTSamf
6.0.000.000
VRTSaslapm
6.0.000.000
VRTSfsadv
6.0.000.000
VRTSgab
6.0.000.000
VRTSllt
6.0.000.000
VRTSperl
5.12.2.6
VRTSsfcpi60 6.0.000.000
VRTSvcs
6.0.000.000
VRTSvcsag
6.0.000.000
VRTSvlic
3.02.60.007
VRTSvxfen
6.0.000.000
VRTSvxfs
6.0.000.000
VRTSvxvm
6.0.000.000
Installed Optional packages for Veritas Storage Foundation and High
Availability 6.0:
#PACKAGE
#VERSION
VRTScps
6.0.000.000
VRTSdbed
6.0.000.000
VRTSfssdk
6.0.000.000
VRTSob
3.4.526
VRTSodm
6.0.000.000
VRTSsfmh
4.1.119.0
VRTSspt
6.0.000.000
VRTSvbs
6.0.000.000
VRTSvcsdr
6.0.000.000

B99
VRTSvcsea
6.0.000.000
Summary:
Packages:
13 of 13 required Veritas Storage Foundation and High Availability
6.0
packages installed
11 of 11 optional Veritas Storage Foundation and High Availability
6.0
packages installed
Attempted to connect to https://sort.symantec.com to check for product
updates,
but connection failed.
Please visit https://sort.symantec.com to check for available product
updates
and information.
Product:
Packages:
Installed Required packages for Veritas Storage Foundation and High
Availability 6.0:
#PACKAGE
#VERSION
VRTSamf
6.0.000.000
VRTSaslapm
6.0.000.000
VRTSfsadv
6.0.000.000
VRTSgab
6.0.000.000
VRTSllt
6.0.000.000
VRTSperl
5.12.2.6
VRTSsfcpi60 6.0.000.000
VRTSvcs
6.0.000.000
VRTSvcsag
6.0.000.000
VRTSvlic
3.02.60.007
VRTSvxfen
6.0.000.000
VRTSvxfs
6.0.000.000
VRTSvxvm
6.0.000.000
106 B100
Installed Optional packages for Veritas Storage Foundation and High

Availability 6.0:
#PACKAGE
#VERSION
VRTScps
6.0.000.000
VRTSdbed
6.0.000.000
VRTSfssdk
6.0.000.000
VRTSob
3.4.526
VRTSodm
6.0.000.000
VRTSsfmh
4.1.119.0
VRTSspt
6.0.000.000
VRTSvbs
6.0.000.000
VRTSvcsdr
6.0.000.000
VRTSvcsea
6.0.000.000
Summary:
Packages:
13 of 13 required Veritas Storage Foundation and High Availability
6.0
packages installed
11 of 11 optional Veritas Storage Foundation and High Availability
6.0

packages installed
Attempted to connect to https://sort.symantec.com to check for product
updates,
but connection failed.
Please visit https://sort.symantec.com to check for available product
updates
and information.
End of Solution
Perform a post installation check on the upgraded SFHA 6.0 cluster on sym3
and sym4 using the following information. To facilitate reviewing the log file
if errors occur, record the directory /opt/VRTS/install/logs directory
that contains the installer log files.
RHEL system names: sym3 and sym4
Do not view the summary file
log files:
Solution
./installsfha -postcheck sym3 sym4
107
Installed product
Platform version
Product prechecks
Observe the following post installation checks.
Checking processes and drivers status

Checking lltstat status
Checking llt configuration
Checking installed VRTSllt
Checking llt starting setting
Checking clusterid configuration
Checking gabconfig ports status
Checking installed VRTSgab

B101
Checking GAB starting setting

Checking vxfen status
Checking installed VRTSvxfen
Checking had status
Checking installed VRTSvcs
Checking Volume Manager status
saved at:
To NOT view the summary file, type: n
End of Solution
108 B102

Exercise 7: Examining the VCS configuration

In this exercise, the VCS configuration for the newly upgraded east cluster
composed of the sym3 and sym4 systems is briefly examined. A comprehensive
examination of SFHA 6.0 configurations is included in the remaining labs.
sym3
Note: You may optionally perform the steps in this exercise on both the sym3
and sym4 cluster nodes, though it is not strictly necessary to do so.
From sym3:terminal1, use the lltstat command to display the Low

Latency Transport (LLT) active status.
Solution
lltstat -nvv active

Node
* 0 sym3
1 sym4
State
OPEN
Link
Status
Address
eth4
eth5
UP
UP
00:0C:29:97:FB:5D
00:0C:29:97:FB:67
eth4
eth5
UP
UP
00:0C:29:C4:1D:0A
00:0C:29:C4:1D:14
OPEN
End of Solution
109
Use the gabconfig command to display the Group Membership and Atomic
Broadcast (GAB) status.
Solution
gabconfig -a
===============================================================
Port a gen
Port b gen
Port h gen
End of Solution

B103
Use the hastatus command to display the cluster summary status.

Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
sym3
sym4
RUNNING
RUNNING
0
0
--B
B
B
B
B
B
GROUP STATE
Group
ClusterService
ClusterService
dbsg
dbsg
websg
websg
System
sym3
sym4
sym3
sym4
sym3
sym4
Probed
Y
Y
Y
Y
Y
Y
AutoDisabled
N
N
N
N
N
N
State
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
End of Solution
Examine the LLT configuration by displaying the contents of the

/etc/llttab and /etc/llthosts files.
Solution
cat /etc/llttab
set-node sym3
set-cluster 777
link eth4 eth-00:0c:29:97:fb:5d - ether - link eth5 eth-00:0c:29:97:fb:67 - ether - -
cat /etc/llthosts
0 sym3
1 sym4
End of Solution
110
Examine the GAB configuration and verify that the number of systems in the
cluster matches the value for the -n flag in the /etc/gabtab file.
Solution
cat /etc/gabtab
/sbin/gabconfig -c -n2
End of Solution
B104

Verify the VCS cluster name and system names by examining the
/etc/VRTSvcs/conf/config/main.cf file.
Solution
more /etc/VRTSvcs/conf/config/main.cf
include
include
include
include
include
"OracleASMTypes.cf"
"types.cf"
"Db2udbTypes.cf"
"OracleTypes.cf"
"SybaseTypes.cf"
cluster east (
UserNames = { admin = hOPhOJoLPkPPnXPjOM, oper = GnnVohMmoI }
ClusterAddress = "10.10.2.52"
Administrators = { admin }
UseFence = SCSI3
)
system sym3 (
)
system sym4 (
)
group ClusterService (
SystemList = { sym3 = 0, sym4 = 1 }
AutoStartList = { sym3, sym4 }
OnlineRetryLimit = 3
--More--(14%)
...
Press the space bar multiple times to display the contents of the file.
End of Solution
Review the VCS engine log file located at

/var/VRTSvcs/log/engine_A.log.
Solution
111
more /var/VRTSvcs/log/engine_A.log
#
#
#
#
#
#
Log Name:
System:
SysInfo:
Created:
2011/11/02
2011/11/02
2011/11/02
2011/11/02
2011/11/02
2011/11/02
engine
sym3
Linux:sym3,#1 SMP Tue Mar 16 21:52:39 EDT 2010,2.6.18...
2011/11/02 08:12:16
08:12:16
08:12:16
08:12:16
08:12:16
08:12:16
08:12:16
VCS
VCS
VCS
VCS
VCS
VCS
NOTICE V-16-1-11022 VCS engine (had) started

NOTICE V-16-1-11050 VCS engine version=6.0
NOTICE V-16-1-11051 VCS engine join version=6.0 ...
NOTICE V-16-1-11052 VCS engine pstamp=6.0.000 ...
NOTICE V-16-1-10114 Opening GAB library
INFO V-16-1-10196 Cluster logger started

B105
2011/11/02 08:12:16
2011/11/02 08:12:16
2011/11/02 08:12:16
2011/11/02 08:12:16
--More--(7%)
...
VCS
VCS
VCS
VCS
NOTICE V-16-1-10619 'HAD' starting on: sym3

INFO V-16-1-51138 Number of processors ...
WARNING V-16-1-51140 In a multi-CPU system, ...
INFO V-16-1-10125 GAB timeout set to 30000 ms ...
Press the space bar multiple times to display the contents of the file. Type
q after browsing a few pages as it is not necessary to view the entire log
file.
End of Solution
End of lab
112
B106

Lab 5: Performing common VCS operations

In this lab, you become familiar with the VCS Cluster Manager Java Console
with the preconfigured (and upgraded to SFHA 6.0 in lab 4)) east cluster on sys3
and sym4.
This lab contains the following exercises.
A verification that the virtual machines needed for this are powered on and
Exercise 2: Orientation
Terminal windows on sym3 and sym4 are set up and the Veritas Cluster
Manager - Java Console is opened.
113
Exercise 3: Viewing status and attributes

The Veritas Cluster Manager Java Console is used to view the status of systems
and service groups
Exercise 4: Manipulating service groups
Service groups are manipulated.
Exercise 5: Manipulating resources
Resources are manipulated.
Exercise 6: Modifying resource attributes
Resource attributes are modified.
Exercise 7: Adding a service group
A service group is added and its resources configured.

B107
Exercise 8: Copying a service group

A copy of a service group is created by copying and modifying an existing service
groups resources.
114
B108


1
opened.
mgt
winclient
Account: Administrator
Password: veritas
sym3
Account: root
Password: veritas
sym4
Account: root
Password: veritas
115

terminal2.
If you have machines running that are not used in this lab, shut down the
operating system and power the machines off.
power on and log into winclient.

B109
Exercise 2: Orientation
In this exercise, you setup the terminal windows on sym3 and sym4 that are
needed to perform this lab and open the Veritas Cluster Manager Java Console
on winclient.
sym3
1
From the sym3:terminal1 window, use the hagrp -state command to

verify that all service groups are online on sym3. If any are online on sym4
use the hagrp -switch command to switch them sym3.
Note: The service groups may be online on sym4 if sym4 was brought online
before sym3. The following steps show how to switch service groups
that are online on sym4 to sym3 and how to reverify they are on
sym3.
Solution
hagrp -state
#Group
ClusterService
ClusterService
dbsg
dbsg
websg
websg
116
Attribute
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
Value
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
hagrp -switch dbsg -to sym3
hagrp -switch websg -to sym3
hagrp -switch ClusterService -to sym3
hagrp -state
#Group
ClusterService
ClusterService
dbsg
dbsg
websg
websg
Attribute
State
State
State
State
State
State
System
sym3
sym4
sym3
sym4
sym3
sym4
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
End of Solution
B110

Navigate to the /var/tmp directory and verify its contents by displaying a

long listing of the db* and web* files.
Note: These files are associated with the numerous resources of type
FileOnOff in the VCS configuration.
Solution
cd /var/tmp
ls -l db* web*
-rw-------rw-------rw-------rw-------rw-------rw-------rw-------rw-------rw-------rw-------rw-------rw-------rw-------
1
1
1
1
1
1
1
1
1
1
1
1
1
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
0
0
0
0
0
0
0
0
0
0
0
0
0
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
1
1
1
1
1
1
1
1
1
1
1
1
1
12:33
12:33
12:33
12:33
12:33
12:33
12:33
12:33
12:33
12:33
12:33
12:33
12:33
dbdg
dbip
dblistener
dbmnt
dbnic
dboracle
dbvol
webapache
webdg
webmnt
webnic
webvip
webvol
End of Solution
winclient
117
Open the Veritas Cluster Manager Java Console and add a new cluster for
the east cluster using the following information:
Host name: east (alternately, use east.example.com or 10.10.2.52)
Port: 14141 (the default value)
Failover retries: 12 (the default value)
Solution
From the winclient computers desktop, double-click the Veritas

Cluster Manager Java Console icon.
From the Cluster Monitor window, from the menu, select File > New
Cluster.
Note: Alternately, click the New Cluster button.

B111
From the New Cluster window, in the Host name field, type: east
Note: Alternately, you may substitute the IP address, 10.10.2.52, or the
fully qualified host name, east.example.com. If you do so,
substitute this name in the lab instructions that reference the east
cluster.
Leave the Port and Failover retires fields at their default values, 14141
and 12 respectively.
Click OK.
From the east:14141 - Login window, click Cancel.
End of Solution
118
B112

Exercise 3: Viewing status and attributes

In this exercise, you use the Veritas Cluster Manager Java Console to view the
status of systems and service groups.
winclient
1
From the Veritas Cluster Manager Java Console (opened in the previous
exercise), log into the east cluster using the following credentials:
User name: oper
password: oper
Note: After login, the cluster explorer window will be displayed.
Solution
From the Cluster Monitor window, click Click here to log in.
In the User name field, type: oper
In the Password field, type: oper
Click OK.
End of Solution
119
From the east Cluster Explorer window, determine the number of cluster
members.
Number of members in this cluster:
Solution
Note: Steps a through c may be redundant as the default starting view of the
Cluster Explorer gives the same results as following these steps.

B113
From the left pane, click the Service Groups tab.
Select the east cluster.
From the right pane click the Status tab.
Notice that the Systems label points to two columns, one for sym3 and one
for sym4.
End of Solution
Determine the status, either online or offline, of the service groups.

Groups
System sym3
System sym4
ClusterService
Online
Offline
dbsg
Online
Offline
websg
Online
Offline
Solution
From the right pane, notice that for sym3, ClusterService, dbsg and
websg are Online.
Notice that for sym4, ClusterService, dbsg and websg are Offline.
End of Solution
120 B114
Determine which service groups have the service group operator privileges set
for the oper account.
Service Group
Operator privileges set for oper account?
ClusterService
No
dbsg
No
websg
Yes
Solution
From the left pane, select the east > ClusterService service group.
From the right pane, click the Properties tab.

Click Show all attributes.
From the Attributes View window, scroll down and select the Operators
attribute.
Notice that the Value column is blank.
Close the Attributes View window.
Repeat steps a f for the dbsg service group and the websg service group
to find that only the websg service group grants Operator privileges for
the oper account.
End of Solution
Determine which resources in the websg service group have the Critical
resource attribute enabled.
Resource
Critical resource attribute set?
webapache
No
webmnt
Yes
webvip
Yes
webvol
No
webnic
Yes
webdb
No
Solution
121
From the left pane, select the east > websg service group.
From the right pane, click the Resources tab.

B115
For each of the resources, mouse over the resource and notice from the tool
tip if Critical is set.
End of Solution
Determine which resource is the top-most parent in the dbsg service group and
on which child resource does the dboracle resource depend.
top-most parent in the dbsg service group:
dblistener
child resource of dboracle:
dbmnt
Solution
122 B116
From the left pane select the east > dbsg service group.
From the right pane, click the Resources tab.
Notice that dblistener is the top-most parent resource.

Notice that the dboracle resource depends on its child resource, dbmnt.
End of Solution
123

B117
Exercise 4: Manipulating service groups

In this exercise, you manipulate service groups.
winclient
1
From the east Cluster Explorer window (opened and logged into in a
previous exercises), attempt to take the ClusterService service group offline
on the sym3 system. What happens and why?
You cannot take the ClusterService group offline because, as the oper user, you do
not have privileges for this service group.
Solution
124 B118
From left pane, right-click the east > ClusterService service group.
Notice that there is not an Offline menu item to allow you to take the
ClusterService service group offline.
End of Solution

Attempt to take the websg service group offline on the sym4 system. What
happens and why?
Although, as the oper user, you have privileges to take the websg service group
offline, websg is a failover service group and is currently online on sym3. Therefore,
websg is already offline on sym4 and not provided as an alternative selection in the
menu.
Solution
125
From the left pane, right-click the east > websg service group and select
Offline.
Notice that there is not a sym4 menu item to take the take the websg
service group offline on sym4.
End of Solution

B119
Attempt to take the websg service group offline on sym3. What happens and
why?
The Offline selection is displayed for the websg service group and as the oper user,
you have privileges to take the group offline. Since websg is online on sym3, it can be
taken offline.
Solution
Offline > sym3.
From the Question window, click Yes.
End of Solution
Attempt to bring the websg service group online on sym4. What happens and
why?
As the oper user, you have privileges to bring the websg service group online. Since
websg is offline on both sym3 and sym4, it can be brought online on sym4.
Solution
126 B120
Online > sym4.
End of Solution
Attempt to take the websg service group offline everywhere. What happens
and why?
As the oper user, you have privileges to take the websg service group offline. Since
websg is online on sym4, it can be taken offline everywhere.
Solution
Offline > All systems.
From the result window, read the message and click OK.
End of Solution
Bring the websg service group online on sym4.

Solution
Online > sym4.
End of Solution
Attempt to switch the websg service group to the sym3 system. What happens
and why?
As the oper user, you have privileges to switch the websg service group. Since websg
is online on sym4 and offline on sym3, it can be switched to sym3. In a two node
cluster, selecting Any System would also switch websg from sym4 to sym3.
127
Solution
Switch To > sym3.
End of Solution

B121
Exercise 5: Manipulating resources

In this exercise, you manipulate resources.
winclient
1
Log out of the east Cluster Explorer window (opened and logged into in a
previous exercises) and from the Cluster Monitor window, log into the
10.10.2.52 cluster with the following credentials:
User name: admin
password: password
Solution
From the menu, select File > Log Out.
From the Cluster Monitor window, click Click here to log in.
In the User name field, type: admin
In the Password field, type: password
Click OK.
End of Solution
128 B122
Attempt to take the dblistener resource in the dbsg service group offline on
the sym3 system. What is the status of dbsg? Why does dbsg not fail over to
the sym4 system?
After taking the dblistener resource offline, the dbsg service group shows a status of
Partial Online on the sym3 system. The dbsg does not fail over even though
dblistener is marked as critical. Taking a resource offline does not cause a failover.

Solution
From the left pane, select the east > dbsg service group.
From the right pane, select the Resources tab.
Right-click the dblistener resource and select Offline > sym3.
From the left pane, select the east cluster.
From the right pane, notice the status of dbsg on sym3 is Partial Online.
End of Solution
Bring the dblistener resource online on the sym3 system.

Solution
Right-click the dblistener resource and select Online > sym3.
End of Solution
129
Attempt to take the dbmnt resource offline on the sym3 system. What
happens and why?
You cannot take the dbmnt resource offline because a dependent resource (dboracle)
is online.
Solution
From the right pane, right-click the dbmnt resource and select Offline >
sym3.

B123
Notice the error window stating Online resources depend on resource

dbmnt.
Click OK.
End of Solution
Attempt to bring the dblistener resource online on the sym4 system. What
happens and why?
You cannot bring the dblistener resource online on the sym4 system because dbsg is
a failover service group that is already online on sym3.
Solution
From the right pane, right-click the dblistener resource and select
Online > sym4.
Notice the error window stating Cannot online: resources group is not
offline elsewhere.
End of Solution
sym3
130 B124
From sym3:terminal1, fault the dboracle resource in the dbsg service group
by removing the /var/tmp/dboracle file and executing the command
hares -probe dboracle -sys sym3.
Solution
cd /var/tmp
rm dboracle

hares -probe dboracle -sys sym3
End of Solution
winclient
From the east Cluster Explorer window, determine what happened to the
dbsg service group and the dboracle resource?
The dbsg service group has failed over to the sym4 system due to the dboracle
resource being faulted.
Solution
From the right pane, from the bottom bar, notice that the sym3 icon is
surrounded by a red box indicating it has faulted.
Click the sym3 icon.
Notice that the dboracle resource is marked with a red X indicating it has
faulted.
From the left pane, select the east cluster.
From the right pane, notice that the status of the dbsg service group is
Faulted on sym3 and Online on sym4.
End of Solution
131
Review the log entries surrounding the fault.

Solution
From the menu, select View > Logs.

Note: Alternately, click the Show the Logs button.

B125
From the Logs window, Scroll through the log entries paying particular
attention to any Critical, Error or Warning entries.
Close the Logs window.
End of Solution
Attempt to switch the dbsg service group back to the sym3 system. What
happens and why?
The Switch To option is not available. You cannot switch a service group to a system
where it is faulted.
Solution
From the left pane, right-click the east > dbsg service group.
Notice that the Switch To option is unavailable.
End of Solution
10 Clear the fault on the dboracle resource in the dbsg service group.
Solution
132 B126
From the bottom display bar, click sym3.
Right-click the dboracle resource and select Clear Fault > sym3.
End of Solution

11 Switch the dbsg service group back to sym3.

Solution
From the left pane, right-click the east > dbsg service group and select
Switch To > sym3.
End of Solution
133

B127
Exercise 6: Modifying resource attributes

In this exercise, you modify resource attributes.
winclient
1
previous exercises), open the cluster for update.
Solution
From the menu, select File > Open Configuration.

End of Solution
Set all resources for the dbsg service group to critical. These resources should
be:
dbip
dbnic
dbmnt
dbvol
dbdg
Solution
134 B128
From the left pane, select east > dbsg.
Right-click the dbip resource and select Critical.
Right-click the dbnic resource and select Critical.
Right-click the dbmnt resource and select Critical.
Right-click the dvol resource and select Critical.
Right-click the dbdg resource and select Critical.

For each of the dbsg resources, mouse over the resource and notice from
the tool tip that Critical is set.
Note: If Critical is not set, then right-click the resource and select
Critical.
From menu select File > Save Configuration.
End of Solution
Set all resources for the websg service group to critical. These resources
should be:
webapache
webdg
webvol
Solution
135
From the left pane, select east > websg.
Right-click the webapache resource and select Critical.
Right-click the webvol resource and select Critical.
Right-click the webdg resource and select Critical.
For each of the websg resources, mouse over the resource and notice from
the tool tip that Critical is set.
Note: If Critical is not set, then right-click the resource and select
Critical.
From the menu select File > Save Configuration.
End of Solution

B129
Exercise 7: Adding a service group

In this exercise, you add a service group and configure its resources.
winclient
1
previous exercises), add a new failover service group named appsg with a
system list of sym3 and sym4 and configure it to auto start on either system
with preference for sym3.
Note: This is a faux service group similar to the websg and dbsg service
groups. The resources that will added to this group in a later step will
all be of type FileOnOff.
Solution
From the menu, select Edit > Add >Service Group.
Note: Alternately, click the Add Service Group button.
136 B130
From the Add Service Group window, in the Service Group name field,
type: appsg
From the Available Systems list, double-click sym3.
Double-click sym4.
From the Systems for Service Group list, from the Startup column, mark
Startup for both systems.
From the Service Group Type option, select Failover.
Click OK.
From the east Cluster Explorer window, from the left pane, select
east > appsg.
From the right pane, select the Properties tab.

From the Attributes View window, scroll down to the SystemList

attribute, expand the Values entry in the Value column and notice the
systems and their priorities.
Note: You may review other attributes to confirm that the appsg service
group has been configured correctly.
End of Solution
Add a resource using the following information.

Name: appnic
Type: FileOnOff
PathName attribute
Scalar Value: /var/tmp/appnic
Option: Global
The resource is NOT Critical (initially).
Enable the resource.
Solution
137
From the east Cluster Explorer window, from the right pane, select the
Resources tab.
Right-click anywhere in the Resource View pane and click Add Resource.
From the Add Resource window, in the Resource name field, type:
appnic
From the Resource Type drop-down menu, select FileOnOff.
From the attribute list, from the PathName attribute row, from the Edit
column, click the Edit icon.
From the Edit Attribute window, in the Scalar Value field, type:
/var/tmp/appnic
From the Scalar Value options, select Global.

B131
Click OK.
From the Add Resource window, clear Critical.
Mark Enabled.
Click OK.
End of Solution
Bring the appnic resource Online on sym3. Then, make the resource critical
and review its attributes to confirm it has been configured correctly. Finally,
save the configuration.
Note: A real resource of type NIC does not have to be brought online. The
appnic resource is a faux NIC resource of type FileOnOff and so it
does need to be brought online.
Solution
138 B132
From the east Cluster Explorer window, from the right pane, right-click
the appnic resource and select Online > sym3.
Mouse over the appnic resource and notice from the tool tip that it is online
on sym3, not critical and Enabled.
Right-click the appnic resource and select Critical.
Right-click the appnic resource and select View > Properties View.
From the Properties View window, click Show all attributes.
From the Attributes window, review the Critical, Enabled and File Name
attributes.
Note: You may review other attributes to confirm that the appnic
resource has been configured correctly.

Close the Properties View window.
From the east Cluster Explorer window, from the menu, select
File > Save Configuration.
End of Solution
139
Repeat step 2 create and configure the following resources.

appip resource
Name: appip
Type: FileOnOff
PathName attribute
Scalar Value: /var/tmp/appip
Option: Global
appdg resource
Name: appdg
Type: FileOnOff
PathName attribute
Scalar Value: /var/tmp/appdg
Option: Global
appvol resource
Name: appvol
Type: FileOnOff
PathName attribute
Scalar Value: /var/tmp/appvol
Option: Global
appmnt resource
Name: appmnt
Type: FileOnOff
PathName attribute
Scalar Value: /var/tmp/appmnt
Option: Global

B133

appproc resource
Name: appproc
Type: FileOnOff
PathName attribute
Scalar Value: /var/tmp/appproc
Option: Global
5
Repeat step 3 to bring the appip, appdg, appvol, appmnt and appproc
resources Online on sym3. Then, make the resources critical and review their
attributes to confirm they have been configured correctly. Finally, save the
configuration.
Note: After the resources have been created and brought online, the resource
pane for the appsg service group should appear similar to the following:
140 B134
Link the resources in the appsg service group using the following information.
Then, save the configuration.
appip depends on appnic

appvol depends on appdg
appmnt depends on appvol
appproc depends on appmnt
appproc depends on appip
Solution
From the Resource View (right) pane, right-click the appip resource and
select Link.
From the Link Resource window, select appnic.

Click OK.
Right-click the appvol resource and select Link.
From the Link Resource window, select appdg.
Click OK.
Right-click the appmnt resource and select Link.
From the Link Resource window, select appvol.
Click OK.
Right-click the appproc resource and select Link.
From the Link Resource window, select appmnt.
Click OK.
m Right-click the appproc resource and select Link.

n
From the Link Resource window, select appmnt.
Click OK.
From the menu, select File > Save Configuration.
End of Solution
Note: After the resources have been created and brought online, the resource
pane for the appsg service group should appear similar to the following:
141

B135
Exercise 8: Copying a service group

In this exercise, you create a copy of a service group by copying and modifying an
existing service groups resources.
winclient
1
previous exercises), add a new failover service group named testappsg with a
system list of sym3 and sym4 and configure it to auto start on either system
with preference for sym3.
Note: This is a faux service group similar to the websg, dbsg, and appsg
service groups. The resources that will added to this group in a later
step will all be of type FileOnOff.
Solution
From the menu, select Edit > Add >Service Group.
Note: Alternately, click the Add Service Group button.
142 B136
From the Add Service Group window, in the Service Group name field,
type: testappsg
From the Available Systems list, double-click sym3.
Double-click sym4.
From the Systems for Service Group list, from the Startup column, mark
Startup for both systems.
From the Service Group Type option, select Failover.
Click OK.
From the east Cluster Explorer window, from the left pane, select
east > teatappsg.

From the right pane, select the Properties tab.
From the Attributes View window, scroll down to the SystemList

attribute, expand the Values entry in the Value column and notice the
systems and their priorities.
Note: You may review other attributes to confirm that the testappsg
service group has been configured correctly.
End of Solution
Copy the resources from the appsg service group to the currently empty
testappsg service group.
Solution
From the left pane, select east > appsg.
Right-click the appproc resource and select Copy > Self and Child
Nodes.
From the left pane, select east > testappsg.
From the right pane, right-click in the blank Resource View pane and
select Paste.
From the Name Clashes window, from the Correction drop-down menu,
select Insert at the beginning.
In the Correction text box, type: test
Click Apply.
Click OK.
Observe the resources and links being added.
End of Solution
143 Lab 5: Performing common VCS operations
B137
Modify the PathName attribute of each resource using the following

information.
testappproc PathName: /var/tmp/testappproc

testappip PathName: /var/tmp/testappip
testappnic Pathname: /var/tmp/testappnic
testappmnt PathName: /var/tmp/testappmnt
testappvol PathName: /var/tmp/testappvol
testappdg PathName: /var/tmp/testappdg
Solution
From the right pane, right-click the appproc resource and select
View > Properties View.
From the Properties View window, scroll down to the File Name attribute
and click the Edit button.
Note: The PathName attribute is labeled File Name in this window.
From the Edit Attribute window, in the Scalar Value field, type:
/var/tmp/testappproc
Note: Double-click the Scalar Value field to edit its contents.
144 B138
Click OK.
Close the Properties View window.
Repeat steps a through e for the testappip, testappnic, testappmnt,

testappvol and testappdg resources using the PathName specified above.
End of Solution
Enable each resource in the testappsg service group in reverse resource

dependency. The order is as follows:
testappnic
testappip
testappdg
testappvol
testappmnt
testappproc

Solution
From the right pane, right-click the testappnic resource and select
Enabled.
Wait a moment to confirm that the testappnic resource is NOT brought

online.
Note: If the resource is brought online, the PathName edit was not
performed correctly. If so, repeat the previous step to edit the
PathName attribute.
Repeat steps a and b for the testappip, testappdg, testappvol,

testappmnt, and testappproc resources.
End of Solution
Bring the testappsg service group online on sym3.

Solution
From the left pane right-click east > testappsg and select Online > sym3.
End of Solution
Save and close the cluster configuration.
Solution
From the menu, select File > Close Configuration.

End of Solution
145 Lab 5: Performing common VCS operations
B139
Log out of the east Cluster Explorer window. Then, close the Cluster
Monitor window.
Solution
From the menu, select File > Log Out.
From the Cluster Monitor window, from the menu, select File > Exit.
End of Solution
End of lab
146 B140

Lab 6: Starting and stopping VCS

In this lab, you stop and start VCS and set several cluster attributes and determine
their impact.
This lab contains the following exercises.
Exercise 2: VCS configuration state and stopping VCS
VCS is started and stopped.
Exercise 3: Configuring automatic backup of the VCS configuration

An automatic backup of the VCS configuration is configured.
Exercise 4: Setting non default VCS stop options
VCS stop options are configured.
147 Lab 6: Starting and stopping VCS
B141

1
opened.
mgt
sym3
Account: root
Password: veritas
sym4
Account: root
Password: veritas

terminal2.
148 B142
Note: If you are completing the lab exercises in order, this will require you
shutdown winclient.

Exercise 2: VCS configuration state and stopping VCS

In this exercise, you stop and start VCS.
sym3
1
From sym3:terminal2, use the tail -f command to monitor the

/var/VRTSvcs/log/engine_A.log VCS engine log.
Note: As cluster commands are executed in this lab, log entries will be added
to the VCS engine log and echoed to sym3:terminal2 where you can
easily review them.
Solution
tail -f /var/VRTSvcs/log/engine_A.log
End of Solution
From sym3:terminal1, use the haconf command to open the cluster

configuration for update. Then use the hastop command to attempt to stop
VCS and review the log entries in sym3:terminal2. What happens and why?
VCS cannot be stopped normally if the VCS configuration is open for update.
Solution
haconf -makerw
hastop -all
VCS WARNING V-16-1-10028 Cluster configuration is writable. Must do 'haconf
-dump -makero' or 'hastop -all -force'
From sym3:terminal2, notice the new log entries.
End of Solution
149 Lab 6: Starting and stopping VCS
B143
Use the haconf command to save and close the cluster configuration. Then,
use the hastop command to attempt to stop VCS on all systems and review
the log entries. What happens and why?
VCS shuts down normally. The VCS engine log indicates that all resources are
brought offline, meaning that all application elements under VCS management are
shutdown as well.
Solution
haconf -dump -makero
hastop -all
From sym3:terminal2, notice the new log entries.
End of Solution
Use the gabconfig command to check Group Membership Services and

Atomic Broadcast (GAB) port memberships on sym3 and sym4 by verifying
that port h is closed. What is the significance of this port being closed?
Port h membership should NOT be present for either system. This indicates that the
VCS engine, the high availability daemon (HAD), is not running.
Solution
150 B144
gabconfig -a
===============================================================
Port a gen
20f703 membership 01
Port b gen
ssh sym4 gabconfig -a

===============================================================
Port a gen
Port b gen
End of Solution

Use the hastart command to start VCS on sym3 and observe the engine
log.
Solution
hastart
From sym3:terminal2, notice the log entries.

...
2011/11/03 11:54:46 VCS
2011/11/03 11:54:46 VCS
2011/11/03 11:54:46 VCS
version=6.0.00.0
2011/11/03 11:54:46 VCS
6.0-2011-10-13-16.30.00
2011/11/03 11:54:46 VCS
2011/11/03 11:54:46 VCS
...
NOTICE V-16-1-11022 VCS engine (had) started

NOTICE V-16-1-11050 VCS engine version=6.0
NOTICE V-16-1-11051 VCS engine join
NOTICE V-16-1-11052 VCS engine pstamp=6.0.000.000NOTICE V-16-1-10114 Opening GAB library
NOTICE V-16-1-10619 'HAD' starting on: sym3
End of Solution
Start VCS on sym4 and observe the engine log.
Solution
151
ssh sym4 /opt/VRTS/bin/hastart
From sym3:terminal2, notice the log entries.

...
2012/01/14 16:10:11 VCS INFO V-16-1-10077 Received new cluster membership
2012/01/14 16:10:11 VCS NOTICE V-16-1-10322 System (Node '1') changed state
from UNKNOWN to INITING
2012/01/14 16:10:11 VCS NOTICE V-16-1-10086 System sym4 (Node '1') is in
Regular Membership - Membership: 0x3
2012/01/14 16:10:11 VCS NOTICE V-16-1-10453 Node: 1 changed name from:
'sym4' to: 'sym4'
2012/01/14 16:10:11 VCS NOTICE V-16-1-10322 System sym4 (Node '1') changed
state from UNKNOWN to INITING
state from INITING to CURRENT_DISCOVER_WAIT
state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/01/14 16:10:11 VCS INFO V-16-1-10455 Sending snapshot to node
membership: 0x2
state from REMOTE_BUILD to RUNNING
...
End of Solution

B145
Exercise 3: Configuring automatic backup of the VCS

configuration
In this exercise, you configure automatic backup of the VCS configuration.
sym3
1

configuration for update.
Solution
haconf -makerw
End of Solution
Navigate to the /etc/VRTSvcs/conf/config directory and perform a

long listing on main.cf.autobackup to confirm that the file does not
exist.
Solution
cd /etc/VRTSvcs/conf/config
ls -l main.cf.autobackup
ls: main.cf.autobackup: No such file or directory
End of Solution
152 B146
Change the BackupInterval cluster attribute to three minutes. Verify the value
of this cluster attribute before and after making the change.
Note: Do NOT save the cluster configuration at this time.
Solution
haclus -value BackupInterval

0

haclus -modify BackupInterval 3

3
End of Solution
Wait for three minutes (the backup interval) and then list the
main.df.autobackup files to verify that an autobackup file exists.
Solution
sleep 180
ls -l main.cf.autobackup
-rw------- 1 root root 4802 Jan 24 10:08 main.cf.autobackup
End of Solution
Use the diff command to compare the main.cf and

main.cf.autobackup files.
Solution
diff main.cf main.cf.autobackup

8a9
>
BackupInterval = 3
End of Solution
153
Save and close the VCS cluster configuration. Then, list the main.cf and
main.df.autobackup files. What has happened?
The main.cf.autobackup file has been deleted and a new main.cf file is saved.
Solution
ls -l main.cf main.cf.autobackup
ls: main.cf.autobackup: No such file or directory
-rw------- 2 root root 4802 Jan 24 10:12 main.cf
End of Solution

B147
Exercise 4: Setting non default VCS stop options

In this exercise, you configure VCS stop options.
sym3
1

configuration for update.
Note: Continue to monitor sym3:terminal2 which has been set up in a
previous exercise to display the VCS engine log.
Solution
haconf -makerw
End of Solution
Display the current EngineShutdown cluster attribute. Then, display the valid
vales for the EngineShutdown cluster attribute by assigning an invalid value
of foo. Finally, change and verify the EngineShutdown cluster attribute to
disable all hastop commands.
Solution
154 B148
haclus -value EngineShutdown

Enable
haclus -modify EngineShutdown foo

VCS WARNING V-16-1-10057 Invalid value provided for EngineShutdown.
Possible values are Enable, Disable, DisableClusStop, PromptClusStop,
PromptLocal, and PromptAlways.
haclus -modify EngineShutdown Disable

Disable
End of Solution
Save and close the cluster configuration and then attempt to stop the VCS
engine on all systems. What happens?
An error is given when the hastop -all command is used stating, Cannot stop engine
as EngineShutdown is set to Disable.
Solution
hastop -all
VCS ERROR V-16-1-10607 Cannot stop engine as EngineShutdown is set to
Disable
End of Solution
Attempt to force stop the VCS engine on all systems by overriding the
EngineShutdown setting. What happens?
The hastop -all -force command is successful
Solution
155
hastop -all -force
End of Solution
Confirm that the VCS engine is shutdown on both sym3 and sym4 by
determining if GAB port h is active. Can you display a summary cluster status
using the hastatus command?
The hastatus -sum command will not display summary information when the
VCS engine is shut down.

B149
Solution
gabconfig -a
===============================================================
Port a gen
Port b gen

===============================================================
Port a gen
Port b gen
hastatus -sum
VCS ERROR V-16-1-10600 Cannot connect to VCS engine
VCS WARNING V-16-1-11046 Local system not available
ssh sym4 /opt/VRTS/bin/hastatus -sum

End of Solution
Use the hastart command to start VCS on each system in the cluster.
Confirm VCS has started on sym3 before starting it on sym4.
Note: The double entry for port h indicates that the VCS engine is running on
sym4 and that GAB is running on sym3, but port h is closed
Solution
156 B150
hastart
gabconfig -a
===============================================================
Port a gen
e9dc01 membership 01
Port b gen
Port h gen
e9dc0d membership 0
Port h gen
e9dc0d
visible ;1

Note: The double entry for port h indicates that the VCS engine is
running on sym3 and that GAB is running on sym4, but port h is
closed.
c
End of Solution
Confirm that the VCS engine is started on both sym3 and sym4 by
determining if GAB port h is active.
Solution
gabconfig -a
===============================================================
Port a gen
Port b gen
Port h gen
End of Solution
Use the haconf command to open the cluster configuration for update and
change and verify the EngineShutdown attribute to prompt before allowing a
command to stop all systems in the cluster.
Solution
157
haconf -makerw

Disable
haclus -modify EngineShutdown PromptClusStop

PromptClusStop
End of Solution

B151
Save and close the cluster configuration and attempt to stop the VCS engine on
all systems. What happens?
When the EngineShutdown attribute is set to PromptClusStop, you are prompted to

confirm that all systems in the cluster will be stopped.
Solution
hastop -all
Are you sure you want to stop VCS (yes/no)?
y
Please type 'yes' or 'no':
yes
End of Solution
10 Confirm that the VCS engine is shutdown on both sym3 and sym4 by

Solution
158 B152
gabconfig -a
===============================================================
Port a gen
Port b gen
End of Solution
11 Start VCS on each system in the cluster using the hastart command.
Confirm VCS has started on sym3 before starting it on sym4.

Solution
hastart
gabconfig -a


===============================================================
Port a gen
Port b gen
Port h gen
e9dc0d membership 0
Port h gen
e9dc0d
visible ;1
Note: The double entry for port h indicates that the VCS engine is
running on sym3 and that GAB is running on sym4, but port h is
closed.
c
End of Solution
12 Confirm that the VCS engine is started on both sym3 and sym4 by

Solution
gabconfig -a
===============================================================
Port a gen
Port b gen
Port h gen
End of Solution
13 Use the hastop command to attempt to stop the VCS engine locally on
sym3. What happens?
VCS is stopped on the local system without any prompt to confirm because
159
EngineShutdown is set to PromptClusStop. If EngineShutdown was set to

PromptLocal, you would have been prompted for confirmation.
Solution
hastop -local
Notice that VCS is stopped on the local system without any prompt.
End of Solution

B153
14 Confirm that the VCS engine is stopped on sym3, but not on sym4 by

Solution
gabconfig -a
===============================================================
Port a gen
Port b gen

===============================================================
Port a gen
Port b gen
Port h gen
20f715 membership ;1
Port h gen
20f715
visible 0
Note: The double entry for port h indicates that the VCS engine is running
on sym4 and that GAB is running on sym3, but port h is closed
End of Solution
15 Use the hastart command to start VCS on sym3 cluster and confirm that
the VCS engine is started on both sym3 and sym4 by determining if GAB
port h is active.
Solution
160 B154
hastart
gabconfig -a
===============================================================
Port a gen
Port b gen
Port h gen
End of Solution

16 Use the haconf command to open the cluster configuration for update and
change and verify the EngineShutdown attribute to Enable. Then save and
close the cluster configuration.
Solution
haconf -makerw

PromptClusStop
haclus -modify EngineShutdown Enable

Enable
End of Solution
17 End the monitoring of the engine log.

Solution
From sym3:terminal2, press Ctrl-C.

End of Solution
End of lab
161

B155
162 B156

Lab 7: Preparing application services

In this lab, you examine the preconfigured storage elements of a simple application
used in future labs.
Exercise 2: Examining storage for the service
The storage elements for the loopy application are examined.
Exercise 3: Examining the application

The loopy application script is examined to determine how it functions and to
determine the scripts output file.
163
Exercise 4: Manually starting and stopping the application

The loopy application is manually started and stopped.

B157

1
opened.
mgt
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas

terminal2.
164 B158
shutdown sym3 and sym4, and power on and log into sym1 and sym2.

Exercise 2: Examining storage for the service

In this exercise, you examine the storage elements for the loopy application.
Note: Storage for the remaining labs has been preconfigured, including disk
groups, volumes, file systems, and mount points. Application elements
such as scripts, executables, etc. have been pre-staged as well.
sym1
1
From sym1:terminal1, use the vxdisk -o alldgs command to list the

preconfigured disks and identify preconfigured disk groups. Then import the
loopydatadg disk group on sym1 and confirm that the disk group has been
imported.
Note: The loopydatadg disk group and associated storage elements are used
is this lab. The symdbdatadg disk group is used in a later Oracle lab
and the sharedatadg disk group is used in a later NFS lab.
Solution
165
vxdisk -o alldgs list

DEVICE
emc0_dd1
emc0_dd2
emc0_dd3
emc0_dd4
emc0_dd5
emc0_dd6
emc0_dd7
emc0_dd8
emc0_dd9
emc0_d10
emc0_d11
emc0_d12
sda
sdb
TYPE
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:none
auto:none
DISK
-
GROUP
STATUS
(symdbdatadg) online
(loopydatadg) online
(sharedatadg) online
online
online
online
online
online
online
online invalid
online invalid
vxdg import loopydatadg
vxdg list
loopydatadg
enabled,cds
1319645563.37.sym1
End of Solution

B159
Use the vxinfo -g loopydatadg -p command to display volume

manager components associated with the loopydatadg disk group.
Note: The simple non-mirrored volume name is loopydatavol. It is started
(active) on import and contains a preconfigured Veritas file system.
Solution
vxinfo -g loopydatadg -p
vol loopydatavol
fsgen
plex loopydatavol-01 ACTIVE
Started
End of Solution
Use the ls -ld /loopyfs command to confirm the presence of the loopy
file system mount point.
Solution
ls -ld /loopyfs
drwxr-xr-x 2 root root 4096 Oct 26 06:46 /loopyfs
End of Solution
Mount the Veritas file system created on the loopydatavol volume in the
loopydatadg disk group to the /loopyfs mount point and use the /bin/
mount command to confirm that the file system is mounted.
Solution
166 B160
/bin/mount -t vxfs /dev/vx/dsk/loopydatadg/

loopydatavol /loopyfs
df -h /loopyfs
df -h /loopyfs
Filesystem
Size Used Avail Use% Mounted on
/dev/vx/dsk/loopydatadg/loopydatavol
1.0G
19M 943M
2% /loopyfs
End of Solution

Exercise 3: Examining the application

In this exercise, you examine the loopy application script to determine how it
functions and to determine the scripts output file.
sym1
1
From sym1:terminal1, navigate to the /loopyfs mounted file system and

perform a log listing of its contents.
Solution
cd /loopyfs
ls -l
total 52
-rwxr-xr-x
-rw-r--r-drwxr-xr-x
-rwxr-xr-x
-rwxr-xr-x
1
1
2
1
1
root
root
root
root
root
root
310 Oct 26 12:10
root 49391 Nov 4 07:29
root
96 Oct 26 09:23
root
262 Oct 26 12:46
root
211 Oct 26 12:58
loopy
loopyout
lost+found
startloopy
stoploopy
End of Solution
Examine the contents of the loopy script. How does it function? How does it
use script arguments? To what file does it write output? Examine the contents
of the output file to confirm your findings.
The loopy script accepts one argument which it includes in date-time stamped
167
messages it appends to the loopyout file.
Solution
more loopy
#!/bin/ksh
# Loopy script for VCS class.
#############################
#
# $1 is Service Group name
# $0 is name of shell script being executed
#

B161
while true
do
echo `date` ${1} Loopy is alive >> ${0}out
sleep 4
echo `date` ${1} Loopy is still alive >> ${0}out
sleep 4
done
Notice that the comments at the beginning of the script indicate that the
first script argument, $1, should be a Service Group name.
Note: You will create the service group in a later lab.
Notice the while-do loop that continuously appends a message containing

the first script argument to the ${0}out file.
Notice that since the ${0} argument translates to the name of the script
which is loopy, ${0}out then translates to loopyout and so the message is
appended to the loopyout file.
head loopyout
Wed
Wed
Wed
Wed
Wed
Wed
Wed
Wed
Wed
...
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
26
26
26
26
26
26
26
26
26
11:29:51
11:29:55
11:29:59
11:30:03
11:30:07
11:30:11
11:30:15
11:30:19
11:38:23
PDT
PDT
PDT
PDT
PDT
PDT
PDT
PDT
PDT
2011
2011
2011
2011
2011
2011
2011
2011
2011
appsg
appsg
appsg
appsg
appsg
appsg
appsg
appsg
appsg
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
is
is
is
is
is
is
is
is
is
alive
still
alive
still
alive
still
alive
still
alive
alive
alive
alive
alive
End of Solution
168 B162

Exercise 4: Manually starting and stopping the application

In this exercise, you manually start and stop the loopy application.
sym1
1
From sym1:terminal1, use the ./loopy test & command to start the
loopy script in the background. Then, use the tail -f command to monitor
the loopyout file for a short time.
Solution
./loopy test &
tail -f loopyout
Fri
Fri
Fri
Fri
Fri
Nov
Nov
Nov
Nov
Nov
4
4
4
4
4
08:01:54
08:01:58
08:02:02
08:02:06
08:02:10
PDT
PDT
PDT
PDT
PDT
2011
2011
2011
2011
2011
test
test
test
test
test
Loopy
Loopy
Loopy
Loopy
Loopy
is
is
is
is
is
still alive
alive
still alive
alive
still alive
After a few lines are printed, press: Ctrl-C
End of Solution
169
To practice configuring a virtual IP address that can be associated with the

loopy application as a secondary IP address on the eth0 interface, do the
following.
Use the ip addr show eth0 command to determine the last eth0:<n>
label.
Use the ip addr add command to add a virtual IP address of 10.10.2.61
to the eth0 interface with a netmask of 255.255.255.0 (/24) and broadcast
address of 10.10.2.255.
Use the ip addr show eth0 command to confirm the new address.
Use the ping 10.10.2.61 command to test the new address.
Use the ip addr del command to remove the virtual IP address.

B163
Use the ip addr show eth0 command to confirm the new address has
been removed.
Solution
ip addr show eth0

2 : eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
1000
inet 10.10.2.51/24 brd 10.10.2.255 scope global secondary eth0:0
Note: Notice that there already is a virtual IP on eth0:0, so eth0:1 must be

used.
b
ip addr add 10.10.2.61/24 broadcast 10.10.2.255

dev eth0 label eth0:1
ip addr show eth0

1000
ping -c 3 10.10.2.61

170 B164
ip addr del 10.10.2.61/24 dev eth0
ip addr show eth0

1000

End of Solution
Using the ps -ef | grep loopy to identify the process id of the loopy
process. Then use the kill <process id> command to stop the loopy
script execution.
Solution
ps -ef | grep loopy

root
root
2406
2431
6047
6047
0 14:48 pts/3
0 14:49 pts/3
00:00:00 /bin/ksh ./loopy test

00:00:00 grep loopy
Notice the process id of the process that is executing ./loopy test.
kill <process id>
Press Enter.
[1]+
Terminated
./loopy test
ps -ef | grep loopy

root
10814
8924
0 11:08 pts/2
00:00:00 grep loopy
End of Solution
Navigate to the root directory, unmount /loopyfs and confirm that it has
been unmounted. Then deport the loopydatadg disk group and confirm the
deportation.
Solution
171
cd /
umount /loopyfs
/bin/mount | grep /loopyfs
vxdg deport loopydatadg
vxdg -list
End of Solution

B165
Note: It is often a good idea to manually migrate the elements of an application to

other cluster nodes and manually determine their suitability to run there.
You can do this as an optional exercise on sym2 by repeating the steps in
this exercise. In a later lab, once the application is under VCS control, the
VCS virtual firedrill will be used to provide similar suitability testing.
End of lab
172 B166

Lab 8: Online configuration of a service group

In this lab, you configure the loopy application in VCS, perform a fire drill, and
ensure that the application runs on both cluster systems.
Exercise 2: Creating a service group for the loopy application
A service group for the loopy application is created.
Exercise 3: Configuring resources for the loopy application

Resources for the loopy application are configured and brought online.
173
Exercise 4: Performing a virtual fire drill on the service group

A virtual fire drill on the appsg service group for the sym2 system is performed.
Exercise 5: Testing the service group
The appsg service group is switched from sym1 to sym2 and then back again.
Exercise 6: Setting resources to critical
The Critical attribute value is set to 1 (one) on all resources in the appsg service
group, thus making them critical resources.
Exercise 7: (Optional) Examining Veritas File System locking
Veritas file system locking by VCS is examined.

B167

1
opened.
mgt
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas

terminal2.
174 B168

Exercise 2: Creating a service group for the loopy application

In this exercise, you create a service group for the loopy application.
sym1
1
From sym1:terminal2, use the tail -f command to monitor the

/var/VRTSvcs/log/engine_A.log VCS engine log.
easily review them.
Solution
End of Solution
From sym1:terminal1, navigate to the /etc/VRTSvcs/conf/config

directory, use the haconf command to open the cluster configuration for
update and use the hagrp command to create a service group named appsg.
Solution
175
haconf -makerw
hagrp -add appsg

VCS NOTICE V-16-1-10136 Group added; populating SystemList and setting the
Parallel attribute recommended before adding resources
Note: The Parallel service group attribute value defaults to 0, specifying a

failover service group type, and does not have to be explicitly set.
End of Solution

B169
Modify the SystemList to allow the appsg service group to run on sym1 and
sym2 preferentially in that order. Then, modify the AutoStartList to allow
appsg to start preferentially on sym1 and then on sym2.
Solution
hagrp -modify appsg SystemList sym1 0 sym2 1
hagrp -modify appsg AutoStartList sym1 sym2
End of Solution
Confirm the appsg configuration and save the VCS configuration without
closing it. Then, identify the changes recorded to the main.cf file by
comparing it to the main.cf.previous file.
Solution
hagrp -display appsg | more
#Group
appsg
appsg
appsg
appsg
appsg
appsg
appsg
appsg
appsg
appsg
...
appsg
...
appsg
...
176 B170
Attribute
AdministratorGroups
Administrators
Authority
AutoFailOver
AutoRestart
AutoStart
AutoStartIfPartial
AutoStartList
AutoStartPolicy
ClusterFailOverPolicy
System
global
global
global
global
global
global
global
global
global
global
Value
Parallel
global
SystemList
global
sym1
haconf -dump
diff main.cf main.cf.previous
0
1
1
1
1
sym1
Order
Manual
sym2
sym2
End of Solution

Exercise 3: Configuring resources for the loopy application

In this exercise, you configure resources for the loopy application and bring them
online.
sym1
Note: Throughout this exercise, the use of VCS CLI commands using the -wait
switch is optional and simply provides a timing or pacing mechanism
between commands.
From sym1:terminal1, add a resource of type NIC named appnic to the

appsg service group using the hares command and then display the appsg
service group to confirm that your input and the key default resource attributes
have been defaulted. The following resource attributes and values are to be
configured.
Critical: 0
Device: eth0
Note: Continue to monitor sym1:terminal2 which has been set up in the
preceding exercise to display the VCS engine log.
Solution
177
hares -add appnic NIC appsg

VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before
agent monitors
hares -modify appnic Critical 0
hares -modify appnic Device eth0
hares -display appnic | more

#Resource
appnic
appnic
appnic
appnic
appnic
Attribute
Group
Type
AutoStart
Critical
Enabled
System
global
global
global
global
global
Value
appsg
NIC
1
0
0

B171
...
appnic
...
Device
global
eth0
End of Solution
Enable the appnic resource by setting the Enabled attribute to 1. Use hares
commands to confirm that it is enabled by displaying the Enabled attribute
and by displaying the state of the appnic resource. Then, display the state of
the appsg service group and save the VCS configuration without closing it and
identify the changes recorded to the main.cf file by comparing it to the
main.cf.previous file.
Solution
hares -modify appnic Enabled 1
hares -value appnic Enabled

1
hares -state appnic

#Resource
appnic
appnic
Attribute
State
State
System
sym1
sym2
Value
ONLINE
ONLINE
Note: It is not necessary to bring a resource of type NIC online. Notice

from the display of its state that it is online on both cluster systems.
d
hagrp -state appsg
#Group
appsg
appsg
178 B172
Attribute
State
State
System
sym1
sym2
Value
|OFFLINE|
|OFFLINE|
Note: The service group state is offline even though the NIC resource
state is online. VCS does not consider persistent resources when
determining the state of a service group.
e
haconf -dump
End of Solution

Use the hares command to add a resource of type IP named appip to the
appsg service group. Then, display the appip resource to confirm that your
input and the key default resource attributes have been defaulted. The
following resource attributes and values are to be configured.
Critical: 0
Device: eth0
Address: 10.10.2.61
NetMask: 255.255.255.0
Solution
hares -add appip IP appsg

agent monitors
hares -modify appip Critical 0
hares -modify appip Device eth0
hares -modify appip Address 10.10.2.61
hares -modify appip NetMask 255.255.255.0
hares -display appip | more

#Resource
appip
appip
appip
appip
appip
...
appip
...
appip
...
appip
...
Attribute
Group
Type
AutoStart
Critical
Enabled
System
global
global
global
global
global
Value
appsg
IP
1
0
0
Address
global
10.10.2.61
Device
global
eth0
NetMask
global
255.255.255.0
End of Solution
179
Use hares commands to enable the appip resource by setting the Enabled
attribute to 1 and to confirm that it is enabled by displaying the attribute.
Solution
hares -modify appip Enabled 1
hares -value appip Enabled

1
End of Solution

B173
Use the hares command to bring the appip resource online on sym1 and
wait for it to come online. Then, use the hares command to confirm that the
appip resource is online by displaying its state. Finally, from the operating
system, use the ip addr command to confirm that appip is online.
Solution
hares -online appip -sys sym1
hares -wait appip State ONLINE -sys sym1
hares -state appip

#Resource
appip
appip
Attribute
State
State
System
sym1
sym2
Value
ONLINE
OFFLINE
ip addr show eth0

1000
End of Solution
Display the state of the appsg service group and save the VCS configuration
without closing it and identify the changes recorded to the main.cf file by
comparing it to the main.cf.previous file.
Solution
180 B174
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
Note: Now that there is at least one properly configured non persistent
resource, VCS can determine the state of the service group.
b
haconf -dump

End of Solution
Use the hares command to add a resource of type DiskGroup named appdg
to the appsg service group. Then, display the appdg resource to confirm that
your input and the key default resource attributes have been defaulted. The
Critical: 0
DiskGroup: loopydatadg
Solution
hares -add appdg DiskGroup appsg

agent monitors
hares -modify appdg Critical 0
hares -modify appdg DiskGroup loopydatadg
hares -display appdg | more

#Resource
appdg
appdg
appdg
appdg
appdg
...
appdg
...
appdg
appdg
...
Attribute
Group
Type
AutoStart
Critical
Enabled
System
global
global
global
global
global
Value
appsg
DiskGroup
1
0
0
DiskGroup
global
loopydatadg
StartVolumes
StopVolumes
global
global
1
1
End of Solution
181
Use hares commands to enable the appdg resource by setting the Enabled
Solution
hares -modify appdg Enabled 1
hares -value appdg Enabled

1
End of Solution

B175
Use the hares command to bring the appdg resource online on sym1 and
appdg resource is online by displaying its state. Also, use the vxdg list
command to confirm the disk group is online. Finally, save the VCS
configuration without closing it and identify the changes recorded to the
main.cf file by comparing it to the main.cf.previous file.
Solution
hares -online appdg -sys sym1
hares -wait appdg State ONLINE -sys sym1
hares -state appdg

#Resource
appdg
appdg
Attribute
State
State
System
sym1
sym2
Value
ONLINE
OFFLINE
vxdg list
NAME
loopydatadg
STATE
enabled,cds
ID
1319645563.37.sym1
haconf -dump
End of Solution
10 Use the hares command to add a resource of type Volume named appvol to
the appsg service group. Then, display the appdg resource to confirm that
182 B176
Critical: 0
Volume: loopydatavol
DiskGroup: loopydatadg
Solution
hares -add appvol Volume appsg

agent monitors
hares -modify appvol Critical 0

hares -modify appvol Volume loopydatavol
hares -modify appvol DiskGroup loopydatadg
hares -display appvol | more

Resource
appvol
appvol
appvol
appvol
appvol
...
appvol
...
appvol
...
Attribute
Group
Type
AutoStart
Critical
Enabled
System
global
global
global
global
global
Value
appsg
Volume
1
0
0
DiskGroup
global
loopydatadg
Volume
global
loopydatavol
End of Solution
11 Use hares commands to enable the appvol resource by setting the Enabled

Solution
hares -modify appvol Enabled 1
hares -value appvol Enabled

1
End of Solution
12 Use the hares command to confirm that the appvol resource is online by
183
displaying its state. Then, from the operating system, use the
vxinfo -g loopydatadg -p command to confirm that the appvol
resource is online. Finally, save the VCS configuration without closing it and
identify the changes recorded to the main.cf file by comparing it to the
Solution
hares -state appvol

#Resource
appvol
appvol
Attribute
State
State
System
sym1
sym2
Value
ONLINE
OFFLINE
vxinfo -g loopydatadg -p
vol loopydatavol
fsgen
plex loopydatavol-01 ACTIVE
Started

B177
haconf -dump
End of Solution
13 Use the hares command to add a resource of type Mount named appmnt to
the appsg service group. Then, display the appmnt resource to confirm that
Critical: 0
BlockDevice: /dev/vx/dsk/loopydatadg/loopydatavol
FsckOpt (escape the dash using the % character): y
FSType: vxfs
MountPoint: /loopyfs
Solution
hares -add appmnt Mount appsg
agent monitors
184 B178
hares -modify appmnt Critical 0
hares -modify appmnt BlockDevice

hares -modify appmnt FsckOpt %-y
hares -modify appmnt FSType vxfs
hares -modify appmnt MountPoint /loopyfs
hares -display appmnt | more

#Resource
appmnt
appmnt
appmnt
appmnt
appmnt
...
appmnt
loopydatavol
...
appmnt
appmnt
...
appmnt
...
Attribute
Group
Type
AutoStart
Critical
Enabled
System
global
global
global
global
global
Value
appsg
Mount
1
0
0
BlockDevice
global
/dev/vx/dsk/loopydatadg/
FSType
FsckOpt
global
global
vxfs
-y
MountPoint
global
/loopyfs
End of Solution

14 Use hares commands to enable the appmnt resource by setting the Enabled

Solution
hares -modify appmnt Enabled 1
hares -value appmnt Enabled

1
End of Solution
15 Use the hares command to bring the appmnt resource online on sym1 and
appmnt resource is online by displaying its state.
Solution
hares -online appmnt -sys sym1
hares -wait appmnt State ONLINE -sys sym1
hares -state appmnt

#Resource
appmnt
appmnt
Attribute
State
State
System
sym1
sym2
Value
ONLINE
OFFLINE
End of Solution
16 From the operating system, use the df -h /loopyfs command to confirm
that the appmnt resource is online. Then, save the VCS configuration without
closing it and identify the changes recorded to the main.cf file by comparing
it to the main.cf.previous file.
185
Solution
df -h /loopyfs
Filesystem
Size Used Avail Use% Mounted on
1.0G
19M 943M
2% /loopyfs
haconf -dump

B179
End of Solution
17 Use the hares command to add a resource of type Process named appproc to
the appsg service group, Then, display the appproc resource to confirm that
Critical: 0
PathName: /bin/ksh
Arguments (double quotes are required): "/loppyfs/loopy appsg"
Solution
hares -add appproc Process appsg
agent monitors
186 B180
hares -modify appproc Critical 0
hares -modify appproc PathName /bin/ksh
hares -modify appproc Arguments

"/loopyfs/loopy appsg"
hares -display appproc | more

#Resource
appproc
appproc
appproc
appproc
appproc
...
appproc
...
appproc
...
Attribute
Group
Type
AutoStart
Critical
Enabled
System
global
global
global
global
global
Value
appsg
Process
1
1
0
Arguments
global
/loopyfs/loopy appsg
PathName
global
/bin/ksh
End of Solution
18 Use hares commands to enable the appproc resource by setting the Enabled

Solution
hares -modify appproc Enabled 1

hares -value appproc Enabled

1
End of Solution
19 Use the hares command to bring the appproc resource online on sym1 and
wait for it to come online. Then, use the hares command to confirm appproc
is online by displaying its state.
Solution
hares -online appproc -sys sym1
hares -wait appproc State ONLINE -sys sym1
hares -state appproc

#Resource
appproc
appproc
Attribute
State
State
System
sym1
sym2
Value
ONLINE
OFFLINE
End of Solution
20 From the operating system, use the ps -ef | grep loopy and
tail -f /loopyfs/loopyout commands to confirm that appproc is

online. Then, save the VCS configuration without closing it and identify the
changes recorded to the main.cf file by comparing it to the
Solution
ps -ef | grep loopy
root
root
187
1 0 13:42 ?
8987 0 13:42 pts/2
00:00:00 /bin/ksh /loopyfs/loopy appsg

00:00:00 grep loopy
tail -f /loopyfs/loopyout
Fri
Fri
Fri
Fri
Fri
Fri
Fri
Fri
Fri
Fri
Fri
4361
4908
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
4
4
4
4
4
4
4
4
4
4
4
13:42:25
13:42:29
13:42:33
13:42:37
13:42:41
13:42:45
13:42:49
13:42:53
13:42:57
13:43:01
13:43:05
PDT
PDT
PDT
PDT
PDT
PDT
PDT
PDT
PDT
PDT
PDT
2011
2011
2011
2011
2011
2011
2011
2011
2011
2011
2011
appsg
appsg
appsg
appsg
appsg
appsg
appsg
appsg
appsg
appsg
appsg
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
is
is
is
is
is
is
is
is
is
is
is
alive
still
alive
still
alive
still
alive
still
alive
still
alive
alive
alive
alive
alive
alive
After a few updates have occurred, press Ctrl-C.

B181
haconf -dump
End of Solution
21 Use the hares command to link the following resources.
appvol requires appdg

appmnt requires appvol
appip requires appnic
appproc requires appmnt
appproc requires appip
Solution
hares -link appvol appdg
hares -link appmnt appvol
hares -link appip appnic
hares -link appproc appmnt
hares -link appproc appip
End of Solution
22 Use the hares command to verify the resources are linked. Confirm the status
of the appsg service group. Then, save the VCS configuration without closing
it and identify the changes recorded to the main.cf file by comparing it to
the main.cf.previous file.
188 B182
Solution
hares -dep
#Group
ClusterService
appsg
appsg
appsg
appsg
appsg
Parent
webip
appip
appmnt
appproc
appproc
appvol
Child
csgnic
appnic
appvol
appip
appmnt
appdg

hagrp -state appsg

#Group Attribute System Value
appsg State sym1 |ONLINE|
appsg State sym2 |OFFLINE|
haconf -dump
End of Solution
189

B183
Exercise 4: Performing a virtual fire drill on the service group

In this exercise, you perform a virtual fire drill on the appsg service group for the
sym2 system.
sym1
1
From sym1:terminal1, determine on which system the appsg service group is

offline.
System where the appsg service group is offline:
sym2
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
190 B184
Create an error condition by removing the /loopyfs mount point on the

system where the appsg service group is offline.
Solution
ssh sym2 rm -rf /loopyfs
ssh sym2 ls -ld /loopyfs

ls: /loopyfs: No such file or directory
End of Solution
Use the havfd command to run a virtual fire drill on the appsg service group
for the sym2 system in verbose mode. Which resources report a failure?
Note: Note: Using the command:

havfd appsg -sys sym2 -v | tee
/var/tmp/havfd.appsg.sym2
will save the output in the file /var/tmp/havfd.appsg.sym2 so
that you can review it later if needed.
Resources that report a failure during the virtual fire drill:
appmnt and appproc
Solution

...
===============================================
Resource appmnt
-----------------------------------------------appmnt: mountpoint.vfd: *** Failure ***
appmnt: mountpoint.vfd: *** Output: [Mountpoint /loopyfs does not exist]
***
appmnt: mounted.vfd: Success
appmnt: mounted.vfd: Output: [Mountpoint /loopyfs is not mounted]
appmnt: vxfslic.vfd: Success
appmnt: vxfslic.vfd: Output: [VxFS is licensed]
appmnt: mountentry.vfd: Success
appmnt: mountentry.vfd: Output: [No entry of Mountpoint /loopyfs in /etc/
fstab]
================================================
Resource appproc
-----------------------------------------------appproc: program.vfd: *** Failure ***
appproc: program.vfd: *** Output: [Program (/loopyfs/loopy) does not exist
or is not executable] ***
191
Note: The appproc error is a noisy error since it is looking for a file
that is only available when the file system is mounted. Even if /
loopyfs existed, the file would not be found since the file system
is mounted on sym1 as part of the appsg service group.
End of Solution
Correct the error condition by creating the /loopyfs mount point on the
system where the appsg service group is offline.
Solution
ssh sym2 mkdir /loopyfs

B185
ssh sym2 ls -ld /loopyfs

drwxr-xr-x 2 root root 4096 Jan 19 14:34 /loopyfs
End of Solution
Use the havfd command to re-run the virtual fire drill on the appsg service
group for the sym2 system in verbose mode. Which resources report a failure?
Resources that report a failure during the virtual fire drill:
appproc only
Solution

...
================================================
Resource appmnt
-----------------------------------------------appmnt: mountpoint.vfd: Success
appmnt: mountpoint.vfd: Output: [Mountpoint /loopyfs exists]
appmnt: mounted.vfd: Success
appmnt: mounted.vfd: Output: [Mountpoint /loopyfs is not mounted]
appmnt: vxfslic.vfd: Success
appmnt: vxfslic.vfd: Output: [VxFS is licensed]
appmnt: mountentry.vfd: Success
appmnt: mountentry.vfd: Output: [No entry of Mountpoint /loopyfs in /etc/
fstab]
================================================
Resource appproc
-----------------------------------------------appproc: program.vfd: *** Failure ***
appproc: program.vfd: *** Output: [Program (/loopyfs/loopy) does not exist
or is not executable] ***
End of Solution
192 B186

Exercise 5: Testing the service group

In this exercise, you switch the appsg service group from sym1 to sym2 and then
back again.
sym1
1
From sym1:terminal1, determine on which system the appsg service group is

offline.
System where the appsg service group is offline:
sym2
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
193
Use the hagrp command to switch the appsg service group to the system
where it is currently offline. Wait for the switch to complete and then check
whether the switch was made by displaying the state of the appsg service
group.
Solution
hagrp -switch appsg -to sym2
hagrp -wait appsg State ONLINE -sys sym2
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|OFFLINE|
|ONLINE|
End of Solution

B187
Use the hagrp command with the -any command option to switch the appsg
service group to the system where it is currently offline. Wait for the switch to
complete and then check whether the switch was made by displaying the state
of the appsg service group.
Solution
hagrp -switch appsg -any
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
194 B188

Exercise 6: Setting resources to critical

In this exercise, you set the critical attribute value to 1 (one) on all resources in the
appsg service group, thus making them critical resources.
sym1
1
From sym1:terminal1, list the resources that are part of the appsg service
group.
Resource in the appsg service group:
appnic, appip, appdg, appvol, appmnt and appproc
Solution
hagrp -resources appsg

appnic
appip
appdg
appvol
appmnt
appproc
End of Solution
195
Confirm that all of the resources in the appsg service group have their Critical
resource attributes set to 0 (zero).
Solution
hares -list Critical=0 Group=appsg

appdg
appdg
appip
appip
appmnt
appmnt
appnic
appnic
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2

B189
appproc
appproc
appvol
appvol
sym1
sym2
sym1
sym2
End of Solution
Set the Critical resource attribute to 1 (one) for each of the resources in the
appsg service group.
Solution
hares -modify appnic Critical 1
hares -modify appip Critical 1
hares -modify appdg Critical 1
hares -modify appvol Critical 1
hares -modify appmnt Critical 1
End of Solution
Confirm that the resource in the appsg service group now have a Critical
resource attribute that is not set to 0 (zero).
Solution
End of Solution
196 B190
Change the BackupInterval cluster attribute to three minutes. Verify the value
of this cluster attribute before and after making the change.
Solution

0
haclus -modify BackupInterval 3


3
End of Solution
Save and close the VCS configuration and identify the changes recorded to the
main.cf file by comparing it to the main.cf.previous file.
Solution
End of Solution
End the monitoring of the engine log.

Solution
End of Solution
197

B191
Exercise 7: (Optional) Examining Veritas File System locking

In this exercise, you examine Veritas file system locking by VCS.
sym1
1
From sym1:terminal1, determine on which system the appmnt resource is

online.
Solution
hares -state appmnt

#Resource
appmnt
appmnt
Attribute
State
State
System
sym1
sym2
Value
ONLINE
OFFLINE
End of Solution
Display the value of the VxFSMountLock resource attribute for the appmnt
resource. Then, from the operating system, confirm there is no lock on the
/loopyfs mount.
Note: The VxFSMountLock attribute should be set to 0.
Solution
198 B192
hares -value appmnt VxFSMountLock

0

/dev/vx/dsk/loopydatadg/loopydatavol on /loopyfs type vxfs
(rw,delaylog,largefiles,ioerror=mwdisable)
End of Solution
Modify the value of the VxFSMountLock resource attribute for the appmnt
resource by setting it to 1 (one) in order to enable vxfs file system locking.
Open, save and close the VCS configuration where necessary and identify the


Solution
haconf -makerw
hares -modify appmnt VxFSMountLock 1

1
End of Solution
Confirm that there is now a mount lock on the /loopyfs file system mount.
Note: The VCS Mount agent will create the /loopyfs file system mount
lock. This may take a few seconds.
Solution

(rw,delaylog,largefiles,ioerror=mwdisable,mntlock=VCS)
End of Solution
199
Take the loopy process offline. Then, freeze the appsg service group. Use the
hastatus -sum command to confirm it is frozen.
Solution
hares -offline appproc -sys sym1
hares -wait appproc State OFFLINE -sys sym1

#Resource
appproc
appproc
Attribute
State
State
System
sym1
sym2
Value
OFFLINE
OFFLINE

B193
hagrp -freeze appsg
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
System
Probed
AutoDisabled
State
B
B
B
B
sym1
sym2
sym1
sym2
Y
Y
Y
Y
N
N
N
N
ONLINE
OFFLINE
PARTIAL
OFFLINE
ClusterService
ClusterService
appsg
appsg
-- GROUPS FROZEN
-- Group
C
appsg
-- RESOURCES DISABLED
-- Group
Type
Resource
H
H
H
H
H
H
appdg
appip
appmnt
appnic
appproc
appvol
appsg
appsg
appsg
appsg
appsg
appsg
DiskGroup
IP
Mount
NIC
Process
Volume
End of Solution
Use the vxumount command to attempt to unmount the /loopyfs file

system. If the command fails, use vxumount command with the -o force
option. What happens?
The /loopy file system cannot be unmounted because it has been mount locked.
200 B194
Solution
vxumount /loopyfs
UX:vxfs vxumount: ERROR: V-3-26388: file system /loopyfs has been mount
locked
vxumount -o force /loopyfs

UX:vxfs vxumount: ERROR: V-3-26388: file system /loopyfs has been mount
locked
End of Solution

Can you unmount the /loopyfs file systems using the vxumount command
with the -o mntunlock=vcs command line option?
Yes, the -o mntunlock=VCS option allows you successfully unmount the

/loopyfs file system.
Solution
vxumount -o mntunlock="VCS" /loopyfs

End of Solution
Mount the /loopyfs file system without a mount lock and confirm that the
file system is mounted.
Note: The device is
Solution
201
/bin/mount -t vxfs /dev/vx/dsk/loopydatadg/

loopydatavol /loopyfs

End of Solution

B195
Unfreeze and online the appsg service group (this will restart the loopy
process). Use the hastatus -sum command to confirm that appsg is no
longer frozen. Is there a mount lock on the /loopyfs file system?
VCS has re-established the mount lock.
Solution
hagrp -unfreeze appsg
hagrp -online appsg -any

VCS NOTICE V-16-1-50735 Attempting to online group on system sym1
hares -wait appmnt State ONLINE -sys sym1
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
System
Probed
AutoDisabled
State
B
B
B
B
sym1
sym2
sym1
sym2
Y
Y
Y
Y
N
N
N
N
ONLINE
OFFLINE
ONLINE
OFFLINE
ClusterService
ClusterService
appsg
appsg

(rw,delaylog,largefiles,ioerror=mwdisable,mntlock=VCS)
End of Solution
202 B196
10 Modify the value of the VxFSMountLock resource attribute for the appmnt
resource by setting it to 0 (zero) in order to disable vxfs file system locking.

Open, save and close the VCS configuration where necessary and identify the
Solution
haconf -makerw


1
hares -modify appmnt VxFSMountLock 0

0
End of Solution
11 After a few seconds, confirm that there is NOT a mount lock on the
/loopyfs files system mount.

Solution

End of Solution
End of lab
203 Lab 8: Online configuration of a service group
B197
204 B198

Lab 9: Offline configuration

In this lab, you update a service group by editing the main.cf file using a system
editor and restart VCS on the west cluster.
Exercise 2: Editing a copy of the main.cf file using a system editor

A copy of the main.cf file is modified using a system editor while the VCS
cluster is operating normally.
Exercise 3: Stopping VCS
The VCS engine is forcibly stopped and the modified main.cf file is copied to
the VCS configuration directory.
Exercise 4: Restarting VCS using the edited main.cf file
The VCS engine is restarted using the edited main.cf file.
205 Lab 9: Offline configuration
B199

1
opened.
mgt
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas

terminal2.
206 B200

Exercise 2: Editing a copy of the main.cf file using a system editor

In this exercise, you modify a copy of the main.cf file using a system editor
while the VCS cluster is operating normally.
sym1
1
From sym1:terminal1, use the haclus command to display the ReadOnly

attribute to confirm that the VCS configuration is closed.
Note: If the value of the ReadOnly attribute is 0 (zero), the VCS
configuration is open for update. Save and close the VCS configuration
and re-confirm it is closed.
Solution
haclus -value ReadOnly

1
End of Solution
Navigate to the /etc/VRTSvcs/conf/config directory and make a

backup copy of the main.cf file. Name the backup file main.cf.orig.
Solution
cp -p main.cf main.cf.orig
End of Solution
B201
Create a subdirectory named test and copy the main.cf file and all the
type definition files into this subdirectory.
Solution
mkdir test
cp -p *.cf test
ls -l test
total 32
-rw-------rw-------rw-------rw-------rw-------rw-------
1
1
1
1
1
1
root
root
root
root
root
root
root 1069
root 1680
root
632
root 1529
root 1360
root 11913
Nov
Nov
Nov
Nov
Nov
Nov
1
7
1
1
1
1
08:23
06:17
08:23
08:23
08:23
08:23
Db2udbTypes.cf
main.cf
OracleASMTypes.cf
OracleTypes.cf
SybaseTypes.cf
types.cf
End of Solution
Navigate to the test directory, open the copied main.cf file in the test
directory in an editor such as vi and add the line Frozen = 1 to the appsg
service group definition only.
Note: The Frozen = 1 line will persistently freeze the service group when
VCS is restarted.
After you have made the edits the affected portion of the file (added portions
are bolded) should be:
208 B202
...
group appsg (
Frozen = 1
)
...
Solution
cd test
vi main.cf

Locate the portion of the file shown in the above code snippet and make the
changes shown in bold.
End of Solution
Continue to edit the main.cf file by adding a resource of type FileOnOff

named appfoo to the appsg service group. The following resource attributes
and values are to be configured:
Critical: 0 (zero) or not critical
PathName (quotes are required): "/loopyfs/appfoo"
After you have made the edits the affected portion of the files (added portions
are bolded) should be:
...
group appsg (
Frozen = 1
)
FileOnOff appfoo (
PathName = "/loopyfs/appfoo"
Critical = 0
)
...
Solution
changes shown in bold.
End of Solution
Continue to edit the main.cf file by modifying the resource dependency

section of the appsg service group to:
remove the dependency between appproc and appmnt
add a dependency between appfoo (parent) and appmnt (child)
add a dependency between appproc (parent) and appfoo (child)
B203
After you have made the edits, the affected portion of the files (added portions
are bolded, deleted portions are strike-through) should be:
Note: The resource dependencies may be listed in a different order.
...
appip requires appnic
appproc requires appip
appvol requires appdg
appmnt requires appvol
appfoo requires appmnt
appproc requires appfoo
...
Solution
changes shown in bold and strike-through.
End of Solution
Save and close the main.cf file.

End of Solution
From the vi editor in command mode, type: :wq

End of Solution
210 B204
Use the hacf command to verify the cluster configuration in the test
directory.
Note: If any errors or warnings are reported, edit the main.cf file and
correct the code that is causing the error or warning.
Solution
hacf -verify .
End of Solution

Exercise 3: Stopping VCS

In this exercise, you forcibly stop the VCS engine and copy the modified main.cf
file to the VCS configuration directory.
sym1
1
From sym1:terminal1, use the hastop command to stop VCS on all

systems, but leave the applications running. Use the gabconfig command to
confirm that VCS is stopped.
Solution
hastop -all -force
gabconfig -a
===============================================================
Port a gen
8e8801 membership 01
End of Solution
Use the ps and tail commands to verify that the loopy application is still
running.
Solution
211
ps -ef | grep /loopyfs/loopy

root
root
10114
1 0 10:18 ?
16445 10826 0 10:32 pts/2

00:00:00 grep loopy
tail -f /loopyfs/loopyout
Mon
Mon
Mon
Mon
Mon
Mon
Mon
Mon
Mon
Mon
Mon
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
Nov
7
7
7
7
7
7
7
7
7
7
7
10:31:38
10:31:42
10:31:46
10:31:50
10:31:54
10:31:58
10:32:02
10:32:06
10:32:10
10:32:14
10:32:18
PST
PST
PST
PST
PST
PST
PST
PST
PST
PST
PST
2011
2011
2011
2011
2011
2011
2011
2011
2011
2011
2011
appsg
appsg
appsg
appsg
appsg
appsg
appsg
appsg
appsg
appsg
appsg
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
Loopy
is
is
is
is
is
is
is
is
is
is
is
alive
still
alive
still
alive
still
alive
still
alive
still
alive
alive
alive
alive
alive
alive
B205
After a few lines are printed, press: Ctrl-C
End of Solution
Copy the edited main.cf file into the VCS configuration directory.
Solution
cp -p main.cf ../main.cf
End of Solution
Navigate to the VCS configuration directory and use the diff command to
compare changes in the new main.cf file to the backup main.cf.orig
file.
Solution
cd ..
pwd
/etc/VRTSvcs/conf/config
212 B206
diff main.cf main.cf.orig

52d51
<
Frozen = 1
56,60d54
< FileOnOff appfoo (
<
PathName = "/loopyfs/appfoo"
<
Critical = 0
<
)
<
95,96c89
<
appfoo requires appmnt
<
appproc requires appfoo
-->
Note: There may be minor differences in the output when compared to

this example output, especially in the dependency section.
End of Solution

Exercise 4: Restarting VCS using the edited main.cf file

In this exercise, you restart the VCS engine using the edited main.cf file.
sym1
1
From sym1:terminal1, start VCS on sym1 which is where the edited

main.cf file is located. Quickly run the hastatus command to observe the
build process and to observe the LOCAL_BUILD and RUNNING system
states on the local system. Terminate the hastatus command when the
ClusterService service group displays as ONLINE.
Note: The system where the edits were performed must be successfully
started first so that the new main.cf file can be propagated to the next
nodes to startup.
Solution
hastart
hastatus
attempting to connect....
attempting to connect....not available; will retry
attempting to connect....retrying
attempting to connect....connected
213
group
resource
system
message
--------------- -------------------- -------------------- ----------------sym1
LOCAL BUILD
sym1
RUNNING
appfoo
sym1
OFFLINE
appvol
sym1
ONLINE
...
------------------------------------------------------------------------appsg
sym1
PARTIAL
webip
sym1
ONLINE
ClusterService
sym1
ONLINE
...
B207
When the ClusterService service group displays a message of ONLINE,

press Ctrl-C.
End of Solution
Display the summary status of the cluster. Why have the resources not been
probed on the non-local system? What effect is seen from the Frozen = 1
line in the main.cf file?
The resources have not been probed on sym2 because HAD is not running there.
The Frozen = 1 line freezes the service groups persistently and their resources are
disabled.
Solution
hastatus -sum | more

-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
UNKNOWN
0
0
sym1
sym2
-- GROUP STATE
-- Group
B
B
B
B
System
ClusterService sym1
ClusterService sym2
appsg
sym1
appsg
sym2
Probed
Y
Y
Y
Y
AutoDisabled
State
N
N
N
Y
ONLINE
OFFLINE
PARTIAL
OFFLINE
-- GROUPS FROZEN
-- Group
214 B208
appsg
-- RESOURCES NOT PROBED

-- Group
Type
Resource
System
E
E
E
E
E
E
E
E
E
webip
csgnic
appdg
appfoo
appip
appmnt
appnic
appproc
appvol
sym2
sym2
sym2
sym2
sym2
sym2
sym2
sym2
sym2
ClusterService
ClusterService
appsg
appsg
appsg
appsg
appsg
appsg
appsg
IP
NIC
DiskGroup
FileOnOff
IP
Mount
NIC
Process
Volume
-- Group
Type
Resource

H
H
H
H
H
H
H
appsg
appsg
appsg
appsg
appsg
appsg
appsg
DiskGroup
FileOnOff
IP
Mount
NIC
Process
Volume
appdg
appfoo
appip
appmnt
appnic
appproc
appvol
End of Solution
Start VCS on sym2. Display a summary status of the cluster. What differences
do you see from the status displayed in step 2?
The resources have now been probed on sym2. The appsg service group is in a partial
online state. The appsg service group remains frozen and its resources are still
disabled.
Solution
Note: Repeat the following hastatus -sum command until sym2 is in a

RUNNING state and all resources on sym2 have been probed.
215
hastatus -sum | more

-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
B
B
B
B
System
ClusterService sym1
ClusterService sym2
appsg
sym1
appsg
sym2
Probed
Y
Y
Y
Y
AutoDisabled
State
N
N
N
N
ONLINE
OFFLINE
PARTIAL
OFFLINE
-- GROUPS FROZEN
-- Group
C
appsg
-- Group
Type
Resource
H
H
H
appdg
appfoo
appip
appsg
appsg
appsg
DiskGroup
FileOnOff
IP
B209
H
H
H
H
appsg
appsg
appsg
appsg
Mount
NIC
Process
Volume
appmnt
appnic
appproc
appvol
End of Solution
Use the hagrp command to verify that the appsg service group is persistently
frozen. Then, use the hares command to verify that the appfoo resource is
offline. Why is appsg in a partial state?
The appfoo resource was created, but the actual file, /loopyfs/appfoo, was
never manually created. Even though the appfoo resource can create the
/loopyfs/appfoo file, since the appsg service group is frozen, VCS will not
perform an online on that resource.
Solution
hagrp -value appsg Frozen

1
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
Value
OFFLINE
OFFLINE
End of Solution
216 B210
Attempt to bring the appsg service group online on sym1. Then, attempt to
bring the appfoo resource in the appsg service group online on sym1. What
happens?
Neither the appsg service group nor any of its non persistent resources can be brought
online while the appsg service group is frozen.
Solution
hagrp -online appsg -sys sym1

VCS WARNING V-16-1-10154 Group appsg is frozen

hares -online appfoo -sys sym1

VCS WARNING V-16-1-10276 Group appsg for resource appfoo is frozen
End of Solution
Use the hares command to confirm that even though the resources in the
appsg service group are disabled, all but the appfoo resource, in fact, monitor
as ONLINE since they remained unaffected by the forced shutdown of VCS.
Solution
hares -display -attribute State -group appsg

#Resource
appdg
appdg
appfoo
appfoo
appip
appip
appmnt
appmnt
appnic
appnic
appproc
appproc
appvol
appvol
csgnic
csgnic
webip
webip
Attribute
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
State
System
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
Value
ONLINE
OFFLINE
OFFLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
ONLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
ONLINE
ONLINE
OFFLINE
End of Solution
217
B211
Manually create the /loopyfs/appfoo file, probe the appfoo resource on

sym1, and verify the appfoo resource state and appsg service group state. Did
the states change? Why or why not?
The appfoo resource state and appsg service group state are now both online. Even
though the appfoo resource and appsg service group cannot be brought explicitly
online by VCS while the appsg service group is frozen, the appfoo resource can be
probed or monitored to discover the presence of the manually created file. All VCS
resource and service group states are still evaluated.
Solution
touch /loopyfs/appfoo
ls -l /loopyfs/appfoo
-rwxr-xr-x 1 root root 311 Nov 11 11:15 /loopyfs/appfoo
hares -probe appfoo -sys sym1
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
Value
ONLINE
OFFLINE
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
End of Solution
218 B212
Open the VCS configuration for update and unfreeze the appsg service group.
Then save and the close the VCS configuration.
Solution
haconf -makerw
hagrp -unfreeze appsg -persistent

hagrp -value appsg Frozen

0
End of Solution
Display a summary status of the cluster. What are the differences from the
previous summary status?
The appsg service group is no longer frozen and the resources are no longer disabled.
Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
B
B
B
B
System
ClusterService sym1
ClusterService sym2
appsg
sym1
appsg
sym2
Probed
Y
Y
Y
Y
AutoDisabled
State
N
N
N
N
ONLINE
OFFLINE
ONLINE
OFFLINE
End of Solution
End of lab
219
B213
220 B214

Lab 10: Configuring notification

In this lab, you use Veritas Operations Manager (VOM) to configure the notifier on
the west cluster and enable triggers.
Exercise 2: Preparing the email notification reader
The mailx email reader for notifications is prepared on mgt.
Exercise 3: Configuring and testing the notifier using VOM

Veritas Operations Manager (VOM) is used to configure notification on the west
cluster made up of sym1 and sym2.
221
Exercise 4: Configuring trigger scripts

Trigger scripts are configured.

B215

222 B216
opened.
mgt
Account: student
Password: veritas
vom
Account: root
Password: veritas
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas
terminal2.

power on vom and log into mgt and vom.
223 Lab 10: Configuring notification
B217
Exercise 2: Preparing the email notification reader

In this exercise, you prepare the mailx email reader for notifications on mgt.
mgt
1
From mgt:terminal1, prep the mailx reader as the student user by deleting all
messages.
Note: The notifier was pre-configured on the east cluster made up of sym3
and sym4. There may be a few notifier messages for that cluster.
Solution
mailx
Mail version 8.1 6/6/93. Type ? for help.

"/var/spool/mail/student": 10 messages 10 new
>N 1 Notifier@mgt.example Wed Oct 26 15:15
N 2 Notifier@mgt.example Wed Oct 26 15:15
224 B218
d 1-$
19/755
19/753
19/789
19/767
19/779
19/735
19/733
19/753
19/712
19/711
"VCS
"VCS
"VCS
"VCS
"VCS
"VCS
"VCS
"VCS
"VCS
"VCS
Information
Information
Information
Information
Information
Information
Information
Information
Information
Information
...
...
...
...
...
...
...
...
...
...
End of Solution

Exercise 3: Configuring and testing the notifier using VOM

In this exercise, you use the Veritas Operations Manager (VOM) to configure
notification on the west cluster which is made up of sym1 and sym2.
vom
1
From vom, open the Firefox browser, navigate to the URL,

https://vom.example.com:14161 and log into the Veritas Operations
Manager using the following account.
Username: root
Password: veritas
Domain: vom.example.com(unixpwd)
Note: Use the FQDN name vom.example.com in the URL to avoid a security
certificate warning. There may be cached URLs in the browser history.
Alternately, select the one that says Login.
Solution
From the lower left corner of desktop, click the Web Browser icon.
From the browser window, in the address field, type:

https://vom.example.com:14161
Click Login.
End of Solution
Note: The faults and risk shown on the Home tab in the Faults and Risks
Overview section, and elsewhere within VOM, for the cps, sym3 and
sym4 servers can be ignored as those virtual machines are powered off.
B219
From the Veritas Operations Manager window, navigate to the Clusters

page by selecting Manage > Clusters from the menu bar.
Solution
From the menu bar, select Manage > Clusters.

End of Solution
Display the service groups associated with the west cluster.

Solution
From the right pane, select the west link.
Select the Service Groups tab.
End of Solution
Add a resource to the ClusterService service group using the following

information.
Name: notifier
Resource type: NotifierMngr
SmtpRecipient attribute
Key: student
Value: Information
SmtpServer attribute: mgt.example.com
SmtpServerVrfyOff attribute: 1
Enabled
Link it as a parent to the csgnic resource (child)
226 B220
Note: All resource attribute values will apply to all cluster systems, that is,
they are global in scope.
Solution
Mark the box to the left of the ClusterService entry.
From the Service Groups section, from the Actions drop-down menu,
select Edit > Resources > Add/Modify Resources.

From the Open Configuration window, mark Close the configuration

after the operation is completed.
Click Next.
In the Name field, type: notifier
From the Type drop-down menu, select NotifierMngr.
Click Add.
From the notifier row, click the ellipsis button.
From the Attributes list, select SmtpRecipients.
In the Key field, type: student
In the Value field, type: Information
Click Add.
m From the Attributes list, select SmtpServer.

n
In the Attribute Value field, type: mgt.example.com
From the Attributes list, select SmtpServerVrfyOff.
In the Attribute Value field, override the 0 (zero) and type: 1
Note: The SmtpServerVrfyOff attribute is not required, but is site

dependant and needed for the lab environment.
q
Click Next.
From the Configure Resources window, from the right pane, from the
notifier row, mark Enabled.
Note: Critical will be set later after testing.
Click Next.
From the Select Parent drop-down list, select notifier.
From the Select Child list, select csgnic.
B221
Click Link.
w Click Finish.
x
Observe the results and click OK.
End of Solution
Bring the notifier resource online on the same system as the webip resource.
Note: Notice that the state of the ClusterService service group is partial. The
notifier resource needs to be brought online.
Solution
From the Service Groups list, from the Name column, select the
ClusterService link.
Select the Resources tab.
From the webip row, from the State column, notice system where it is
Online.
Note: The webip resource should be Online on sym1.
228 B222
Mark the box to the left of the notifier entry.
From the Actions drop-down menu, select Online.
From the Online Resource window, from the Select the system you want
to online resource notifier on drop-down menu, select sym1.
Click OK.
End of Solution

Select the Resource Dependency tab and confirm the dependency between the
notifier resource (parent) and the csgnic resource (child).
Solution
Select the Resource Dependency tab.
Notice the dependency between the notifier resource (parent) and the
csgnic resource (child).
End of Solution
mgt
7
From mgt:terminal1, use the mailx reader to review the notifier messages
from the west cluster. Then, delete the messages and exit the mailx reader.
Solution
mailx

>N 1 Notifier@mgt.example Fri Jan 20 14:01
S"
N 2 Notifier@mgt.example Fri Jan 20 14:01
S"
S"
S"
Resou"
21/791
"VCS Information for
21/755
21/769
21/781
21/751
"VCS Warning for
At the & prompt, review each message by typing each message number and
pressing Enter.
d 1-$
End of Solution
B223
vom
8
From the Veritas Operations Manager, confirm that the notifier resource is
set to critical.
Solution
From the Resource dependency view, mouse-over the notifier resource

and notice whether the tool tip shows that the resource is set to Critical.
If the notifier resource is set to critical, skip to step 9.
Right-click the notifier resource and select Mark Critical.
From the Open configuration window, mark Close the configuration

Click Next.
Click Finish.
Mouse-over the notifier resource and notice the tool tip shows the resource
is set to Critical.
End of Solution
230 B224
Change the alert level for the student email recipient from Information to
Warning.
Note: This will limit the number of messages generated.
Solution
Right-click the notifier resource and select View details.
Select the radio button for the SmtpRecipents resource attribute.
From the Important Attributes section, from the Actions drop-down

menu, select Edit Attribute.
From the Open configuration window, mark Close the configuration

Click Next.
From the Edit Attribute window, in the Key field, type: student
In the Value field, type: Warning
Click Add.
Click Finish.
Notice that the SmtpRecipents resource attribute value is updated.
End of Solution
10 Take the notifier resource Offline and then bring it Online.
Note: The resource attributes for the notifier resource are updated and
verified. However, the notifier must be restarted for the configuration
changes to be read into the notifier process.
Solution
231
Select the Resource Dependency tab.
Right-click the notifier resource and select Offline.
From the Offline Resource window, confirm the Select the system you
want to offline resource notifier on drop-down menu is set to sym1.
Note: Make no other changes.
Click OK.
Right-click the notifier resource and select Online.

B225
From the Online Resource window, confirm the Select the system you
want to offline resource notifier on drop-down menu is set to sym1.
Note: Make no other changes.
Click OK.
Mouse-over the notifier resource and notice the tool tip shows the resource
is Online on sym1.
End of Solution
11 Log out of the Veritas Operations Manager and close the browser window.
Solution
From the top right corner of the Veritas Operations Manager console,
from the User (root) drop-down menu, select Logout.
Close the browser window.
End of Solution
232 B226

Exercise 4: Configuring trigger scripts

In this exercise, you configure trigger scripts.
sym1
1
From sym1:terminal1, navigate to the /opt/VRTSvcs/bin directory and

perform a long directory listing on *trigger*.
Note: The file named hatrigger is the trigger executable.
Solution
cd /opt/VRTSvcs/bin
ls -ld *trigger*
-r-xr----drwxr-xr-x
drwxr-xr-x
drwxr-xr-x
1
2
3
2
root
root
root
root
root 12279 Oct 13

root 4096 Nov 1
root 4096 Nov 1
root 4096 Oct 13
17:00
08:20
08:20
17:00
hatrigger
internal_triggers
sample_triggers
triggers
End of Solution
Explore each of the sub directories that were found by performing a long
listing. Optionally, display some of the internal_trigger files.
Note: Files in the internal_triggers directory are reserved for use by
VCS and should not be modified except under the direction of Support.
The exception is the violation trigger which can be modified, for
example, to send email alerts. Files in the sample_triggers
directory can be edited and copied to other trigger locations like the
triggers directory which is normally empty. Triggers in the
internal_triggers and sample_triggers directories are
subject to overwrite during product updates.
B227
Solution
ls -l internal_triggers
total 20
-rwxr-x---rwxr-x---rwxr-x---rwxr-x---
1
1
1
1
root
root
root
root
root
root
root
root
2264
2313
2319
7490
Oct
Oct
Oct
Oct
cpuusage
dump_tunables
globalcounter_not_updated
violation
1 08:20 VRTSvcs
ls -l sample_triggers/VRTSvcs
total 88
-rwxr--r--rwxr--r--rwxr--r--rwxr--r--rwxr--r--rwxr--r--rwxr--r--rwxr-----rwxr-----rwxr--r--rwxr--r--rwxr--r--rwxr--r--rwxr--r--rwxr--r--rwxr--r--rwxr--r--rwxr--r--rwxr--r--
17:00
17:00
17:00
17:00
ls -l sample_triggers
total 4
drwxr-xr-x 2 root root 4096 Nov
13
13
13
13
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
2434
2992
2805
2427
2465
2452
5077
6265
4598
2834
2574
2713
3235
3197
2573
2815
2560
2655
4001
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
13
13
13
13
13
13
13
13
13
13
13
13
13
13
13
13
13
13
13
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
17:00
cpuusage
injeopardy
loadwarning
nofailover
postoffline
postonline
preonline
preonline_ipc
preonline_vvr
resadminwait
resfault
resnotoff
resrestart
resstatechange
sysjoin
sysoffline
sysup
unable_to_restart_agent
unable_to_restart_had
ls -l triggers
total 0
234 B228
more internal_triggers/violation
...
exit;
}
}
# put your code here...
#
#
#
#
#
#
#
#
Here is a sample code to notify a bunch of users.

@recipients=("username\@servername.com");
$msgfile="/tmp/violation";
ècho system = $ARGV[0], group = $ARGV[1] > $msgfile`;
foreach $recipient (@recipients) {
# Must have elm setup to run this.

#
èlm -s violation $recipient < $msgfile`;
# }
# `rm $msgfile`;
#
exit;
End of Solution
Navigate to the VCS log directory and grep for the term hatrigger in the
engine log.
Note: Notice internal triggers are activated by default if conditions result in
their firing. The dump_tunable trigger is an example of this. The
sysjoin and sysup triggers are also activated, but while they do
exists in the sample_triggers/VRTSvcs directory they are not,
by default, copied to the default triggers directory. Trigger activity is
minimally logged which is true of all VCS supplied triggers and is
recommended for custom triggers. A more verbose level of logging can
be enabled. Triggers apply to all cluster systems although, strictly
speaking, trigger configuration does not have to match on all cluster
systems though that is strongly discouraged.
Solution
cd /var/VRTSvcs/log
grep hatrigger engine_A.log | more

2011/11/01 08:23:28 VCS INFO V-16-6-15015 (sym1) hatrigger:/opt/VRTSvcs/
bin/trig
gers/sysup is not a trigger scripts directory or can not be executed
VCS_LOG_SCRIPT_NAME=hatrigger
2011/11/01 08:23:28 VCS INFO V-16-6-15002 (sym1) hatrigger:hatrigger
executed /o
pt/VRTSvcs/bin/internal_triggers/dump_tunables sym1 1
successfully
bin/trig
gers/sysjoin is not a trigger scripts directory or can not be executed
executed /o
pt/VRTSvcs/bin/internal_triggers/dump_tunables sym2 1
successfully
...
End of Solution
B229
Navigate to the and copy the VCS sysjoin and sysup triggers from
sample_triggers to the triggers directory. Verify the mode is 744 (rwxr--r--).
Note: These triggers will not be modified, but will log successful execution to
the VCS engine log. Since they are activated by default in VCS they
will fire if conditions warrant.
Solution
cd /opt/VRTSvcs/bin/triggers
cp -p ../sample_triggers/VRTSvcs/sysjoin .
cp -p ../sample_triggers/VRTSvcs/sysup .
ls -l
-rwxr--r-- 1 root root 2573 Nov
-rwxr--r-- 1 root root 2560 Nov
8 09:37 sysjoin
8 09:37 sysup
End of Solution
Copy the email customized resadminwait, resfault, and

nofailover triggers from the /student/labs/vcs/vcs60/
triggers directory to the current triggers directory. Verify the mode is
744.
Note: These triggers were copied from sample_triggers and
customized to email a message to student@mgt.example.com. These
triggers have not been activated yet and so will not fire.
Solution
236 B230
cp -p /student/labs/vcs/vcs60/triggers/* .
ls -l
total 20
-rwxr--r--rwxr--r--rwxr--r--rwxr--r--rwxr--r--
1
1
1
1
1
root
root
root
root
root
root
root
root
root
root
2406
2778
2529
2573
2560
Nov
Nov
Nov
Nov
Nov
8
8
8
8
8
09:42
09:42
09:42
09:37
09:37
nofailover
resadminwait
resfault
sysjoin
sysup
End of Solution

Examine the last 20 lines of the resfault trigger to determine how the script
was modified to email a message.
Note: The good idea comment is a reminder to configuring triggers to log
more verbosely.
Solution
tail -20 resfault

#
# It is a good idea to log what you're doing.
# VCSAG_LOG_MSG("I", "(resfault) Invoked with arg0=$ARGV[0], arg1=$ARGV[1],
arg2=$ARGV[2]", msgid, $ARGV[0], $ARGV[1], $ARGV[2]);
#
# put your code here...
#
# Here is a sample code to notify a bunch of users.
@recipients=("student\@mgt.example.com");
#
$msgfile="/tmp/resfault";
ècho system = $ARGV[0], resource = $ARGV[1], oldstate = $ARGV[2] >
$msgfile`;
foreach $recipient (@recipients) {
`/bin/mail -s resfault $recipient < $msgfile`;
}
`rm $msgfile`;
#
exit;
End of Solution
Copy the scripts in the triggers directory from sym1 to sym2 and confirm
the files were copied.
Solution
pwd
/opt/VRTSvcs/bin/triggers
scp * sym2:/opt/VRTSvcs/bin/triggers/
nofailover
resadminwait
resfault
sysjoin
sysup
100%
100%
100%
100%
100%
2406
2778
2529
2573
2560
2.4KB/s
2.7KB/s
2.5KB/s
2.5KB/s
2.5KB/s
00:00
00:00
00:00
00:00
00:00
B231
ssh sym2 ls -l /opt/VRTSvcs/bin/triggers

total 20
-rwxr--r--rwxr--r--rwxr--r--rwxr--r--rwxr--r--
1
1
1
1
1
root
root
root
root
root
root
root
root
root
root
2406
2778
2529
2573
2560
Nov
Nov
Nov
Nov
Nov
8
8
8
8
8
09:51
09:51
09:51
09:51
09:51
nofailover
resadminwait
resfault
sysjoin
sysup
End of Solution
Open the VCS configuration for update and display the TriggersEnabled
service group attribute value for the appsg service group, and the appfoo
resource.
Note: Notice that TriggeresEnabled is not global by default and must
remain that way.
Solution
haconf -makerw
hagrp -display appsg -attribute TriggersEnabled

#Group
appsg
appsg
Attribute
TriggersEnabled
TriggersEnabled
System
sym1
sym2
Value
hares -display appfoo -attribute TriggersEnabled

#Resource
appfoo
appfoo
Attribute
TriggersEnabled
TriggersEnabled
System
sym1
sym2
Value
End of Solution
238 B232
Change the value of the TriggersEnabled service group attribute for the appsg
service group to the value of NOFAILOVER RESFAULT for both sym1 and
sym2 and confirm the change was made.
Note: The resource level trigger RESADMINWAIT cannot be enabled at the
service group level.
Solution
hagrp -modify appsg TriggersEnabled NOFAILOVER

RESFAULT -sys sym1

hagrp -modify appsg TriggersEnabled NOFAILOVER

RESFAULT -sys sym2
hagrp -display appsg -attribute TriggersEnabled

#Group
appsg
appsg
Attribute
TriggersEnabled
TriggersEnabled
System
sym1
sym2
Value
NOFAILOVER
NOFAILOVER
RESFAULT
RESFAULT
End of Solution
10 Change the value of the TriggersEnabled resource attribute for the appfoo
resource to the value of RESADMINWAIT for both sym1 and sym2 and
confirm the change was made. Then save and close the VCS configuration.
Note: Most resource triggers, including the resadminwait and resfault
triggers can be enabled at the service group level for all resources in the
service group. One exception is the resadminwait trigger which can
only be enabled for each resource.
Solution
hares -modify appfoo TriggersEnabled RESADMINWAIT

-sys sym1
hares -modify appfoo TriggersEnabled RESADMINWAIT

-sys sym2
hares -display appfoo -attribute TriggersEnabled

#Resource
appfoo
appfoo
Attribute
TriggersEnabled
TriggersEnabled
System
sym1
sym2
Value
RESADMINWAIT
RESADMINWAIT
End of Solution
11 From sym1:terminal2, use the tail -f command to monitor the VCS
engine log for successful trigger executions.

Solution
End of Solution
B233
12 From sym1:terminal1, shutdown VCS normally on sym2 first and then on
sym1. From sym1:terminal2, confirm VCS is shutdown by observing the

entries in the VCS engine log.
Note: Look for and EXITED message related to the respective cluster node
name.
Solution
From sym1:terminal1, type: hastop -sys sym2
From sym1:terminal2, observe the log entries.
2011/11/08 10:42:53 VCS INFO V-16-1-50135 User root fired command:

MSG_CLUSTER_STOP_SYS sym2 from localhost
state from RUNNING to LEAVING
state from LEAVING to EXITING
state from EXITING to EXITED
2011/11/08 10:42:59 VCS NOTICE V-16-1-10112 System (sym1) - Membership:
0x1, DDNA: 0x2
2011/11/08 10:42:59 VCS ERROR V-16-1-10113 System sym2 (Node '1') is in DDNA
Membership - Membership: 0x1, Visible: 0x0
2011/11/08 10:42:59 VCS NOTICE V-16-1-10449 Group appsg autodisabled on
node sym2 until it is probed
2011/11/08 10:42:59 VCS NOTICE V-16-1-10449 Group VCShmg autodisabled on
bin/triggers/sysoffline is not a trigger scripts directory or can not be
executed
240 B234
From sym1:terminal1, type: hastop -sys sym1

2011/11/08 10:48:36 VCS INFO V-16-1-10305 Resource appvol (Owner:
Unspecified, Group: appsg) is offline on sym1 (VCS initiated)
2011/11/08 10:48:36 VCS NOTICE V-16-1-10300 Initiating Offline of Resource
appdg (Owner: Unspecified, Group: appsg) on System sym1
2011/11/08 10:48:38 VCS INFO V-16-1-10305 Resource appdg (Owner:
2011/11/08 10:48:38 VCS NOTICE V-16-1-10446 Group appsg is offline on system
sym1
2011/11/08 10:48:38 VCS NOTICE V-16-1-10010 Stopping all agents
state from LEAVING to EXITING
state from EXITING to EXITED
2011/11/08 10:48:39 VCS NOTICE V-16-1-11059 GAB registration monitoring
action set to log system message
2011/11/08 10:48:39 VCS NOTICE V-16-1-11057 GAB registration monitoring
timeout set to 0 ms
End of Solution

13 From sym1:termina1, restart VCS on sym1 first and then sym2. From
sym1:terminal2, monitor the VCS engine log for the sysup and sysjoin
trigger log entries to confirm that they now fire successfully.
Note: It may be difficult to see the sysup for sym1 and the sysjoin for sym2
trigger log entries. Alternatively use the more, grep or vi commands
to display the engine log.
Solution
From sym1:terminal1, type: hastart

...
executed /opt/VRTSvcs/bin/triggers/sysup sym1 RUNNING
successfully
...
From sym1:terminal1, type:


...
executed /opt/VRTSvcs/bin/triggers/sysjoin sym2 RUNNING
successfully
...
End of Solution
14 From sym1:terminal1, induce a fault of the appfoo resource by removing the
/loopyfs/appfoo file on the online system and performing a manual

probe of the appfoo resource on that system. From sym1:terminal2, confirm
the fault of the appfoo resource and the resfault trigger firing by observing the
the engine log.
Solution
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
rm -f /loopyfs/appfoo
Value
ONLINE
OFFLINE
B235
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
Value
FAULTED
ONLINE

...
executed /opt/VRTSvcs/bin/triggers/resfault sym1 appfoo ONLINE
successfully
...
End of Solution
mgt
15 From mgt:terminal1, verify that email for the resfault was delivered to
student@mgt.example.com using the mailx reader. Optionally, read the

emails. Delete the emails and quit the mail program.
Solution
mailx
>N 1 Notifier@mgt.example Tue Nov 8 11:04
N 2 root@sym1.example.co Tue Nov 8 11:04
N 3 Notifier@mgt.example Tue Nov 8 11:04
&
21/728
19/865
21/734
"VCS Error for ...

"resfault"
"VCS Error for ...
Note: Three messages are delivered. Two by the notifier for the resource
fault and subsequent service group fault and the message from the
resfault trigger.
242 B236
pressing Enter.
d 1-$
End of Solution

sym1
16 From sym:terminal1, clear the appfoo resource fault and switch the appsg
service group back to sym1. Confirm that the fault has been cleared and the
service group has been switched.
Solution
hares -clear appfoo -sys sym1
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
17 From sym1:terminal2, end the monitoring of the engine log.

Solution

End of Solution
End of lab
B237
244B238

Lab 11: Configuring resource fault behavior

In this lab, you observe how VCS responds to faults in a variety of scenarios for
your service group.
Exercise 2: Observing non-critical resource faults

Non-critical resource fault behavior is examined on the west cluster made up of
sym1 and sym2.
Exercise 3: Observing critical resource faults
Critical resource fault behavior is examined on the west cluster made up of sym1
and sym2.
Exercise 4: (Optional) Observing faults in frozen service groups
Resource faults in a frozen service group are examined on the west cluster made
up of sym1 and sym2.
Exercise 5: (Optional) Observe ManageFaults behavior
Service group failover using the ManageFaults service group attribute is
examined on the west cluster made up of sym1 and sym2.
Exercise 6: (Optional) Observing restart limit behavior
Restart limit behavior is examined for a resource on the west cluster made up of
sym1 and sym2.
245 Lab 11: Configuring resource fault behavior
B239

1
opened.
mgt
Account: student
Password: veritas
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas
246 B240

terminal2.
shutdown vom.

Exercise 2: Observing non-critical resource faults

In this exercise, you examine non-critical resource fault behavior on the west
sym1
1
Open a third terminal window, sym1:terminal3, use the hastatus

command to continuously display cluster status.
Note: As cluster commands are executed in this lab, you can monitor their
effects on the cluster status.
Solution
Right-click on an empty portion of the desktop and select Konsole.
hastatus
End of Solution
From sym1:terminal2, use the tail -f command to monitor the VCS

engine log.
easily review them.
Solution
End of Solution
B241
From sym1:terminal1, open the VCS configuration for update and verify that
the Critical resource type attribute value is set to 0 for the appfoo and
appproc resources in the appsg service group. If necessary, set the Critical
attribute to 0. Then save, but do not close the VCS configuration.
Solution
haconf -makerw

appfoo
appfoo
sym1
sym2
Note: The appfoo resource should already be set to non critical.

c

appfoo
appfoo
appproc
appproc
sym1
sym2
sym1
sym2
haconf -dump
End of Solution
248 B242
Change the MonitorInterval resource type attribute value to 10 seconds for

the FileOnOff resource type. Override the OfflineMonitorInterval resource
type attribute value to 30 seconds for the appfoo resource (of type
FileOnOff). Then save, but do not close the VCS configuration.
Note: This change is to decrease the monitor wait times of the appfoo
resource. Modifying the resource type attribute value affects all
resources of that type. Overriding the resource type attribute value for a
specific resource of that type affects only that resource. Using both
techniques for the different but related monitor attribute values may not
be ideal in a production environment and is done in this lab for
demonstrative purposes only.
Solution
hatype -value FileOnOff MonitorInterval

60

hatype -modify FileOnOff MonitorInterval 10

10
hatype -value FileOnOff OfflineMonitorInterval

300
hares -value appfoo OfflineMonitorInterval

VCS WARNING V-16-1-10048 Attribute OfflineMonitorInterval does not exist
hares -override appfoo OfflineMonitorInterval

300
hares -display -ovalues appfoo

#Resource
appfoo
appfoo
appfoo
Attribute
Group
Type
OfflineMonitorInterval
System
global
global
global
Value
appsg
FileOnOff
300
hares -modify appfoo OfflineMonitorInterval 30

30

300
haconf -dump
End of Solution
Verify the appsg service group is online on sym1. If it is not, then bring it
online on sym1 using the appropriate commands.
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
B243
Induce a fault of the appfoo resource on sym1 by removing the

/loopyfs/appfoo file. Then, probe the appfoo resource on sym1 and
determine its state.
Solution
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
rm /loopyfs/appfoo
Value
ONLINE
OFFLINE
ls: /loopyfs/appfoo: No such file or directory
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
Value
FAULTED
OFFLINE
End of Solution
Display the state of the appsg service group and the state of each resource in
the appsg service group. Did the faulting of the appfoo resource cause a
failover of the appsg service group? Why or why not?
The appsg service group does not failover because the resource that faulted on sym1,
appfoo, or a dependent resource that was brought offline due to a faulted resource,
250 B244
appproc, were non critical resources.
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|PARTIAL|
|OFFLINE|


#Resource
appdg
appdg
appfoo
appfoo
appip
appip
appmnt
appmnt
appnic
appnic
appproc
appproc
appvol
appvol
Attribute
State
State
State
State
State
State
State
State
State
State
State
State
State
State
System
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
Value
ONLINE
OFFLINE
FAULTED
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
ONLINE
OFFLINE
OFFLINE
ONLINE
OFFLINE
End of Solution
Note: Generally the phrase resource state is of interest on all cluster nodes for
which a service group containing a resource or resources is configured to
run. However, if it is sure that resource state for a particular cluster node is
desired then the following two commands can be substituted as applicable
throughout the lab:
hares -display -attribute State -group appsg -sys
sym1
hares -display -attribute State -group appsg -sys

sym2
251

B245
mgt
8
From mgt:terminal1, check to see if any email was delivered to

student@mgt.example.com using the mailx reader. What email was
delivered, if any? Read and delete any email and quit the mail program.
One email from the notifier and one from the resfault trigger were generated and
delivered.
Solution
mailx
"/var/spool/mail/student": 2 messages 2 unread
>U 1 Notifier@mgt.example Wed Nov 9 13:03 22/738
U 2 root@sym1.example.co Wed Nov 9 13:03 20/875
&
"VCS Error for ...

"resfault"
pressing Enter.
d 1-$
End of Solution
252 B246
sym1
9
From sym1:terminal1, clear the fault on the appfoo resource and display the
state again.
Solution

hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
Value
OFFLINE
OFFLINE
End of Solution
10 Bring the appfoo and appproc resources online on the system where the
appsg service group is partially online. Wait for them to come online and
confirm that the state of the appsg service group is ONLINE.
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
B247
Exercise 3: Observing critical resource faults

In this exercise, you examine the critical resource fault behavior on the west
sym1
1
From sym1:terminal1, set the appfoo and appproc resources to critical in the
appsg service group. Then, verify that all resources in the appsg service group
are set to critical.
Note: A Critical resource attribute value can only be set to 0 (zero) or 1
(one). No output when testing for Critical=1 means all resources are
set to Critical=0 or vice versa.
Note: Continue to monitor sym1:terminal2 which has been set up in the

preceding exercise to display the VCS engine log and sym1:terminal3
which has been set up in the preceding exercise to display the VCS
status.
Solution
appfoo
appfoo
appproc
appproc
254 B248
sym1
sym2
sym1
sym2
hares -modify appfoo Critical 1
End of Solution

Verify that the appsg service group is online on sym1.

Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution

/loopyfs/appfoo file. Then, probe the appfoo resource on sym1 and
determine its state.
Solution
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
rm /loopyfs/appfoo
System
sym1
sym2
Value
ONLINE
OFFLINE
hares -state appfoo
#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
Value
FAULTED
OFFLINE
End of Solution
B249
Wait for the appsg service group to failover. Then, display the state of the
appsg service group and the state of each resource in the appsg service group.
Did the faulting of the appfoo resource cause a failover of the appsg service
group? Why or why not?
The appsg service group fails over to sym2 because the resource that faulted on
sym1, appfoo, and a dependent resource that was brought offline on sym1 due to a
faulted resource, approc, are critical resources.
Solution
hagrp -state appsg

#Group
appsg
appsg
System
sym1
sym2
Value
|OFFLINE|FAULTED|
|ONLINE|

#Resource
appdg
appdg
appfoo
appfoo
appip
appip
appmnt
appmnt
appnic
appnic
appproc
appproc
appvol
appvol
256 B250
Attribute
State
State
Attribute
State
State
State
State
State
State
State
State
State
State
State
State
State
State
System
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
Value
OFFLINE
ONLINE
FAULTED
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
ONLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
End of Solution

mgt
5

Two emails from the notifier and one from the resfault trigger were generated and
delivered.
Solution
mailx
"/var/spool/mail/student": 3 messages 3 unread
>U 1 Notifier@mgt.example Thu Nov 10 06:47 22/739
U 2 root@sym1.example.co Thu Nov 10 06:47 20/879
U 3 Notifier@mgt.example Thu Nov 10 06:47 22/745
Service"
&
"VCS Error for ...

"resfault"
"VCS Error for
pressing Enter.
d 1-$
End of Solution
sym1
6
From sym1:terminal1, induce a fault of the appfoo resource on sym2 by

removing the /loopyfs/appfoo file. Then, probe the appfoo resource on
sym2 and determine its state.
Solution
ssh sym2 rm -f /loopyfs/appfoo
B251
ssh sym2 ls -l /loopyfs/appfoo

hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
Value
FAULTED
FAULTED
End of Solution
Wait for the appsg service group to be declared faulted on sym2. Then,
display the state of the appsg service group and the state of each resource in
the appsg service group. Did the faulting of the appfoo resource cause a
failover of the appsg service group? Why or why not?
The appsg service group is already faulted on sym1 so there is no other system for it
to try to come online. It becomes faulted on both systems.
Solution
hagrp -wait appsg State "OFFLINE|FAULTED" -sys

sym2
hagrp -state appsg

#Group
appsg
appsg
258 B252
Attribute
State
State
System
sym1
sym2
Value
|OFFLINE|FAULTED|
|OFFLINE|FAULTED|

#Resource
appdg
appdg
appfoo
appfoo
appip
appip
appmnt
appmnt
appnic
appnic
appproc
appproc
appvol
appvol
Attribute
State
State
State
State
State
State
State
State
State
State
State
State
State
State
System
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
Value
OFFLINE
OFFLINE
FAULTED
FAULTED
OFFLINE
OFFLINE
OFFLINE
OFFLINE
ONLINE
ONLINE
OFFLINE
OFFLINE
OFFLINE
OFFLINE
End of Solution

mgt
8

Three emails from the notifier, one from the resfault trigger, and one from the
nofailover trigger were generated and delivered.
Solution
mailx

>N 1 Notifier@mgt.example Thu Nov 10 07:40
Resource"
N 2 root@sym2.example.co Thu Nov 10 07:40
N 3 Notifier@mgt.example Thu Nov 10 07:40
Service "
Service "
&
21/729
"VCS Error for
19/869
21/735
"resfault"
"VCS Error for
22/917
"VCS Error for
19/863
"nofailover"
pressing Enter.
d 1-$
End of Solution
sym1
9
From sym1:terminal1, clear the appfoo resource fault on all systems and
bring the appsg service group online on sym1.
B253
Solution
hares -clear appfoo
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
Value
OFFLINE
OFFLINE
System
sym1
sym2
Value
|OFFLINE|
|OFFLINE|
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
10 Change the MonitorInterval resource type attribute value back to the default
value of 60 seconds for the FileOnOff resource type. Undo the override of the
OfflineMonitorInterval resource type attribute so that the appfoo resource
will be determined by the value of the OfflineMonitorInterval value for the
FileOnOff resource type. Save and close the VCS configuration.
Solution
260 B254

10
hatype -modify FileOnOff MonitorInterval 60

60

30

hares -undo_override appfoo OfflineMonitorInterval

VCS WARNING V-16-1-10048 Attribute OfflineMonitorInterval does not exist

300
End of Solution
11 If you do NOT wish to perform the optional exercises in this lab, terminate the
status and log file displays.

Solution
End of Solution
261

B255
Exercise 4: (Optional) Observing faults in frozen service groups

In this exercise, you examine resource faults in a frozen service group on the west
sym1
Note: The cluster configuration should be closed during this exercise.
From sym1:terminal1, verify that all resources in the appsg service group are
set to critical and that the appsg service group is online on sym1.
which has been set up in a preceding exercise to display the VCS status.
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
262 B256
Temporarily freeze the appsg service group. What does a summary status of
the VCS cluster show?
The appsg is frozen and its resources are disabled.
Solution
hagrp -freeze appsg

hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
B
B
B
B
System
ClusterService sym1
ClusterService sym2
appsg
sym1
appsg
sym2
Probed
Y
Y
Y
Y
AutoDisabled
State
N
N
N
N
OFFLINE
ONLINE
ONLINE
OFFLINE
-- GROUPS FROZEN
-- Group
C
appsg
-- Group
Type
Resource
H
H
H
H
H
H
H
appdg
appfoo
appip
appmnt
appnic
appproc
appvol
appsg
appsg
appsg
appsg
appsg
appsg
appsg
DiskGroup
FileOnOff
IP
Mount
NIC
Process
Volume
End of Solution

/loopyfs/appfoo file. Probe the appfoo resource on sym1 and determine
its state.
Solution
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
rm /loopyfs/appfoo
System
sym1
sym2
Value
ONLINE
OFFLINE
B257
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
Value
FAULTED
OFFLINE
End of Solution
Display the state of the appsg service group. Did the faulting of the appfoo
resource cause a failover of the appsg service group? Why or why not?
The appsg service group does not fail over to sym2 because it is frozen. However,
resource and service groups are still monitored and their states are determined.
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|FAULTED|PARTIAL|
|OFFLINE|
End of Solution
Display the state of each resource in the appsg service group. Were any
resources dependent on the appfoo resource affected?
The dependant, or parent, appproc resource was not taken offline on sym1.
Solution
264 B258

#Resource
appdg
appdg
appfoo
appfoo
appip
appip
appmnt
appmnt
appnic
appnic
appproc
appproc
appvol
Attribute
State
State
State
State
State
State
State
State
State
State
State
State
State
System
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
Value
ONLINE
OFFLINE
FAULTED
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
ONLINE
ONLINE
OFFLINE
ONLINE

appvol
State
sym2
OFFLINE
End of Solution
Verify that the appsg service group is still frozen. Information shown in the
previous two steps is also available or can be inferred from this output.
Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
B ClusterService
B ClusterService
B appsg
PARTIAL|FAULTED
B appsg
System
Probed
AutoDisabled
State
sym1
sym2
sym1
Y
Y
Y
N
N
N
OFFLINE
ONLINE
sym2
OFFLINE
-- GROUPS FROZEN
-- Group
appsg
-- RESOURCES FAILED
-- Group
Type
Resource
System
appfoo
sym1
appsg
FileOnOff
-- Group
Type
Resource
H
H
H
H
H
H
H
appdg
appfoo
appip
appmnt
appnic
appproc
appvol
appsg
appsg
appsg
appsg
appsg
appsg
appsg
DiskGroup
FileOnOff
IP
Mount
NIC
Process
Volume
End of Solution
B259
mgt
7

One email from the notifier and one from the resfault trigger were generated and
delivered.
Solution
mailx
Resource"
&
21/729
"VCS Error for
19/869
"resfault"
pressing Enter.
d 1-$
End of Solution
266 B260
sym1
8
From sym1:terminal1, create an empty /loopyfs/appfoo file and probe

the appfoo resource on sym1.
Note: The probe causes VCS to quickly monitor appfoo rather than wait for
the next OfflineMonitorInterval which can be up to 5 minutes by
default.

Solution
-rw-r--r-- 1 root root 0 Nov 10 08:52 /loopyfs/appfoo
End of Solution
Determine the state of the appsg service group and the resources in the appsg
service group.
Note: Recall that the appfoo resource was faulted on sym1 and the appsg
service group was faulted (and partial) on sym1. The external creation
of the file managed by the appfoo resource followed by a probe of the
appfoo resource on sym1 resulted in VCS clearing the fault and
declaring the appfoo resource online. In this case, since the appfoo
resource was the only faulted or offline resource, the appsg service
group is also declared online on sym1.
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|

#Resource
appdg
appdg
appfoo
appfoo
appip
appip
appmnt
appmnt
appnic
appnic
appproc
appproc
appvol
appvol
Attribute
State
State
State
State
State
State
State
State
State
State
State
State
State
State
System
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
Value
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
ONLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
End of Solution
B261
10 Verify that the appsg service group is still frozen. Information shown in the
previous step is also available or can be inferred from this output.

Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
System
Probed
AutoDisabled
State
B
B
B
B
sym1
sym2
sym1
sym2
Y
Y
Y
Y
N
N
N
N
OFFLINE
ONLINE
ONLINE
OFFLINE
ClusterService
ClusterService
appsg
appsg
-- GROUPS FROZEN
-- Group
C
appsg
-- Group
Type
Resource
H
H
H
H
H
H
H
appdg
appfoo
appip
appmnt
appnic
appproc
appvol
appsg
appsg
appsg
appsg
appsg
appsg
appsg
DiskGroup
FileOnOff
IP
Mount
NIC
Process
Volume
End of Solution
mgt
268 B262
11 From mgt:terminal1, check to see if any email was delivered to

One email from the notifier was generated and delivered.
Solution

mailx
"/var/spool/mail/student": 1 message 1 new
&
21/751
"VCS Warning ...
pressing Enter.
d 1-$
End of Solution
sym1
12 From sym1:terminal1, unfreeze the appsg service group and verify that it is
unfrozen.
Solution
hagrp -unfreeze appsg
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
System
Probed
AutoDisabled
State
B
B
B
B
sym1
sym2
sym1
sym2
Y
Y
Y
Y
N
N
N
N
ONLINE
OFFLINE
ONLINE
OFFLINE
ClusterService
ClusterService
appsg
appsg
End of Solution
B263
13 If you do NOT wish to perform the remaining optional exercises in this lab,
terminate the status and log file displays.

Solution
End of Solution
270 B264

Exercise 5: (Optional) Observe ManageFaults behavior

In this exercise, you examine service group failover using the ManageFaults
service group attribute on the west cluster made up of sym1 and sym2.
sym1
1
all resources in the appsg service group are set to critical and that the appsg
service group is online on sym1.
Solution
haconf -makerw
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
271
Set the ManageFaults service group attribute to the value of NONE for the
appsg service group. Save, but do not close the VCS configuration.
Solution
hagrp -value appsg ManageFaults

ALL
hagrp -modify appsg ManageFaults NONE

NONE

B265
haconf -dump
End of Solution

/loopyfs/appfoo file. Probe the appfoo resource on sym1 and determine
its state.
Solution
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
rm /loopyfs/appfoo
Value
ONLINE
OFFLINE
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
Value
ONLINE|ADMIN WAIT
OFFLINE
End of Solution
272 B266
Display the state of the appsg service group. Does appsg failover? Why or
why not?
The appsg service group does not fail over to sym2 and it remains in an online state
on sym1.
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution

Display the state of each resource in the appsg service group. What happens to
the resources in the appsg service group?
The dependant, or parent, appproc resource was not taken offline on sym1. The
appfoo resource is in an admin wait state on sym1.
Solution

#Resource
appdg
appdg
appfoo
appfoo
appip
appip
appmnt
appmnt
appnic
appnic
appproc
appproc
appvol
appvol
Attribute
State
State
State
State
State
State
State
State
State
State
State
State
State
State
System
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
Value
ONLINE
OFFLINE
ONLINE|ADMIN WAIT
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
ONLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
End of Solution
mgt

One email from the notifier and one from the resadminwait trigger were generated
and delivered.
Solution
mailx
21/751
"VCS Error for
B267
Resource appfoo, Res"

N 2 root@sym1.example.co
&
Thu Nov 10 11:24
19/868
"resadminwait"
pressing Enter.
d 1-$
End of Solution
sym1
7
From sym1:terminal1, create an empty /loopyfs/appfoo file and probe

the appfoo resource on sym1.
Solution
-rw-r--r-- 1 root root 0 Nov 10 08:52 /loopyfs/appfoo
hares -state appfoo
#Resource
appfoo
appfoo
274 B268
Attribute
State
State
System
sym1
sym2
Value
ONLINE
OFFLINE
End of Solution
Determine the state of the appsg service group and the resources in the appsg
service group.
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|


#Resource
appdg
appdg
appfoo
appfoo
appip
appip
appmnt
appmnt
appnic
appnic
appproc
appproc
appvol
appvol
Attribute
State
State
State
State
State
State
State
State
State
State
State
State
State
State
System
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
Value
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
ONLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
End of Solution
Again, induce a fault of the appfoo resource on sym1 by removing the

/loopyfs/appfoo file. Probe the appfoo resource and check its state.
Solution
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
rm /loopyfs/appfoo
System
sym1
sym2
Value
ONLINE
OFFLINE
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
Value
ONLINE|ADMIN WAIT
OFFLINE
End of Solution
B269
mgt

One email from the notifier and one from the resadminwait trigger were generated
and delivered.
Solution
mailx
&
21/751
"VCS Error for
19/868
"resadminwait"
pressing Enter.
d 1-$
End of Solution
276 B270

sym1
11 From sym1:terminal1, use the hares -clearadminwait command to
recover the appfoo resource from the admin wait state. Does the appsg service
group failover to sym2? What happens to the state of the appfoo resource?
The appsg service group is faulted on sym1, fails over, and is online on sym2. The
admin wait state for the appfoo resource is changed to faulted on sym1 and online on
sym2.
Solution
hagrp -clearadminwait -fault appsg -sys sym1
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|OFFLINE|FAULTED|
|ONLINE|
System
sym1
sym2
Value
FAULTED
ONLINE
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
End of Solution
mgt

delivered.
B271
Solution
mailx
Service Group appsg,"
&
21/729
"VCS Error for
19/869
21/735
"resfault"
"VCS Error for
pressing Enter.
d 1-$
End of Solution
sym1
13 From sym1:terminal1, clear the appfoo resource fault and bring the appsg
service group online on sym1 and confirm.
Solution
278 B272
hares -state appfoo

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
Value
OFFLINE
ONLINE
System
sym1
sym2
Value
|OFFLINE|
|ONLINE|
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State

hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
14 Set the ManageFaults service group attribute for the appsg service group
back to the default value of ALL. Save and close the VCS configuration.
Solution

NONE
hagrp -modify appsg ManageFaults ALL

ALL
End of Solution
15 If you do NOT wish to perform the remaining optional exercises in this lab,
terminate the status and log file displays.
Solution
End of Solution
B273
Exercise 6: (Optional) Observing restart limit behavior

In this exercise, you examine restart limit behavior for a resource on the west
Note: Unlike previous exercises, the resource monitoring interval attributes are
not modified in this exercise in order to accelerate any effects because the
resource to be faulted is being monitored by IMF which is covered in detail
in a subsequent lab.
sym1
1
all resources in the appsg service group are set to critical and that the appsg
service group is online on the sym1.
Solution
280 B274
haconf -makerw
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
Override the RestartLimit resource type attribute value for the appproc
resource to 1. Save, but do not close the VCS configuration.
Solution
hatype -value Process RestartLimit

0

hares -value appproc RestartLimit

VCS WARNING V-16-1-10048 Attribute RestartLimit does not exist
hares -override appproc RestartLimit

0
hares -display -ovalues appproc

#Resource
appproc
appproc
appproc
Attribute
Group
Type
RestartLimit
System
global
global
global
Value
appsg
Process
0
hares -modify appproc RestartLimit 1

1

0
haconf -dump
End of Solution
Stop the loopy process outside of VCS by sending a kill signal to its process ID
(PID).
Note: Take note of the PID for the loopy process.
281
Solution
ps -ef | grep loopy

root
root
18245
1 0 12:05 ?
24256 10510 0 13:09 pts/2

00:00:00 grep loopy
kill <loopy PID>
End of Solution

B275
Display the PID of the loopy process. Has it changed? Why or why not?
Note: It may take a while for the loopy process to restart. Note the new PID
for the loopy process.
The loopy process was restarted automatically on the same system and evidenced by
the new PID.
Solution
ps -ef | grep loopy

root
root
24286
1 0 13:09 ?
24290 10510 0 13:09 pts/2

00:00:00 grep loopy
End of Solution
Display the state of the appsg service group. Has it failed over? Display the
state of each resource in the appsg service group. What happens to the
resources in the appsg service group?
No failover occurs. The resource states for all resources in the appsg service group
remain as before.
Solution
282 B276
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|

#Resource
appdg
appdg
appfoo
appfoo
appip
appip
appmnt
appmnt
appnic
appnic
Attribute
State
State
State
State
State
State
State
State
State
State
System
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
Value
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
ONLINE

appproc
appproc
appvol
appvol
State
State
State
State
sym1
sym2
sym1
sym2
ONLINE
OFFLINE
ONLINE
OFFLINE
End of Solution
mgt
6

delivered.
Solution
mailx
No mail for root
End of Solution
sym1
From sym1:terminal3, observe the latest output from the cluster status
regarding appproc.
Solution
...
group
resource
system
message
--------------- -------------------- --------------- -------------------appproc
sym1
|RESTARTING|
End of Solution
From sym1:terminal2, observe the entries in the VCS engine log regarding
appproc.
B277
Solution
...
2011/11/10 13:31:36 VCS ERROR V-16-2-13067 (sym1) Agent is calling clean for
resource(appproc) because the resource became OFFLINE unexpectedly, on its
own.
2011/11/10 13:31:36 VCS INFO V-16-2-13068 (sym1) Resource(appproc) - clean
completed successfully.
2011/11/10 13:31:36 VCS ERROR V-16-2-13073 (sym1) Resource(appproc) became
OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 1)
the resource.
2011/11/10 13:31:41 VCS NOTICE V-16-2-13076 (sym1) Agent has successfully
restarted resource(appproc).
...
End of Solution
Stop the loopy process outside of VCS a second time by sending a kill signal to
its process ID (PID).
Solution
ps -ef | grep loopy

root
root
24286
1 0 13:09 ?
24290 10510 0 13:09 pts/2
kill <loopy PID>
ps -ef | grep loopy

root
25737 10510
0 13:10 pts/2

00:00:00 grep loopy
00:00:00 grep loopy
End of Solution
10 Display the state of the appsg service group. Has it failed over? Display the
state of each resource in the appsg service group. What happens to the
resources in the appsg service group?
284 B278
The appsg service group is faulted on sym1 and fails over to sym2. The appproc
resource now shows faulted on sym1 and online on sym2.
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|OFFLINE|FAULTED|
|ONLINE|


#Resource
appdg
appdg
appfoo
appfoo
appip
appip
appmnt
appmnt
appnic
appnic
appproc
appproc
appvol
appvol
Attribute
State
State
State
State
State
State
State
State
State
State
State
State
State
State
System
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
Value
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
ONLINE
ONLINE
FAULTED
ONLINE
OFFLINE
ONLINE
End of Solution
11 Perform a summary status for the cluster. Information shown in the previous
step is also available or can be inferred from this output.

Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
B ClusterService
B ClusterService
B appsg
OFFLINE|FAULTED
B appsg
System
Probed
AutoDisabled
State
sym1
sym2
sym1
Y
Y
Y
N
N
N
OFFLINE
ONLINE
sym2
ONLINE
-- RESOURCES FAILED
-- Group
Type
Resource
System
appproc
sym1
appsg
Process
End of Solution
B279
mgt

delivered.
Solution
mailx
Resource appproc, Re"
Service Group appsg,"
&
21/729
"VCS Error for
19/870
21/735
"resfault"
"VCS Error for
pressing Enter.
d 1-$
End of Solution
286 B280
sym1
13 From sym1:terminal3, observe the latest output from the cluster status
regarding appsg.

Solution
...
appsg
appproc
sym1
sym1
sym1
PARTIAL *FAULTED*
*FAULTED*
STOPPING PARTIAL
appsg
*FAULTED*
------------------------------------------------------------------------appfoo
sym1
WAITING FOR OFFLINE
appip
sym1
WAITING FOR OFFLINE
appfoo
sym1
OFFLINE
appmnt
sym1
WAITING FOR OFFLINE
appip
sym1
OFFLINE
End of Solution
14 From sym1:terminal2, observe the entries in the VCS engine log regarding
appproc.
Solution
...
2011/11/10 13:36:25 VCS ERROR V-16-2-13067 (sym1) Agent is calling clean for
resource(appproc) because the resource became OFFLINE unexpectedly, on its
own.
2011/11/10 13:36:25 VCS INFO V-16-2-13068 (sym1) Resource(appproc) - clean
completed successfully.
2011/11/10 13:36:26 VCS INFO V-16-1-10307 Resource appproc (Owner:
Unspecified, Group: appsg) is offline on sym1 (Not initiated by VCS)
appfoo (Owner: Unspecified, Group: appsg) on System sym1
appip (Owner: Unspecified, Group: appsg) on System sym1
2011/11/10 13:36:26 VCS INFO V-16-1-10305 Resource appfoo (Owner:
appmnt (Owner: Unspecified, Group: appsg) on System sym1
executed /opt/VRTSvcs/bin/triggers/resfault sym1 appproc ONLINE
successfully
...
End of Solution
15 Clear the appproc resource fault and bring the appsg service group online on
sym1.
Solution
hares -clear appproc -sys sym1

#Resource
appfoo
appfoo
Attribute
State
State
System
sym1
sym2
Value
OFFLINE
ONLINE
B281
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|OFFLINE|
|ONLINE|
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
16 Remove the override of the RestartLimit resource type attribute so that the
appproc resource will be affected by the value of the RestartLimit value for
the Process resource type.
Solution

1
hares -undo_override appproc RestartLimit

VCS WARNING V-16-1-10048 Attribute RestartLimit does not exist
288 B282
End of Solution
17 Save and close the VCS configuration and terminate the status and log file
displays.
Solution
From sym1:terminal1, type: haconf -dump -makero

End of Solution
End of lab
B283
290 B284

Lab 12: IMF and AMF

In this lab, you examine the configuration of The Intelligent Monitoring
Framework (IMF) including the Asynchronous Monitoring Framework (AMF)
kernel component.
Exercise 2: Examining IMF monitoring on a resource

IMF monitoring is reconfigured on a resource of type Process.
291
Exercise 3: (Optional) Examining the IMF default configuration

The default configuration and status for IMF and AMF is examined on the west
cluster.
Lab 12: IMF and AMF
B285

1
opened.
mgt
Account: student
Password: veritas
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas
292 B286

terminal2.

Exercise 2: Examining IMF monitoring on a resource

In this exercise, you reconfigure IMF monitoring on a resource of type Process.
sym1
1

engine log.
Solution
End of Solution
From sym1:termial1, verify that the appsg service group is online on sym1.
Switch it if necessary.
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
Display the value of the resource type attribute named IMF for the Process
resource. Display the resource attribute value named MonitorMethod for the
appproc resource (of type Process).
Solution
hatype -display Process -attribute IMF

#Type
Attribute
Process
IMF
RegisterRetryLimit
293 Lab 12: IMF and AMF
Value
Mode
MonitorFreq
B287
hares -display appproc -attribute MonitorMethod

#Resource
appproc
appproc
Attribute
MonitorMethod
MonitorMethod
System
sym1
sym2
Value
IMF
IMF
End of Solution
(PID) and confirm that it is stopped. How soon after doing this does the fault
show up in the VCS engine log?
The fault is logged almost immediately.
Solution
ps -ef | grep loopy

root
root
18245
1 0 12:05 ?
24256 10510 0 13:09 pts/2
kill <loopy PID>
ps -ef | grep loopy

root
7212
9707
0 02:49 pts/2

00:00:00 grep loopy
00:00:00 grep loopy
End of Solution
294 B288
Wait for the appsg service group to fail over and come online on sym2. Then,
clear the fault on the appproc resource and switch the appsg service group to
sym1.
Solution

#Resource
appproc
appproc
Attribute
State
State
System
sym1
sym2
Value
FAULTED
ONLINE


#Resource
appproc
appproc
Attribute
State
State
System
sym1
sym2
Value
OFFLINE
ONLINE
System
sym1
sym2
Value
|OFFLINE|
|ONLINE|
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
Use the haimfconfig command to disable IMF for the Process agent.
Confirm that the appproc resource is being monitored using the Traditional
method.
Solution
haimfconfig -disable -agent Process

VCS INFO V-16-52-53 Log created at '/var/VRTSvcs/log/
haimfconfig_script.20581.log'.
VCS WARNING V-16-52-55 This operation will change configuration and dump the
changes by 'haconf -dump' command.
Do you want to continue (yes/no)?: [yes]
yes
Disabled IMF for all specified agents.

#Type
Attribute
Process
IMF
RegisterRetryLimit
Value
Mode
MonitorFreq

#Resource
appproc
appproc
Attribute
MonitorMethod
MonitorMethod
System
sym1
sym2
Value
Traditional
Traditional
End of Solution
B289
Confirm the MonitorInterval and OfflineMonitorInterval attribute values

are the default of 60 and 300 seconds respectively.
Solution
hatype -value Process MonitorInterval

60
hatype -value Process OfflineMonitorInterval

300
End of Solution
(PID) and confirm. Then, wait for the appproc resource to fault on sym1. How
soon after doing this does the fault show up in the VCS engine log?
The fault takes up to 60 seconds to be recognized. It can be shorter, depending on

when the last monitor occurred, but it will be delayed compared to detection under
IMF.
Solution
ps -ef | grep loopy
root
root
296 B290
17597 10480 0 11:49 pts/2

29732
1 0 11:15 ?
kill <loopy PID>
ps -ef | grep loopy

root
7212
9707
0 02:49 pts/2
00:00:00 grep loopy

00:00:00 grep loopy
hares -wait appproc State FAULTED -sys sym1
End of Solution

Clear the fault on the appproc resource and switch the appsg service group to
sym1.
Solution

#Resource
appproc
appproc
Attribute
State
State
System
sym1
sym2

#Resource
appproc
appproc
Attribute
State
State
Value
FAULTED
ONLINE
System
sym1
sym2
Value
OFFLINE
ONLINE
System
sym1
sym2
Value
|OFFLINE|
|ONLINE|
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
10 Confirm that the MonitorMethod attribute value for the appproc resource has
not changed.
Solution

#Resource
appproc
appproc
Attribute
MonitorMethod
MonitorMethod
System
sym1
sym2
Value
Traditional
Traditional
End of Solution
B291
11 Use the haimfconfig command to enable IMF for the Process agent. Probe the
approc resource on both systems and confirm that the appproc resource is now
is being monitored by IMF.
Solution
haimfconfig -enable -agent Process

VCS INFO V-16-52-53 Log created at '/var/VRTSvcs/log/
haimfconfig_script.24213.log'.
VCS WARNING V-16-52-55 This operation will change configuration and dump the
changes by 'haconf -dump' command.
yes
VCS WARNING V-16-52-112 This may restart the agents if agents are not
already IMF enabled.
yes
Restarting 'Process' agent to enable IMF.
IMF is successfully enabled for 'Process' agent.
hares -probe appproc -sys sym1

#Resource
appproc
appproc
Attribute
MonitorMethod
MonitorMethod
System
sym1
sym2
Value
IMF
Traditional
Note: If a normal monitor is performed by VCS for appproc on sym2

before this command is run then IMF will display for both nodes.
298 B292
hares -probe appproc -sys sym2

#Resource
appproc
appproc
Attribute
MonitorMethod
MonitorMethod
System
sym1
sym2
Value
IMF
IMF

#Type
Attribute
Process
IMF
RegisterRetryLimit
Value
Mode
MonitorFreq
End of Solution

12 From sym1:terminal2, terminate the VCS engine log display.

Solution
End of Solution
B293
Exercise 3: (Optional) Examining the IMF default configuration

In this exercise, you examine the default configuration and status for IMF and
AMF on the west cluster.
sym1
1
From sym1:terminal1, navigate to the /etc/sysconfig directory and

display the amf file. How is AMF set for system startup and shutdown?
Note: These files are identical by default on both systems and should be kept
that way unless advised otherwise by Symantec Support. It is not
necessary to check on both system unless directed otherwise.
AMF is set to start up at system boot and stop at system shutdown.
Solution
300 B294
cd /etc/sysconfig
cat amf
#
# AMF driver configuration file
#
# This file is sourced from amf initrc script
#
# Set the two environment variables below as follows:
#
#
1 = start or stop amf
#
0 = do not start or stop amf
#
AMF_START=1
AMF_STOP=1
End of Solution

Navigate to the /etc directory and display the amftab file.

Solution
cd /etc
cat amftab
#
# AMF driver configuration file
#
# This file is used by amf initrc script
#
#
# In order to change the default debug buffer size use:
# /opt/VRTSamf/bin/amfconfig -T dbglogsz=<dbg_logsize_in_kb> -c
#
# In order to change the default debug buffer size, process hash size
# and mount hash size use:
# /opt/VRTSamf/bin/amfconfig -T
dbglogsz=<dbg_logsize_in_kb>,processhashsz=<proc
esshashsz>,mnthashsz=<mnthashsz> -c
# To keep all the defaults use:
/opt/VRTSamf/bin/amfconfig c
End of Solution
Use the service command to check the status of AMF on both systems.
Note: Status is important on both cluster nodes in a production environment.
However, for this exercise, you will check the status on only one node.
Solution
301
service amf status

AMF: Module loaded and configured
ssh sym2 service amf status

AMF: Module loaded and configured
End of Solution
Lab 12: IMF and AMF
B295
Use the haimfconfig -display command to show which agents are

enabled (by default) for IMF.
Solution
haimfconfig -display
#Agent
Application
Db2udb
Mount
Netlsnr
Oracle
Process
Sybase
SybaseBk
STATUS
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
End of Solution
Use the hares -list MonitorMethod=IMF command to show which

resources are being monitored using IMF.
Solution
hares -list MonitorMethod=IMF

appmnt
appmnt
appproc
appproc
sym1
sym2
sym1
sym2
End of Solution
Confirm that the appsg is online on sym1.
Solution
302 B296
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution

Use the amfstat command to show the ONLINE monitors for both the
appproc and appmnt resources on sym1 because the appsg service group is
online on sym1.
Solution
amfstat
AMF Status Report
Registered Reapers (2):
=======================
RID
PID
0
13462
7
11445
MONITOR
1
1
Process ONLINE Monitors (1):

============================
RID
R_RID
PID
8
7
10042
TRIGG
0
0
REAPER
VCSMountAgent
VCSProcessAgent
GROUP
appproc
Mount ONLINE Monitors (1):

==========================
RID
R_RID
FSTYPE DEVICE
MOUNTPOINT
GROUP
CONTAINER
6
0
vxfs
/dev/vx/dsk/loopydatadg/loopydatavol /loopyfs
appmnt
<none>
End of Solution
Confirm loopy is running on sym1 and notice that the loopys PID matches
what amfstat shows for the PID for the appproc resource in the previous
step.
Solution
ps -ef | grep loopy

root
root
10042
27592
1 0 02:53 ?
9707 0 03:23 pts/2

00:00:00 grep loopy
End of Solution
B297
Confirm /loopyfs is mounted on sym1 and notice that the device and
mount point match what amfstat shows for the DEVICE and MOUNT
POINT for the appmnt resource in a previous step.
Solution

End of Solution
10 Use the amfstat command to show the OFFLINE monitors for both the
appproc and appmnt resources on sym2 because the appsg service group is
offline on sym2.
Solution
ssh sym2 /opt/VRTS/bin/amfstat

AMF Status Report
Registered Reapers (2):
=======================
RID
PID
0
8834
1
8838
MONITOR
1
1
Process OFFLINE Monitors (1):

=============================
RID
R_RID
PATH
ARGV0
EGID
GROUP
CONTAINER
ACTION
11
1
/bin/ksh
/bin/ksh
0<any>
appproc <none>
Allow
TRIGG
0
0
REAPER
VCSMountAgent
VCSProcessAgent
ARGS
UID
EUID
/loopyfs/loopy appsg 0
GID
<any>
Mount OFFLINE Monitors (1):

===========================
RID
R_RID
FSTYPE DEVICE
MOUNTPOINT
GROUP
CONTAINER
12
0
vxfs
/dev/vx/dsk/loopydatadg/loopydatavol /loopyfs
appmnt
<none>
End of Solution
End of lab
304 B298

Lab 13: Cluster communications

In this lab, you examine private cluster communications and make configuration
changes to LLT.
Exercise 2: Reconfiguring LLT
The LLT configuration files are examined and LLT is reconfigured.
Exercise 3: Jeopardy membership

Jeopardy membership in the cluster is examined.
305 Lab 13: Cluster communications
B299

1
opened.
mgt
Account: student
Password: veritas
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas
306 B300

terminal2.

Exercise 2: Reconfiguring LLT

In this exercise, you examine the LLT configuration files reconfigure LLT.
sym1
1
From sym1:terminal1, display a summary status of the cluster and notice of

the systems where the service groups are online.
Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
System
Probed
AutoDisabled
State
B
B
B
B
sym1
sym2
sym1
sym2
Y
Y
Y
Y
N
N
N
N
ONLINE
OFFLINE
ONLINE
OFFLINE
ClusterService
ClusterService
appsg
appsg
End of Solution
Display the ports open in GAB.
Solution
gabconfig -a
===============================================================
Port a gen
cfca06 membership 01
Port h gen
End of Solution
B301
Display the configured and active status of the LLT links.

Solution
lltstat -nvv configured

Node
* 0 sym1
1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
eth4
eth5
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
OPEN
lltstat -nvv active

Node
* 0 sym1
1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
eth4
eth5
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
OPEN
End of Solution
Navigate to the /etc directory and display the contents of the llthosts and
llttab files.
Solution
308 B302
cd /etc
cat llthosts
0 sym1
1 sym2
cat llttab
set-node sym1
set-cluster 42845
link eth4 eth-00:0c:29:5a:b1:16 - ether - link eth5 eth-00:0c:29:5a:b1:20 - ether - -
End of Solution

sym2
From sym2:terminal1, navigate to the /etc directory and display the

contents of the lltthosts and llttab files. What is the difference
between these files on the two system?
The set-node directive in the llttab file is different on each system. On Linux
systems, the default device listed in each link directive includes the MAC address of
the device. This will be unique for each device on each system.
Solution
cd /etc
cat llthosts
0 sym1
1 sym2
cat llttab
set-node sym2
set-cluster 42845
link eth4 eth-00:0c:29:2d:77:9a - ether - link eth5 eth-00:0c:29:2d:77:a4 - ether - -
End of Solution
sym1
From sym1:terminal1, create a backup copy of the llttab file. Modify the
llttab file by making the following changes:
Change the value of set-cluster directive to 5.
Add a low priority LLT link after the link eth5 line by adding the line
link-lowpri eth0 eth0 - ether - After you have made the edits, the affected portion of the files (added and
changed portions are bolded) should be:
B303
set-node sym1
set-cluster 5
link eth4 eth-00:0c:29:5a:b1:16 - ether - link eth5 eth-00:0c:29:5a:b1:20 - ether - link-lowpri eth0 eth0 - ether - After you have made the changes, save and close the llttab file.
Solution
cp -p llttab llttab.ORIG
vi llttab
two changes shown in bold.
cat llttab
set-node sym1
set-cluster 5
link eth4 eth-00:0c:29:5a:b1:16 - ether - link eth5 eth-00:0c:29:5a:b1:20 - ether - link-lowpri eth0 eth0 - ether - -
End of Solution
sym2
310 B304
From sym2:terminal1, create a backup copy of the llttab file. Modify the
llttab file by making the following changes:
Change the value of set-cluster directive to 5.
Add a low priority LLT link after the link eth5 line by adding the line
link-lowpri eth0 eth0 - ether - After you have made the edits, the affected portion of the files (added and
changed portions are bolded) should be:
set-node sym2
set-cluster 5
link eth4 eth-00:0c:29:2d:77:9a - ether - link eth5 eth-00:0c:29:2d:77:a4 - ether - link-lowpri eth0 eth0 - ether - After you have made the changes, save and close the llttab file.

Solution
cp -p llttab llttab.ORIG
vi llttab
two changes shown in bold.
cat llttab
set-node sym2
set-cluster 42845
link eth4 eth-00:0c:29:2d:77:9a - ether - link eth5 eth-00:0c:29:2d:77:a4 - ether - link-lowpri eth0 eth0 - ether - -
End of Solution
sym1
From sym1:terminal1, confirm that the VCS configuration is closed and force
stop VCS on all cluster nodes. Then, confirm that VCS is shutdown by waiting
for GAB port h to close on both systems and confirm that the loopy process is
still running.
Solution
311

Note: An error indicating that the VCS configuration is not writable can
be ignored as the VCS configuration is already closed.
hastop -all -force
gabconfig -a
===============================================================
Port a gen
4c9c01 membership 01

B305

===============================================================
Port a gen
4c9c01 membership 01
ps -ef | grep loopy

root
root
11328
18099
1 0 Jan21 ?
9341 0 02:21 pts/3

00:00:00 grep loopy
End of Solution
Use the gabconfig -U command to shutdown GAB.

Solution
gabconfig -U
gabconfig -a
===============================================================
End of Solution
sym2
10 From sym2:terminal1, use the gabconfig -U command to shutdown
GAB.
Solution
312 B306
gabconfig -U
gabconfig -a
===============================================================
End of Solution

sym1
11 From sym1:terminal1, use the lltconfig -U command to shutdown LLT.

Solution
lltconfig -U
lltconfig: this will attempt to stop and reset LLT. Confirm (y/n)?
y
LLT lltstat ERROR V-14-2-15000 open /dev/llt failed: No such file or
directory

Node
0 sym1
1 sym2
State
IDLE
IDLE
Link
Status
Address
End of Solution
sym2
12 From sym2:terminal1, use the lltconfig -U command to shutdown LLT.
Solution
313
lltconfig -U
lltconfig: this will attempt to stop and reset LLT. Confirm (y/n)?

Node
0 sym1
1 sym2
State
IDLE
IDLE
Link
Status
Address
End of Solution

B307
sym1
13 From sym1:terminal1, use the lltconfig -c command to start up LLT.
Use the lltstat -nvv configured command to confirm it has started.

Solution
lltconfig -c

Node
* 0 sym1
1 sym2
State
OPEN
Link
Status
eth4
eth5
eth0
UP
UP
UP
eth4
eth5
eth0
DOWN
DOWN
DOWN
Address
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
IDLE
Note: The DOWN status is expected since LLT has not yet been started
on sym2 and GAB has not yet been started. Therefore, no I/O
activity across the LLT links has occurred.
End of Solution
sym2
314 B308
14 From sym2:terminal1, use the lltconfig -c command to start up LLT.
Use the lltstat -nvv configured command to confirm it has started.

Solution
lltconfig -c

Node
State
Link
Status
Address

0 sym1
IDLE
* 1 sym2
eth4
eth5
eth0
DOWN
DOWN
DOWN
eth4
eth5
eth0
UP
UP
UP
OPEN
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
End of Solution
15 Use the sh /etc/gabtab command to start up GAB and use the
gabconfig -a command to attempt to confirm GAB has started. Use the

lltstat -nvv configured command to confirm LLT has started.
Solution
sh /etc/gabtab
gablogd: Keeping 20 log files of 8388608 bytes each in |/var/log/gab_ffdc|
directory. Daemon log size limit 8388608 bytes
Started gablogd
gabconfig -a
===============================================================
Note: GAB will not seed until it is started on the sym1. LLT will also not
yet show any status due to lack of GAB I/O.
c

Node
0 sym1
315
* 1 sym2
State
Link
CONNWAIT
eth4
eth5
eth0
OPEN
eth4
eth5
eth0
Status
Address
DOWN
DOWN
DOWN
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
Note: The state of sym1 shows CONNWAIT instead of IDLE and sym1s
MAC addresses are shown since LLT is now talking on the two nodes.
End of Solution

B309
sym1
16 From sym1:terminal1, use the sh /etc/gabtab command to start up
GAB and use the gabconfig -a command to confirm GAB has started.
Use the lltstat -nvv configured command to confirm LLT has
started.
Solution
sh /etc/gabtab
gablogd: Keeping 20 log files of 8388608 bytes each in |/var/log/gab_ffdc|
directory. Daemon log size limit 8388608 bytes
Started gablogd
gabconfig -a
===============================================================
Port a gen
8da501 membership 01
Note: It may take a few seconds for GAB to show port a membership.
c

Node
* 0 sym1
1 sym2
316 B310
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
End of Solution

sym2
17 From sym2:terminal1, use the gabconfig -a command to confirm GAB
status. Then use the lltstat -nvv configured command to confirm

LLT status. Once confirmed, start VCS, use another gabconfig -a
command to check GAB status and notice the difference in the output.
Solution
gabconfig -a
===============================================================
Port a gen

Node
0 sym1
* 1 sym2
317
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
hastart
gabconfig -a
===============================================================
Port a gen
81fb01 membership 01
Port h gen
81fb03 membership ;1
Port h gen
81fb03
visible 0
End of Solution

B311
sym1
18 From sym1:terminal1, start VCS and confirm GAB and summary cluster
status. If necessary, switch the ClusterService service group to sym1.

Note: The ClusterService service group can be online on sym2. It is
switched here to align it with subsequent sample lab solution output.
Solution
hastart
gabconfig -a
===============================================================
Port a gen
Port h gen
318 B312
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
System
Probed
AutoDisabled
State
B
B
B
B
sym1
sym2
sym1
sym2
Y
Y
Y
Y
N
N
N
N
OFFLINE
ONLINE
ONLINE
OFFLINE
ClusterService
ClusterService
appsg
appsg
hagrp -wait ClusterService State ONLINE -sys sym1
hagrp -state ClusterService

#Group
Attribute
ClusterService State
ClusterService State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution

Exercise 3: Jeopardy membership

In this exercise, jeopardy membership in the cluster is examined.
sym1
1

engine log.
Solution
End of Solution
From sym1:terminal1, confirm that all service groups are online on sym1 and
display GAB status to confirm membership on port a and port h.
Solution
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
319
Attribute
State
State
State
State
System
sym1
sym2
sym1
sym2
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
gabconfig -a
===============================================================
Port a gen
Port h gen
End of Solution

B313
Display the active status for LLT.

Solution
lltstat -nvv active

Node
* 0 sym1
1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
End of Solution
sym2
From sym2:terminal1, display the active status for LLT.

Solution
lltstat -nvv active

Node
0 sym1
* 1 sym2
320 B314
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
End of Solution

sym1

/student/labs/vcs/vcs60/lltlink directory, perform a long listing
of the contents.
Solution
cd /student/labs/vcs/vcs60/lltlink
ls -l
total 12
-rwxr-xr-x 1 root root 4355 Nov 15 13:35 lltlink.pl
drwxr-xr-x 3 root root 4096 Mar 29 2011 setup
End of Solution
Run the lltlink.pl script. Enter ssh as the access method. Select option 4
to disable the eth4 link on sym1.
Solution
./lltlink.pl
Checking your Operating System and setting some variables . . .
Your Operating System is . . . Linux, x86_64 . . . Done!
What is the remote communication method? rsh or ssh [ssh]:
321
ssh
Enable or Disable LLT links Script
This script can be used to enable and disable LLT links on the local
system or on all nodes in the cluster.
Choose the desired option from the list below.
1. Disable all private links (eth4 eth5 eth0) on all nodes in the
cluster
2.
3.
4.
5.
6.
7.
Disable all private links (eth4 eth5 eth0) on sym1 only

Enable all private links (eth4 eth5 eth0) on sym1 only
Disable eth4 on sym1 only
Enable eth4 on sym1 only

B315
8. Disable eth0 on sym1 only

9. Enable eth0 on sym1 only
Which setup do you wish to run? Enter 1 - 9:
4
Disabling link eth4 on sym1 . . . done.
Notice the LLT messages in sym1:terminal2 for the eth4 LLT link timing
out. This may take a few seconds.
End of Solution
Display GAB status and the configured and active status of LLT. What has
changed?
eth4 for sym2 shows as DOWN.
Solution
gabconfig -a
===============================================================
Port a gen
Port h gen

Node
* 0 sym1
322 B316
1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
DOWN
UP
UP
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
Link
Status
OPEN
lltstat -nvv active

Node
* 0 sym1
State
OPEN
eth4
eth5
eth0
1 sym2
UP
UP
UP
Address
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
OPEN

eth4
eth5
eth0
DOWN
UP
UP
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
End of Solution
sym2
From sym2:terminal1, display the configured and active status of llt. Is there
any difference from the sym1?
On sym2, the eth4 link for sym1 is show as DOWN. On sym1, the eth4 link for
sym2 was shown as DOWN.
Solution

Node
0 sym1
* 1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
eth0
DOWN
UP
UP
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
lltstat -nvv active

Node
0 sym1
* 1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
eth0
DOWN
UP
UP
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
End of Solution
B317
sym1
From sym1:terminal2, run the lltlink.pl script. Select option 6 to

disable the eth5 link on sym1.
Solution
./lltlink.pl
cluster
2.
3.
4.
5.
6.
7.
8.
9.

6
Disabling link eth5 on sym1 . . . done.
324 B318
Notice the LLT messages in sym1:terminal2 for the eth5 LLT link timing
out. This may take a few seconds.
End of Solution

10 Display GAB status and the configured and active status of LLT. What has
changed?
Note: It may take a few seconds for the status to change.
The cluster is in jeopardy. The eth4 and eth5 links for sym2 shows as DOWN.
Solution
gabconfig -a
===============================================================
Port a gen
Port a gen
8da501
jeopardy ;1
Port h gen
Port h gen
8da504
jeopardy ;1

Node
* 0 sym1
1 sym2
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
DOWN
DOWN
UP
00:0C:29:2D:77:72
Link
Status
OPEN
lltstat -nvv active

Node
* 0 sym1
State
OPEN
1 sym2
State
OPEN
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
DOWN
DOWN
UP
00:0C:29:2D:77:72
OPEN
End of Solution
B319
sym2
11 From sym2:terminal1, display GAB status and the configured and active
status of LLT. Is there any difference from the sym1?
On sym2, the eth4 and eth5 links for sym1 is show as DOWN.
Solution
gabconfig -a
===============================================================
Port a gen
Port a gen
8da501
jeopardy ;1
Port h gen
Port h gen
8da504
jeopardy ;1

Node
0 sym1
* 1 sym2
326 B320
State
OPEN
Link
Status
Address
eth4
eth5
eth0
DOWN
DOWN
UP
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
lltstat -nvv active

Node
0 sym1
* 1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
eth0
DOWN
DOWN
UP
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
End of Solution

sym1
12 From sym1:terminal2, review the end of the VCS engine log and notice the
log entries related to the jeopardy status.

Note: You can use the terminal window scroll bar to navigate to earlier
entries. Alternatively use the tail command from the
sym1:terminal1.
Solution
...
2011/11/21 13:42:03 VCS WARNING V-16-1-11141 LLT heartbeat link status
changed. Previous status = eth4 UP eth5 UP eth0 UP; Current status = eth4
DOWN eth5 UP eth0 UP.
0x3, DDNA: 0x2
2011/11/21 13:53:11 VCS ERROR V-16-1-10111 System sym2 (Node '1') is in
Regular and Jeopardy Memberships - Membership: 0x3, Jeopardy: 0x2
bin/triggers/injeopardy is not a trigger scripts directory or can not be
executed
changed. Previous status = eth4 DOWN eth5 UP eth0 UP; Current status = eth4
DOWN eth5 DOWN eth0 UP.
...
End of Solution
13 From sym1:terminal1, Run the lltlink.pl script. Select option 5 to
enable the eth4 link on sym1.
Solution
./lltlink.pl
cluster
2.
3.
4.
5.

B321
6.
7.
8.
9.

5
Enabling link eth4 on sym1 . . . done.
End of Solution
changed?
Note: It may take a few seconds for the status to change.
The cluster is no longer in jeopardy membership.
Solution
gabconfig -a
===============================================================
Port a gen
Port h gen
328 B322

Node
* 0 sym1
1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
DOWN
UP
00:0C:29:2D:77:9A
Link
Status
OPEN
00:0C:29:2D:77:72
lltstat -nvv active

Node
* 0 sym1
State
OPEN
eth4
eth5
UP
UP
Address
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20

1 sym2
eth0
UP
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
DOWN
UP
00:0C:29:2D:77:9A
OPEN
00:0C:29:2D:77:72
End of Solution
15 From sym1:terminal2, examine the end to the VCS engine log and take note
of the log entries related re-establishment of regular membership status.

Solution
...
changed. Previous status = eth4 DOWN eth5 DOWN eth0 UP; Current status =
eth4 UP eth5 DOWN eth0 UP.
0x3, DDNA: 0x0
...
End of Solution
16 From sym1:terminal1, run the lltlink.pl script. Select option 7 to enable
the eth5 link on sym1.

Solution
./lltlink.pl
cluster
2.
3.
4.
5.
6.
7.
8.
9.

7
End of Solution
B323
changed?
All LLT links now show as up.
Solution
gabconfig -a
===============================================================
Port a gen
Port h gen

Node
* 0 sym1
1 sym2
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
lltstat -nvv active
1 sym2
Link
OPEN

Node
* 0 sym1
330 B324
State
OPEN
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
End of Solution
18 From sym1:terminal2, terminate the monitoring of the engine log.

Solution
Press Ctrl-C.
End of Solution

sym2
19 From sym2:terminal1, display GAB status and the configured and active
status of LLT. Is there any difference from the sym1?
There is no difference.
Solution
gabconfig -a
===============================================================
Port a gen
Port h gen

Node
0 sym1
* 1 sym2
331
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
lltstat -nvv active

Node
0 sym1
* 1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
End of Solution
End of lab

B325
332 B326

Lab 14: Configuring SCSI3 disk-based I/O fencing

In this lab, you enable disk-based I/O fencing and observe cluster behavior due to
private interconnect communication failures on the west cluster.
Exercise 2: Fencing configuration pre-checks
Several I/O fencing pre-checks are performed prior to configuring I/O fencing.
Exercise 3: Configuring and testing disks for I/O fencing

Three disks are configured and tested for I/O Fencing coordinator disk suitability.
A representative single disk is tested for data disk suitability.
Exercise 4: Configuring VCS for I/O fencing
VCS is configured for SCSi3 disk based I/O fencing.
Exercise 5: I/O fencing configuration verification
The I/O fencing configuration is verified.
Exercise 6: Verifying data disks for I/O fencing
Data disks are verified for I/O Fencing.
Exercise 7: Testing communication failures
The impact of I/O fencing on private network communication failures is examined.
333 Lab 14: Configuring SCSI3 disk-based I/O fencing
B327

1
opened.
mgt
Account: student
Password: veritas
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas
334 B328

terminal2.

Exercise 2: Fencing configuration pre-checks

In this exercise, you perform several I/O Fencing pre-checks prior to configuring
I/O fencing.
sym1
1

examine the vxfen file to determine how I/O fencing is started and stopped at
system startup and shutdown.
Note: These steps apply to both cluster systems and strictly speaking do not
need to be performed on sym2 unless otherwise noted.
Solution
cd /etc/sysconfig
ls -l vxfen
-rw-r--r-- 1 root root 3396 Nov 14 09:57 vxfen
more vxfen
...
#
# This file is sourced from /etc/init.d/vxfen.
#

#
#1 = start or stop vxfen
#0 = do not start or stop vxfen
#
VXFEN_START=0
VXFEN_STOP=0
...
End of Solution
B329
Navigate to the /etc directory and recursively list I/O fencing configuration
files whose name starts with vxfen. List the contents of any subdirectories
found.
Solution
cd /etc
ls -lR vxfen.d
vxfen.d:
total 32
-r-xr--r-drwxr-xr-x
-rwxr--r--r-xr--r--r-xr--r--r-xr--r--r-xr--r--
1
2
1
1
1
1
1
root
root
root
root
root
root
root
root 1304 Oct 13 17:06 README

root 4096 Oct 25 14:16 script
root 4113 Oct 13 17:06 vxfenmode_cps
root 281 Oct 13 17:06 vxfenmode_disabled
root 516 Oct 13 17:06 vxfenmode_scsi3_dmp
root 677 Oct 13 17:06 vxfenmode_scsi3_raw
root 416 Oct 13 17:06 vxfenmode_sybase
vxfen.d/script:
total 44
-r-xr--r-- 1 root root 42067 Oct 13 17:06 vxfen_scriptlib.sh
End of Solution
Is the I/O fencing GAB port open?
GAB port b is not open.
Solution
gabconfig -a

===============================================================
Port a gen
Port h gen
336 B330
End of Solution
Determine the value of the cluster attribute named UseFence.

UseFence value:
NONE
Solution
haclus -value UseFence

NONE
End of Solution

Use the vxfenadm command to display I/O fencing status. Use the
vxfenconfig command to determine if there are any coordination points in
use.
Solution
vxfenadm -d
VXFEN vxfenadm ERROR V-11-2-1101 Open failed for device: /dev/vxfen
vxfenconfig -l
VXFEN vxfenconfig ERROR V-11-2-1002 Open failed for device: /dev/vxfen with
error 2
Note: These errors are expected when I/O fencing has not been
configured.
End of Solution
B331
Exercise 3: Configuring and testing disks for I/O fencing

In this exercise, you configure and test three disks for I/O fencing coordinator disk
suitability. You then test a representative single disk for data disk suitability.
sym1
1
From sym1:terminal1, initialize three disks for use as coordinator disk

candidates, but only if the disks show a status of in error or online invalid. If
they show a status of online then skip to the next step. The disks to use are:
emc0_d10
emc0_d11
emc0_d12
Solution
DEVICE
emc0_dd1
emc0_dd2
emc0_dd3
emc0_dd4
emc0_dd5
emc0_dd6
emc0_dd7
emc0_dd8
emc0_dd9
emc0_d10
emc0_d11
emc0_d12
sda
sdb
338 B332
TYPE
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:none
auto:none
DISK
GROUP
STATUS
loopydatadg01 loopydatadg online
online
online
online
online invalid
online invalid
online invalid
online invalid
online invalid
vxdisksetup -i emc0_d10

DEVICE
emc0_dd1
emc0_dd2
emc0_dd3
emc0_dd4
TYPE
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
DISK
-
GROUP
STATUS

emc0_dd5
emc0_dd6
emc0_dd7
emc0_dd8
emc0_dd9
emc0_d10
emc0_d11
emc0_d12
sda
sdb
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:none
auto:none
loopydatadg01 loopydatadg online clone_disk

(sharedatadg) online clone_disk
online
online
online
online
online
online
online invalid
online invalid
End of Solution
Create a disk group named testwestdg made up of the three initialed disks:
emc0_d10
emc0_d11
emc0_d12
Solution
vxdg init testwestdg emc0_d10 emc0_d11 emc0_d12
vxdg list
NAME
loopydatadg
testwestdg
STATE
enabled,cds
enabled,cds
ID
1319645563.37.sym1
1321293346.59.sym1
End of Solution
Use the vxfentsthdw command to test the disks as dmp nodes and for
coordinator disk suitability. The cluster nodes are:
sym1
sym2
Solution
vxfentsthdw -c testwestdg -d
Veritas vxfentsthdw version 6.0.000.000-GA Linux
The utility vxfentsthdw works on the two nodes of the cluster.

The utility verifies that the shared storage one intends to use is
configured to support I/O fencing. It issues a series of vxfenadm
commands to setup SCSI-3 registrations on the disk, verifies the
registrations on the disk, and removes the registrations from the disk.
The logfile generated for vxfentsthdw is /var/VRTSvcs/log/vxfen/
vxfentsthdw.log.19640
Enter the first node of the cluster:
B333
sym1
Enter the second node of the cluster:
sym2
********************************************
Testing sym1 /dev/vx/rdmp/emc0_d10s3 sym2 /dev/vx/rdmp/emc0_d10s3
Evaluate the disk before testing ........................ No Pre-existing
keys
RegisterIgnoreKeys on disk /dev/vx/rdmp/emc0_d10s3 from node sym1 ......
Passed
Verify registrations for disk /dev/vx/rdmp/emc0_d10s3 on node sym1 .....
Passed
RegisterIgnoreKeys on disk /dev/vx/rdmp/emc0_d10s3 from node sym2 ......
Passed
Passed
Passed
Preempt and abort key KeyA using key KeyB on node sym2 .................
Passed
Passed
Passed
Remove key KeyB on node sym2 ...........................................
Passed
Check to verify there are no keys from node sym1 .......................
Passed
ALL tests on the disk /dev/vx/rdmp/emc0_d10s3 have PASSED.
The disk is now ready to be configured for I/O Fencing on node sym1
as a COORDINATOR DISK.
...
Removing test keys and temporary files, if any...
End of Solution
340 B334
Destroy the disk group named testwestdg made up of the three initialed disks.
Solution
vxdg destroy testwestdg
vxdg list
NAME
loopydatadg
STATE
enabled,cds
ID
1319645563.37.sym1
End of Solution

Initialize a spare data disk for I/O fencing data disk suitability testing using the
disk emc0_dd9, but only if it does not already show a status of online.
Otherwise, skip to the next step.
Solution

DEVICE
emc0_dd1
emc0_dd2
emc0_dd3
emc0_dd4
emc0_dd5
emc0_dd6
emc0_dd7
emc0_dd8
emc0_dd9
emc0_d10
emc0_d11
emc0_d12
sda
sdb
DISK
GROUP
STATUS
loopydatadg01 loopydatadg online
online
online
online invalid
online
online
online
online invalid
online invalid
vxdisksetup -i emc0_dd9

DEVICE
emc0_dd1
emc0_dd2
emc0_dd3
emc0_dd4
emc0_dd5
emc0_dd6
emc0_dd7
emc0_dd8
emc0_dd9
emc0_d10
emc0_d11
emc0_d12
sda
sdb
TYPE
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:none
auto:none
TYPE
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
auto:none
auto:none
DISK
GROUP
STATUS
(symdbdatadg) online clone_disk
loopydatadg01 loopydatadg online clone_disk
(sharedatadg) online clone_disk
online
online
online
online
online
online
online invalid
online invalid
End of Solution
Note: Step 6 is optional.
Use the vxfentsthdw command to test the initialized disk for data disk
suitability. Use the following information:
Test in dmp mode and to be manually prompted for the node names and
device name for each node.
Respond y to continue when warned about disk data destruction.
Node names: sym1 and sym2
B335
DMP device name on both systems:

/dev/vx/rdmp/emc0_dd9
Record the log file located in the /var/VRTSvcs/log/vxfen directory.
Log file:
The log file will be in the form of vxfentsthdw.log.xxxxx
Note: The logfile name displays early and quickly. It is easily missed.
Solution
vxfentsthdw -d -m
Veritas vxfentsthdw version 6.0.000.000-GA Linux
The utility vxfentsthdw works on the two nodes of the cluster.
The utility verifies that the shared storage one intends to use is
configured to support I/O fencing. It issues a series of vxfenadm
commands to setup SCSI-3 registrations on the disk, verifies the
registrations on the disk, and removes the registrations from the disk.
******** WARNING!!!!!!!! ********

THIS UTILITY WILL DESTROY THE DATA ON THE DISK!!
Do you still want to continue : [y/n] (default: n)
y
The logfile generated for vxfentsthdw is /var/VRTSvcs/log/vxfen/
Enter the first node of the cluster:
sym1
Enter the second node of the cluster:
342 B336
sym2
Enter the disk name to be checked for SCSI-3 PGR on node sym1 in the format:
for dmp: /dev/vx/rdmp/sdx
for raw: /dev/sdx
Make sure it is the same disk as seen by nodes sym1 and sym2
Enter the disk name to be checked for SCSI-3 PGR on node sym2 in the format:
for dmp: /dev/vx/rdmp/sdx
for raw: /dev/sdx
Make sure it is the same disk as seen by nodes sym1 and sym2

Evaluate the disk before testing ........................ No Pre-existing
keys
RegisterIgnoreKeys on disk /dev/vx/rdmp/emc0_dd9 from node sym1 ........
Passed
Verify registrations for disk /dev/vx/rdmp/emc0_dd9 on node sym1 .......
Passed
Passed
Passed
...
Passed
Passed
Check to verify there are no reservations on disk /dev/vx/rdmp/emc0_dd9 from
node sym1 Passed
Check to verify there are no reservations on disk /dev/vx/rdmp/emc0_dd9 from
node sym2 Passed
Passed
Passed
Passed
Passed
Clear PGR on node sym1 .................................................
Passed
Passed
ALL tests on the disk /dev/vx/rdmp/emc0_dd9 have PASSED.
The disk is now ready to be configured for I/O Fencing on node sym1.
ALL tests on the disk /dev/vx/rdmp/emc0_dd9 have PASSED.
The disk is now ready to be configured for I/O Fencing on node sym2.
Removing test keys and temporary files, if any...
ls -l /var/VRTSvcs/log/vxfen/vxfentsthdw.log.xxxxx
-rw-r----- 1 root root 4498 Nov 15 10:52 /var/VRTSvcs/log/vxfen/

End of Solution
B337
Exercise 4: Configuring VCS for I/O fencing

In this exercise, you configure VCS for SCSI3 disk based I/O fencing.
sym1
1
From sym1:terminal1, navigate to the /opt/VRTS/install directory and

list the contents.
Solution
ls -l
total 60
drwxr-x--- 3 root root 4096 Nov 1 08:21 bin
-rwxr-x--- 1 root root 1222 Nov 1 08:21 installdmp
-rwxr-x--- 1 root root 1221 Nov 1 08:21 installfs
-rwxr-x--- 1 root root 1221 Nov 1 08:21 installsf
-rwxr-x--- 1 root root 1223 Nov 1 08:21 installsfha
-rwxr-x--- 1 root root 1222 Nov 1 08:21 installvcs
-rwxr-x--- 1 root root 1221 Nov 1 08:21 installvm
drwxr-xr-x 10 root root 4096 Nov 14 11:36 logs
-rwxr-x--- 1 root root 1384 Nov 1 08:21 showversion
-rwxr-x--- 1 root root 1222 Nov 1 08:21 uninstalldmp
-rwxr-x--- 1 root root 1221 Nov 1 08:21 uninstallfs
-rwxr-x--- 1 root root 1221 Nov 1 08:21 uninstallsf
-rwxr-x--- 1 root root 1223 Nov 1 08:21 uninstallsfha
-rwxr-x--- 1 root root 1222 Nov 1 08:21 uninstallvcs
-rwxr-x--- 1 root root 1221 Nov 1 08:21 uninstallvm
End of Solution
344B338

Use the installsfha script with the fencing option to begin configuring
VCS for I/O fencing. Use the following information.
Configure disk based fencing
Acknowledge that VCS will be restarted
Confirm that you have SCSI3 PR enabled disks
Solution
./installsfha -fencing
Observe the system checks.

Checking communication on sym1 .................................... Done
Checking release compatibility on sym1 ............................ Done
Checking VCS installation on sym1 .................. Version 6.0.000.000
Observe the cluster information verification

Cluster information verification:
Cluster Name: west
Cluster ID Number: 5
Systems: sym1 sym2
To configure I/O fencing on the cluster, type: y
Observe the cluster verification checks.
Checking
Checking
Checking
Checking
Checking
Checking
communication on sym1 .................................... Done

release compatibility on sym1 ............................ Done
VCS installation on sym1 .................. Version 6.0.000.000
communication on sym2 .................................... Done
To Configure disk based fencing, type: 2
To acknowledge that fencing requires a restart of VCS and to continue,

type: y
To confirm that you have SCSI3 PR enabled disks, type: y
End of Solution
B339
Continue configuring VCS for I/O fencing using the following information.
Create a new disk group using the disks:
emc0_d10
emc0_d11
emc0_d12
Disk group name: westfendg
Disk policy for the disks: dmp
Stop VCS and apply fencing configuration on all nodes at this time
Solution
To Create a new disk group, type: 1
Observe the list of available disks to create a new disk group.
1)
2)
3)
4)
5)
6)
346 B340
emc0_dd7
emc0_dd8
emc0_dd9
emc0_d10
emc0_d11
emc0_d12
To Select odd number of disks and at least three disks to form a disk
group, type: 4 5 6
To Enter the new disk group name, type: westfendg
To continue with the selected disk group, type: y
To Enter disk policy for the disk(s), type: dmp
To confirm that the I/O fencing configuration is correct, type: y
To stop VCS and apply fencing configuration on all nodes at this time,
type: y
Observe the steps taken to complete the I/O fencing configuration.

Stopping
Stopping
Starting
Starting
Updating
Starting
Starting
VCS on sym1 ..............................................

VCS on sym2 ..............................................
Fencing on sym1 ..........................................
Fencing on sym2 ..........................................
main.cf with fencing .....................................
VCS on sym1 ..............................................
VCS on sym2 ..............................................
Done
Done
Done
Done
Done
Done
Done
I/O Fencing configuration ......................................... Done
End of Solution

Complete the configuring of VCS for I/O fencing by reviewing the summary
file. Optionally, examine the log file.
Solution
End of Solution
B341
Exercise 5: I/O fencing configuration verification

In this exercise, you verify the I/O fencing configuration.
sym1
1

examine the vxfen file to determine how I/O fencing is now configured to be
started and stopped at system startup and shutdown.
Note: These checks should be performed on both cluster systems, but for
purposes of this lab, checking them on just sym1 is okay unless
otherwise directed.
Will I/O Fencing be started on system startup? Will I/O Fencing be stopped at
system shutdown?
I/O fencing will be started on system boot and stopped on system shutdown.
Solution
cd /etc/sysconfig
ls -l vxfen
-rw-r--r-- 1 root root 3396 Nov 15 09:57 vxfen
348 B342
more vxfen
...
#
# This file is sourced from /etc/init.d/vxfen.
#

#
#1 = start or stop vxfen
#0 = do not start or stop vxfen
#
VXFEN_START=1
VXFEN_STOP=1
...
End of Solution

Navigate to the /etc directory and list I/O fencing configuration files whose
name starts with vxfen. Display the file contents. Ignore the vxfen.d sub
directory as that was examined earlier. Which file contains the name of the
coordinator disk group used during I/O Fencing configuration? What is in the
vxfentab file?
The vxfendg file contains the name of the coordinator disk group. The vxfentab
file lists the coordinator disks for this configuration.
Solution
cd /etc
ls -ld vxfen*
drwxr-xr-x
-rw-r--r--r-xr--r--rw-r-----
3
1
1
1
root
root
root
root
root 4096 Jan 23 14:05

root
10 Jan 26 13:44
root 516 Jan 26 13:44
root 192 Jan 26 13:44
vxfen.d
vxfendg
vxfenmode
vxfentab
more vxfendg
westfendg
more vxfenmode
#
# vxfen_mode determines in what mode VCS I/O Fencing should work.
#
# available options:
# scsi3
- use scsi3 persistent reservation disks
# customized - use script based customized fencing
# disabled
- run the driver but don't do any actual fencing
#
vxfen_mode=scsi3
#
# scsi3_disk_policy determines the way in which I/O Fencing communicates
with
# the coordination disks.
#
# dmp - use dynamic multipathing
# raw - connect to disks using the native interface
#
scsi3_disk_policy=dmp
more vxfentab
#
# /etc/vxfentab:
# DO NOT MODIFY this file as it is generated by the
# VXFEN rc script from the file /etc/vxfendg.
B343
#
/dev/vx/rdmp/emc0_d10s3
End of Solution
Determine if the I/O fencing GAB port is open.

Solution
gabconfig -a
===============================================================
Port a gen
Port b gen
Port h gen
End of Solution
Determine the value of the cluster attribute named UseFence.

Solution

SCSI3
End of Solution
Use the vxfenadm command to display I/O fencing status and use the
vxfenconfig command to determine if there are any devices in use as
coordination disks or coordination points.
Solution
350 B344
vxfenadm -d
I/O Fencing Cluster Information:
================================
Fencing
Fencing
Fencing
Cluster
Protocol Version: 201

Mode: SCSI3
SCSI3 Disk Policy: dmp
Members:
* 0 (sym1)
1 (sym2)
RFSM State Information:

node
0 in state
node
1 in state
8 (running)
8 (running)

vxfenconfig -l
I/O Fencing Configuration Information:
======================================
Single Disk Flag
: 0
Count
: 3
Disk List
Disk Name
Major
201
201
201
Minor
131
83
35
Serial Number
512345600000000A
5123456000000009
512345600000000B
Policy
dmp
dmp
dmp
Note: The order of the command output may be different than what is
displayed in this sample output.
End of Solution
Display the state of the coordinator disk group.

Solution
vxdisk -o alldgs list | grep westfendg

emc0_d10
emc0_d11
emc0_d12
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
(westfendg)
(westfendg)
(westfendg)
online
online
online
End of Solution
sym2
351
From sym2:terminal1, display the state of the coordinator disk group. Is it

imported or deported on all cluster nodes?
The westfendg disk group is deported on all cluster nodes.
Solution

emc0_d10
emc0_d11
emc0_d12
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
(westfendg)
(westfendg)
(westfendg)
online
online
online
End of Solution

B345
sym1
8
From sym1:terminal1, use the vxdisk list emc0_d10 | grep

flags command to confirm that the coordinator flag is set. This applies to all
coordinator disks and the coordinator disk group.
Solution
vxdisk list emc0_d10 | grep flags

flags:
online ready private autoconfig coordinator
End of Solution
Use the vxdisk path | grep em0_d10 command to determine how

many dmp I/O paths there are to each coordinator disk using the emc0_d10
coordinator disk as an example. Then, display the registrations placed on the
coordinator disks. How many registrations are placed on each disk? Are any
reservations place on the coordinator disks?
There are 2 I/O paths per disk. Since there are 2 systems in the cluster, there are
2 X 2 = 4 registrations on each coordinator disk. There are no reservations put on
coordinator disks.
Solution
352 B346
vxdisk path | grep emc0_d10

sdt
ENABLED
sdw
ENABLED
emc0_d10
emc0_d10
vxfenadm -s all -f /etc/vxfentab

Device Name: /dev/vx/rdmp/emc0_d10s3
Total Number Of Keys: 4
key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,48
[Character Format]: VF000500
*
[Node Format]: Cluster ID: 5
Node ID: 0
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48
Node Name: sym1

*
Node ID: 0
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
Node Name: sym1
Node Name: sym2
Node Name: sym2
Node Name: sym1
Node Name: sym1
Node Name: sym2
Node Name: sym2
Node Name: sym1
Node Name: sym1
Node Name: sym2
Node Name: sym2
vxfenadm -r all -f /etc/vxfentab

No keys...
No keys...
No keys...
End of Solution
B347
Exercise 6: Verifying data disks for I/O fencing

In this exercise, you verify the data disks for I/O fencing.
sym1
1
From sym1:terminal1, determine the system where the appsg service group is
online. Confirm that the loopydatadg disk group is imported.
Note: The appsg service group should be online on sym1. If not, switch the
appsg service group to sym1.
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
vxdisk -o alldgs list | grep loopydatadg

emc0_dd5
auto:cdsdisk
loopydatadg01
loopydatadg
online
End of Solution
354 B348

Use the vxfenadm command to display the I/O fencing reservations and
registration on the loopy data disk. Why are there the number of reservations
and registrations as compared to the coordinator disks shown in the previous
exercise?
The data disks are part of regular volume manager disk groups and so are only
imported on one system at a time and not shared cluster volume manager disk groups
which can be imported on multiple cluster systems at one time. Therefore, only one
registration per dmp path on the node upon which the disk group is imported is created
by I/O fencing. In this lab environment, there are two registrations for the dual paths
on the one node that the disk group is imported. Only a single reservation is needed
regardless of the number of dmp paths and that reservation is managed by I/O fencing
on the node upon which the disk group is imported. The coordinator disk group is not
used as a writable disk group and so I/O fencing does not create reservations on
coordinator disks, but does manages multiple node registrations as noted earlier.
Solution
vxfenadm -s /dev/vx/rdmp/emc0_dd5
Reading SCSI Registration Keys...

Device Name: /dev/vx/rdmp/emc0_dd5
key[0]:
[Numeric Format]: 65,86,67,83,0,0,0,0
[Character Format]: AVCS
[Node Format]: Cluster ID: unknown Node ID: 0
key[1]:
[Numeric Format]: 65,86,67,83,0,0,0,0
Node Name: sym1
Node Name: sym1
vxfenadm -r /dev/vx/rdmp/emc0_dd5
Reading SCSI Reservation Information...
Key[0]:
Reservation Type: SCSI3_RESV_WRITEEXCLUSIVEREGISTRANTSONLY
Key Value [Numeric Format]: 65,86,67,83,0,0,0,0
Key Value [Character Format]: AVCS
End of Solution
B349
sym2
3
From sym2:terminal1, use the vxfenadm command to display the I/O

fencing reservations and registration on the loopy data disk even though the
disk group containing the data disk is not imported on sym2. Are there any
differences in the output for that on sym1?
There are no differences in the output.
Solution
key[0]:
[Numeric Format]: 65,86,67,83,0,0,0,0
key[1]:
[Numeric Format]: 65,86,67,83,0,0,0,0
Node Name: sym1
Node Name: sym1
356 B350

Key[0]:
End of Solution
Switch the appsg service group to sym2 and use the vxfenadm command to
display the I/O fencing reservations and registration on the loopy data disk
now that the disk group containing the data disk is imported. Are there any
differences in the output from the previous step?
The registrations and reservation change to reflect that the disk group is imported on
sym2.
Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
hagrp -state appsg

#Group
appsg
appsg
System
sym1
sym2
Value
|OFFLINE|
|ONLINE|

emc0_dd5
Attribute
State
State
auto:cdsdisk
loopydatadg01
loopydatadg
online
key[0]:
[Numeric Format]: 66,86,67,83,0,0,0,0
[Character Format]: BVCS
key[1]:
[Numeric Format]: 66,86,67,83,0,0,0,0
Node Name: sym2
Node Name: sym2

Key[0]:
Key Value [Character Format]: BVCS
End of Solution
B351
Exercise 7: Testing communication failures

In this exercise, you examine the impact of I/O fencing on private network
communication failures. Three scenarios are presented, manual concurrency
violation, response to system failure and response to interconnect failures.
sym2
1
From sym2:terminal2, use the tail -f command to monitor

/var/log/messages file.
Solution
tail -f /var/log/messages
End of Solution
sym1
2

Solution
End of Solution
358 B352
Scenario 1: Manual concurrency violation

1
From sym1:terminal1, confirm that the loopydatadg disk group is not

imported.
Solution

emc0_dd5
auto:cdsdisk
(loopydatadg) online
End of Solution

Attempt an import of the loopydatadg disk group. Then, attempt an import of

the loopydatadg disk group by clearing the host ID on all disks in the disk
group. From sym1:terminal2 review the log messages. Was the import
successful?
The data disk import should fail because the node where the disk group is not
imported does not have rights to write to the disk and therefore cannot import the disk
group and update the private region header information. The error message indicates
that data corruption from a possible concurrency violation has been prevented. The
vxdg -C import command would have been successful if I/O fencing was not
configured.
Solution
vxdg import loopydatadg

VxVM vxdg ERROR V-5-1-10978 Disk group loopydatadg: import failed:
Disk is in use by another host
vxdg -C import loopydatadg

VxVM vxdg ERROR V-5-1-10978 Disk group loopydatadg: import failed:
No valid disk found containing disk group
From sym1:terminal2, review the log messages.

...
Nov 15 10:23:46 sym1 kernel: sd 4:0:0:4: reservation conflict
Nov 15 10:23:46 sym1 kernel: sd 4:0:0:4: SCSI error: return code =
0x00000018
Nov 15 10:23:46 sym1 kernel: end_request: I/O error, dev sdy, sector 288
0x00000018
Nov 15 10:23:46 sym1 kernel: end_request: I/O error, dev sdw, sector 288
Nov 15 10:23:46 sym1 kernel: VxVM vxdmp V-5-0-0 i/o error occurred
(errno=0x20a) on dmpnode 201/0x20
Nov 15 10:23:46 sym1 kernel:
0x00000018
Nov 15 10:23:46 sym1 kernel: end_request: I/O error, dev sdw, sector 65760
0x00000018
Nov 15 10:23:46 sym1 kernel: end_request: I/O error, dev sdy, sector 65760
Nov 15 10:23:46 sym1 vxvm:vxconfigd: V-5-1-10977 da_join failed, for device
emc0_dd5: Disk write failure
B353
Nov 15 10:23:46 sym1 vxvm:vxconfigd: V-5-1-16253 Disk group import of

loopydatadg failed with error 150 - No valid disk found containing disk
group
Nov 15 10:23:46 sym1 kernel: VxVM vxdmp V-5-0-0 i/o error occurred
(errno=0x20a) on dmpnode 201/0x20
...
End of Solution
Scenario 2: Response to system failure

1
From sym1:terminal1, switch the appsg service group to sym1.

Solution
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|OFFLINE|
|ONLINE|
hagrp -state appsg

#Group
appsg
appsg
Attribute
State
State
System
sym1
sym2
Value
|ONLINE|
|OFFLINE|
End of Solution
Display a summary status of the cluster taking notice of the systems where the
service groups are online.
Solution
hastatus -sum
360 B354
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
B
B
B
B
System
ClusterService sym1
ClusterService sym2
appsg
sym1
appsg
sym2
Probed
Y
Y
Y
Y
AutoDisabled
State
N
N
N
N
ONLINE
OFFLINE
ONLINE
OFFLINE
End of Solution

Display the ports open in GAB.

Solution
gabconfig -a
===============================================================
Port a gen
Port b gen
Port h gen
End of Solution
Display the active status of the LLT links.

Solution
lltstat -nvv active

Node
* 0 sym1
1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
End of Solution
Display the I/O fencing status.

Solution
vxfenadm -d
361

================================
Fencing
Fencing
Fencing
Cluster

Mode: SCSI3
Members:
* 0 (sym1)
1 (sym2)

node
0 in state
node
1 in state
8 (running)
8 (running)
End of Solution

B355
Display the I/O fencing registrations for the coordinator disks and the
registrations and reservation on the /dev/vx/rdmp/emc0_dd5 disk
(which is part of the loopydatadg disk group).
Solution

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
362 B356

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
Node Name: sym2
Node Name: sym2
Node Name: sym1
Node Name: sym1
Node Name: sym2
Node Name: sym2
Node Name: sym1
Node Name: sym1
Node Name: sym2
Node Name: sym2
Node Name: sym1
Node Name: sym1

key[0]:
[Numeric Format]: 65,86,67,83,0,0,0,0
key[1]:
[Numeric Format]: 65,86,67,83,0,0,0,0
Node Name: sym1
Node Name: sym1
Key[0]:
End of Solution
Cause a system failure on sym1 by powering off the sym1 virtual machine.
Solution
Use the procedures outlined in Lab 2 to power off sym1.

End of Solution
sym2
8
From sym2:terminal2, observe the messages logged to the

/var/log/messages file. Which node ID was declared in trouble?
Node 0 was declared in trouble.
In which file is the node ID defined?
The node ID is defined in the /etc/llthosts file.
B357
What is the node name associated with the node ID that was declared in
trouble?
The node name associated with the node ID that was declared in trouble is sym1.
Which system won the race for the coordination points?
sym2 won the race for the coordination points.
Solution
Observe the log messages.
364 B358
...
trouble
trouble
trouble
inactive 8 sec (140088)
LLT INFO V-14-1-10205 link 1 (eth5) node 0 in

LLT INFO V-14-1-10032 link 1 (eth5) node 0

Nov 22 06:26:06 sym2 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link
1 (eth5) node 0. 4 more to go.
Nov 22 06:26:07 sym2 kernel: LLT INFO V-14-1-10032 link 1 (eth5) node 0
expired
expired
expired
Nov 22 06:26:08 sym2 kernel: GAB INFO V-15-1-20239 Initiating FFDC data
collection
Nov 22 06:26:12 sym2 Had[10909]: VCS INFO V-16-1-10077 Received new cluster
membership
Nov 22 06:26:12 sym2 kernel: GAB INFO V-15-1-20036 Port a[GAB_Control
(refcount 1)] gen
8da502 membership ;1
Nov 22 06:26:12 sym2 kernel: GAB INFO V-15-1-20036 Port h[GAB_USER_CLIENT
(refcount 0)] gen
8da505 membership ;1
Nov 22 06:26:12 sym2 Had[10909]: VCS ERROR V-16-1-10079 System sym1 (Node
'0') is in Down State - Membership: 0x2
'0') changed state from RUNNING to FAULTED
Nov 22 06:26:16 sym2 avahi-daemon[8196]: Registering new address record for
10.10.2.61 on eth0.
10.10.2.51 on eth0.
...
End of Solution
B359
From sym2:terminal1, using the date-time stamp observed in the

/var/log/messages file, examine the VCS engine log. Which service
group or groups were brought online on sym2?
The appsg and ClusterService service groups were brought online on sym2.
Which LLT network interface links were declared down?
The eth0, eth4 and eth5 LLT network interfaces were declared down.
Which file system had a log replay performed on it and which VCS resource
manages that file system?
The /loopyfs file system which is managed by the appmnt resource had a log
replay performed on it.
Solution
366 B360
...
0x2, DDNA: 0x0
2011/11/22 06:26:12 VCS ERROR V-16-1-10079 System sym1 (Node '0') is in Down
State - Membership: 0x2
2011/11/22 06:26:12 VCS ERROR V-16-1-10322 System sym1 (Node '0') changed
state from RUNNING to FAULTED
2011/11/22 06:26:12 VCS NOTICE V-16-1-10446 Group ClusterService is offline
on system sym1
2011/11/22 06:26:12 VCS NOTICE V-16-1-10446 Group appsg is offline on system
sym1
2011/11/22 06:26:12 VCS INFO V-16-1-10493 Evaluating sym1 as potential
target node for group ClusterService
2011/11/22 06:26:12 VCS INFO V-16-1-10494 System sym1 not in RUNNING state
target node for group ClusterService
target node for group appsg
2011/11/22 06:26:12 VCS INFO V-16-1-10494 System sym1 not in RUNNING state
target node for group appsg
2011/11/22 06:26:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource
webip (Owner: Unspecified, Group: ClusterService) on System sym2
notifier (Owner: Unspecified, Group: ClusterService) on System sym2
appdg (Owner: Unspecified, Group: appsg) on System sym2
appip (Owner: Unspecified, Group: appsg) on System sym2


executed
2011/11/22 06:26:13 VCS INFO V-16-1-10298 Resource notifier (Owner:
Unspecified, Group: ClusterService) is online on sym2 (VCS initiated)
2011/11/22 06:26:13 VCS NOTICE V-16-10031-1514 (sym2)
DiskGroup:appdg:online:Diskgroups will be imported without reservations.
2011/11/22 06:26:14 VCS WARNING V-16-10031-1509 (sym2)
DiskGroup:appdg:online:vxdg import succeeded on Disk Group loopydatadg.
2011/11/22 06:26:14 VCS NOTICE V-16-10031-1559 (sym2)
DiskGroup:appdg:online:Volumes in DiskGroup loopydatadg will be started
automatically as part of import command,the system level autostartvolume is
set On
DOWN eth5 DOWN eth0 DOWN.
Unspecified, Group: appsg) is online on sym2 (VCS initiated)
appvol (Owner: Unspecified, Group: appsg) on System sym2
2011/11/22 06:26:16 VCS INFO V-16-10031-12501 (sym2)
Volume:appvol:online:Volume loopydatavol is started. Any mirrors are
updated in background.
appmnt (Owner: Unspecified, Group: appsg) on System sym2
2011/11/22 06:26:19 VCS INFO V-16-2-13716 (sym2) Resource(appmnt): Output
of the completed operation (online)
==============================================
log replay in progress
replay complete - marking super-block as CLEAN
==============================================
2011/11/22 06:26:20 VCS INFO V-16-1-10298 Resource appmnt (Owner:
appfoo (Owner: Unspecified, Group: appsg) on System sym2
2011/11/22 06:26:25 VCS INFO V-16-1-10298 Resource appip (Owner:
appproc (Owner: Unspecified, Group: appsg) on System sym2
2011/11/22 06:26:29 VCS NOTICE V-16-1-10447 Group appsg is online on system
sym2
2011/11/22 06:26:33 VCS INFO V-16-1-10298 Resource webip (Owner:
Unspecified, Group: ClusterService) is online on sym2 (VCS initiated)
2011/11/22 06:26:33 VCS NOTICE V-16-1-10447 Group ClusterService is online
on system sym2
...
End of Solution
10 Display a summary status of the cluster taking notice of the system state and
systems where the service groups are online.
B361
Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
FAULTED
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
B
B
System
ClusterService
appsg
Probed
sym2
sym2
Y
Y
AutoDisabled
State
N
N
ONLINE
ONLINE
Note: sym1 shows as faulted and only state information related to sym2
is displayed.
End of Solution
11 Display the ports that are open in GAB.

Solution
gabconfig -a
===============================================================
Port a gen
cfca05 membership ;1
Port b gen
Port h gen
Note: GAB membership is only one node, the sym2 node.
End of Solution
368 B362
12 Display the configured and active status of the LLT links.

Solution

Node
0 sym1
State
Link
CONNWAIT
eth4
eth5
eth0
Status
Address
DOWN
DOWN
DOWN

* 1 sym2
OPEN
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
lltstat -nvv active

Node
* 1 sym2
State
OPEN
Link
eth4
eth5
eth0
Status
UP
UP
UP
Address
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
Note: LLT communications from sym1 are no longer active though they
are still configured.
End of Solution
13 Display the I/O fencing status.

Solution
vxfenadm -d
================================
Fencing
Fencing
Fencing
Cluster

Mode: SCSI3
Members:
* 1 (sym2)

node
1 in state
8 (running)
Note: I/O fencing is only running on one node and membership displayed
is consistent with GAB.
End of Solution
14 Display the I/O fencing registrations for the coordinator disks and the

Note: Notice that there are only two registrations for each coordinator disk.
B363
Solution

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
Node Name: sym2
Node Name: sym2
Node Name: sym2
Node Name: sym2
Node Name: sym2
Node Name: sym2
Note: Only registrations for the dual paths for sym2 are active.
b
370 B364

key[0]:
[Numeric Format]: 66,86,67,83,0,0,0,0
key[1]:
[Numeric Format]: 66,86,67,83,0,0,0,0
Node Name: sym2
Node Name: sym2
Note: Only registrations for the dual paths for sym2 are active.

Key[0]:
Note: The reservation now belongs to the sym2 system.

End of Solution
sym1
15 Power on sym1 and log in as:
Account: root
Password: veritas
sym2
16 From sym2:terminal1, use the hastatus command to observe the cluster
activity. Exit the status when sym1 is running. At the same time, from
sym2:terminal2, observe the messages in the /var/log/messages file.
371

B365
Solution
hastatus
...
------------------------------------------------------------------------sym1
INITING
sym1
REMOTE BUILD
sym1
RUNNING
webip
sym1
OFFLINE
appip
sym1
OFFLINE
...
After sym1 is running, press Ctrl-C.
End of Solution
17 Display a summary status of the cluster taking notice of the systems where the

Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
B
B
B
B
System
ClusterService
ClusterService
appsg
appsg
sym1
sym2
sym1
sym2
Probed
Y
Y
Y
Y
AutoDisabled
State
N
N
N
N
OFFLINE
ONLINE
OFFLINE
ONLINE
Note: State information is now shown for both nodes of the cluster.
372 B366
End of Solution

Solution
gabconfig -a
===============================================================

Port a gen
Port b gen
Port h gen
Note: Two node membership has been restored.

End of Solution

Solution

Node
0 sym1
* 1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
lltstat -nvv active

Node
0 sym1
* 1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
Note: All configured LLT links are now active.

End of Solution

Solution
vxfenadm -d
================================
B367
Fencing
Fencing
Fencing
Cluster

Mode: SCSI3
Members:
0 (sym1)
* 1 (sym2)

node
0 in state
node
1 in state
8 (running)
8 (running)
Note: I/O fencing is running on both nodes.

End of Solution

Note: Notice that there are now four registrations for each coordinator disk.
Solution
374 B368

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[2]:
Node Name: sym2
Node Name: sym2
Node Name: sym1
Node Name: sym1
Node Name: sym2
Node Name: sym2

*
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,48

Node ID: 0
Node Name: sym1
[Numeric Format]: 86,70,48,48,48,53,48,48

Node ID: 0
Node Name: sym1

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
Node Name: sym2
Node Name: sym2
Node Name: sym1
Node Name: sym1
Note: Registrations for dual paths for both nodes are active on the
coordinator disks.
key[0]:
[Numeric Format]: 66,86,67,83,0,0,0,0
key[1]:
[Numeric Format]: 66,86,67,83,0,0,0,0
Node Name: sym2
Node Name: sym2
Note: No change in the data disk registrations.

c
Key[0]:
B369
Note: No change in the data disk reservation.

End of Solution
sym1
22 From sym1:terminal1, switch the ClusterService and appsg service groups
to sym1.
Solution
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
Attribute
State
State
State
State
System
sym1
sym2
sym1
sym2
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
End of Solution
Scenario 3: Response to interconnect failures
376 B370
From sym1:terminal1, display a summary status of the cluster taking notice of

Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
B
B
ClusterService
ClusterService
System
sym1
sym2
Probed
Y
Y
AutoDisabled
N
N
State
ONLINE
OFFLINE

B
B
appsg
appsg
sym1
sym2
Y
Y
N
N
ONLINE
OFFLINE
End of Solution
Display the ports that are open in GAB.

Solution
gabconfig -a
===============================================================
Port a gen
Port b gen
Port h gen
End of Solution
Display the active status of the LLT links.

Solution
lltstat -nvv active

Node
* 0 sym1
1 sym2
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
End of Solution

Solution
vxfenadm -d
================================
Fencing
Fencing
Fencing
Cluster

Mode: SCSI3
Members:
* 0 (sym1)
1 (sym2)
B371

node
0 in state
node
1 in state
8 (running)
8 (running)
End of Solution

Note: Continue to monitor the /var/log/message file in
sym2: terminal.
Solution
End of Solution

/student/labs/vcs/vcs60/lltlink directory and run the
lltlink.pl script. If necessary, enter ssh as the access method. Select
option 1 to disable all private links on all nodes in the cluster.
Note: Disabling all private links simulates a failure of all LLT links at the
same time. It does this by disabling the LLT links on each cluster node.
Most likely, something external to the cluster in the physical or logical
networking would cause such a failure.
Solution
378 B372
./lltlink.pl
Checking your Operating System and setting some variables . . .
Your Operating System is . . . Linux, x86_64 . . . Done!
What is the remote communication method? rsh or ssh [ssh]:
ssh

cluster
2.
3.
4.
5.
6.
7.
8.
9.

1
Disabling
Disabling
Disabling
Disabling
Disabling
Disabling
link
link
link
link
link
link
eth4
eth5
eth0
eth4
eth5
eth0
on
on
on
on
on
on
sym1
sym1
sym1
sym2
sym2
sym2
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
done.
done.
done.
done.
done.
done.
End of Solution
Note: One of the cluster systems will panic and stop. In the remainder of this
scenario, the examples shown are for sym2 stopping and sym1 remaining
up. Your results may be reversed.
From sym1:terminal2, observe the log messages. Notice LLT link expiration,
cluster membership changes and I/O fencing coordination point race events.
Solution
Observe the log messages.

...
trouble
trouble
trouble

B373
380 B374

expired
expired
expired
collection

collection
membership
Nov 22 09:36:29 sym1 kernel: GAB INFO V-15-1-20036 Port b[VxFen (refcount
2)] gen
c4cd08 membership 0
(refcount 1)] gen
c4cd02 membership 0
(refcount 0)] gen
c4cd0b membership 0
Nov 22 09:36:29 sym1 kernel: VXFEN INFO V-11-1-80 RACER Node is: 0
Nov 22 09:36:29 sym1 kernel: VXFEN INFO V-11-1-87 Initiating Race for
Coordination Point
Nov 22 09:36:29 sym1 kernel: VXFEN INFO V-11-1-88 RACER Node won the race
for the coordination points
Nov 22 09:36:29 sym1 kernel: VXFEN INFO V-11-1-90 Sending WON_RACE
Nov 22 09:36:29 sym1 kernel: VXFEN INFO V-11-1-67 call to VM ioctl
VOL_CLEAR_PR returned non-zero
Nov 22 09:36:29 sym1 kernel: VXFEN INFO V-11-1-84 Completed Fencing
Operation.
'1') is in Down State - Membership: 0x2
'1') changed state from RUNNING to FAULTED
Nov 22 09:36:31 sym1 vxvm:vxconfigd: V-5-1-11401 : dg import with I/O fence
enabled
Nov 22 09:36:31 sym1 vxvm:vxconfigd: V-5-1-11401 loopydatadg: dg import
with I/O fence enabled
10.10.2.51 on eth0.
10.10.2.61 on eth0.
...
End of Solution
From sym1:terminal1, using the date-time stamp observed in the /var/

log/messages file, examine the VCS engine log. Notice LLT link
expiration, cluster membership changes and I/O Fencing events.
Solution
381
...
DOWN eth5 DOWN eth0 DOWN.
0x1, DDNA: 0x0
2011/11/22 09:36:29 VCS NOTICE V-16-1-10034 RECONFIG received. VCS waiting
for I/O fencing to be completed
2011/11/22 09:36:30 VCS NOTICE V-16-1-10036 I/O fencing completed
2011/11/22 09:36:30 VCS ERROR V-16-1-10079 System sym2 (Node '1') is in Down
State - Membership: 0x1
2011/11/22 09:36:30 VCS ERROR V-16-1-10322 System sym2 (Node '1') changed
state from RUNNING to FAULTED

B375
executed
...
End of Solution
Display a summary status of the cluster taking notice of the systems where the
Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
FAULTED
0
0
sym1
sym2
-- GROUP STATE
-- Group
B
B
System
Probed
ClusterService sym1
appsg
sym1
Y
Y
AutoDisabled
State
N
N
ONLINE
ONLINE
End of Solution

Solution
gabconfig -a
===============================================================
Port a gen
cfca07 membership 0
Port b gen
cfca06 membership 0
Port h gen
cfca0a membership 0
End of Solution
382 B376

Solution

Node
* 0 sym1
eth0
1 sym2
State
OPEN
UP
CONNWAIT
Link
Status
Address
eth4
UP
00:0C:29:5A:B1:16
eth5
UP
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
DOWN
DOWN
DOWN

lltstat -nvv active

Node
* 0 sym1
State
OPEN
Link
eth4
eth5
eth0
Status
Address
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
End of Solution

Solution
vxfenadm -d
================================
Fencing
Fencing
Fencing
Cluster

Mode: SCSI3
Members:
* 0 (sym1)

node
0 in state
8 (running)
End of Solution
registration and reservation on the /dev/vx/rdmp/emc0_dd5 disk (which

is part of the loopydatadg disk group).
Solution

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
Node Name: sym1
Node Name: sym1
Node Name: sym1
B377
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48

Node ID: 0

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
Node Name: sym1
Node Name: sym1
Node Name: sym1
key[0]:
[Numeric Format]: 65,86,67,83,0,0,0,0
key[1]:
[Numeric Format]: 65,86,67,83,0,0,0,0
Node Name: sym1
Node Name: sym1
Key[0]:
End of Solution
384 B378
14 Navigate to the /student/labs/vcs/vcs60/lltlink directory and
run the lltlink.pl script. Select option 3 to enable all private links on
sym1 only.
Note: This is the node that won the coordinator disk race and survived so
your system may be different.
Solution

./lltlink.pl
cluster
2.
3.
4.
5.
6.
7.
8.
9.

Enable eth0 on sym\1 only
3
End of Solution
sym2
15 Power off and power on sym2. Then log in as:
Account: root
Password: veritas
Note: This is the node that lost the coordinator disk race so your system may
be different.
Solution
Use the procedures outlined in Lab 2 to power off and power on sym2.
End of Solution
B379
16 Use the hastatus command to observe the cluster activity. Exit the status
when sym2 is running. At the same time, from sym1:terminal2, observe the
messages in the /var/log/messages file.
Note: This is the node that lost the race for the coordinator disks and has
stopped so your system may be different.
Solution
hastatus
sym2
sym2
sym2
sym2
sym2
webip
appvol
INITING
REMOTE BUILD
RUNNING
OFFLINE
OFFLINE
After sym2 is running, press Ctrl-C.
End of Solution
17 Display a summary status of the cluster taking notice of the systems where the

Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
386 B380
B
B
B
B
System
ClusterService sym1
ClusterService sym2
appsg
sym1
appsg
sym2
Probed
Y
Y
Y
Y
AutoDisabled
State
N
N
N
N
ONLINE
OFFLINE
ONLINE
OFFLINE
End of Solution

Solution
gabconfig -a
===============================================================
Port a gen
Port b gen
Port h gen
cfca0b membership 01
End of Solution

Solution

Node
* 0 sym1
1 sym2
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
lltstat -nvv active

Node
* 0 sym1
1 sym2
State
OPEN
State
OPEN
Link
Status
Address
eth4
eth5
eth0
UP
UP
UP
00:0C:29:5A:B1:16
00:0C:29:5A:B1:20
00:0C:29:5A:B1:EE
eth4
eth5
eth0
UP
UP
UP
00:0C:29:2D:77:9A
00:0C:29:2D:77:A4
00:0C:29:2D:77:72
OPEN
End of Solution

Solution
vxfenadm -d
================================
Fencing Protocol Version: 201
Fencing Mode: SCSI3
B381
Fencing SCSI3 Disk Policy: dmp

Cluster Members:
* 0 (sym1)
1 (sym2)
node
0 in state
node
1 in state
8 (running)
8 (running)
End of Solution
registration and reservation on the /dev/vx/rdmp/emc0_dd5 disk (which

is part of the loopydatadg disk group).
Note: Notice there are now four registrations on the coordinator disks. The
registration and reservation on the data disk did not change.
Solution

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
388 B382

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,49
Node Name: sym1
Node Name: sym1
Node Name: sym2
Node Name: sym2
Node Name: sym1
Node Name: sym1
Node Name: sym2

Node ID: 1

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
Node Name: sym2
Node Name: sym1
Node Name: sym1
Node Name: sym2
Node Name: sym2
key[0]:
[Numeric Format]: 65,86,67,83,0,0,0,0
key[1]:
[Numeric Format]: 65,86,67,83,0,0,0,0
Node Name: sym1
Node Name: sym1

Key[0]:
End of Solution
Note: For reference purposes, below are the sym1:/var/log/messages file and
sym1:/var/VRTSvcs/log/engine_A.log files segments during the period of
time when sym2 rejoined the cluster.
sym1:/var/log/messages
...
active
B383
active
active
Nov 22 10:11:52 sym1 kernel: AMF NOTICE V-292-1-67 Signal received while
waiting for event on reaper 'VCSProcessAgent'. Returning.
Nov 22 10:11:52 sym1 AgentFramework[15771]: VCS ERROR V-16-2-13734
Thread(4158811024) IMF thread interrupted
(refcount 1)] gen
c4cd03 membership 01
membership
(refcount 0)] gen
c4cd0c membership 01
Nov 22 10:11:52 sym1 kernel: GAB INFO V-15-1-20036 Port b[VxFen (refcount
2)] gen
c4cd09 membership 01
Nov 22 10:11:52 sym1 Had[15629]: VCS NOTICE V-16-1-10086 System sym1 (Node
'0') is in Regular Membership - Membership: 0x3
..
sym1:engine_A.log
390 B384
...
changed. Previous status = eth4 DOWN eth5 DOWN eth0 DOWN; Current status =
eth4 UP eth5 UP eth0 DOWN.
0x3, DDNA: 0x0
2011/11/22 10:11:52 VCS NOTICE V-16-1-10322 System (Node '1') changed state
from UNKNOWN to INITING
2011/11/22 10:11:52 VCS NOTICE V-16-1-10449 Group appsg autodisabled on
2011/11/22 10:11:52 VCS NOTICE V-16-1-10449 Group VCShmg autodisabled on
2011/11/22 10:11:52 VCS NOTICE V-16-1-10453 Node: 1 changed name from:
'sym2' to: 'sym2'
state from FAULTED to INITING
state from INITING to CURRENT_DISCOVER_WAIT
state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2011/11/22 10:11:52 VCS INFO V-16-1-10455 Sending snapshot to node
membership: 0x2
state from REMOTE_BUILD to RUNNING
executed /opt/VRTSvcs/bin/triggers/sysjoin sym1 RUNNING
successfully
2011/11/22 10:11:54 VCS INFO V-16-6-15023 (sym2) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_HOME=/opt/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG=/var/VRTSvcs
########## Other Environment Variables ##########
CONSOLE=/dev/pts/0
HOME=/
INIT_VERSION=sysvinit-2.86
LANG=en_US.UTF-8
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:

PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/
usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin
previous=N
PREVLEVEL=N
PWD=/var/VRTSvcs/diag/had
runlevel=5
RUNLEVEL=5
SELINUX_INIT=YES
SHLVL=5
TERM=linux
_=/usr/bin/env
executed /opt/VRTSvcs/bin/internal_triggers/dump_tunables sym2 1
successfully
2011/11/22 10:11:54 VCS INFO V-16-1-10304 Resource appip (Owner:
Unspecified, Group: appsg) is offline on sym2 (First probe)
2011/11/22 10:11:54 VCS INFO V-16-1-10304 Resource webip (Owner:
Unspecified, Group: ClusterService) is offline on sym2 (First probe)
2011/11/22 10:11:54 VCS INFO V-16-1-10304 Resource notifier (Owner:
Unspecified, Group: ClusterService) is offline on sym2 (First probe)
2011/11/22 10:11:55 VCS NOTICE V-16-1-10438 Group ClusterService has been
probed on system sym2
2011/11/22 10:11:55 VCS INFO V-16-1-10304 Resource appmnt (Owner:
2011/11/22 10:11:56 VCS NOTICE V-16-1-10438 Group appsg has been probed on
system sym2
2011/11/22 10:11:58 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on
system sym2
2011/11/22 10:11:58 VCS NOTICE V-16-1-10435 Group VCShmg will not start
automatically on System sym2 as the system is not a part of AutoStartList
attribute of the group.
changed. Previous status = eth4 UP eth5 UP eth0 DOWN; Current status = eth4
UP eth5 UP eth0 UP.
...
22 Display the states of the service groups. If any are online on sym2, switch
them to sym1.
391
Solution
hagrp -state
#Group
ClusterService
ClusterService
appsg
appsg
Attribute
State
State
State
State
System
sym1
sym2
sym1
sym2
Value
|ONLINE|
|OFFLINE|
|ONLINE|
|OFFLINE|
End of Solution

B385
23 From sym1:terminal2, terminate the log message display.

Solution
End of Solution
\
End of lab
392 B386

Lab 15: Configuring CPS-based I/O fencing

In this lab, you configure a Coordination Point server (CP server) for I/O fencing,
convert an existing disk this configured only with SCSI3-dmp based I/O fencing to
include a CP server. You then observe fencing behavior due to communication
failures from the cluster to the CP server.
A single node SFHA6.0 cluster is configured as a CP server.
Exercise 2: Configuring a single node CP server
The CP server configuration is verified.
Exercise 3: Verifying a CP server configuration

The CP server configuration is verified.
Exercise 4: Configuring for I/O fencing with a CP server
The I/O fencing configuration is changed from SCSI3 disk-based to customized,
made up of two SCSI3 disks and one CP server.
Exercise 5: Testing CP server communication failure
The failure of communications to one and then both of the CP server VIPs is
examined. I/O fencing caching is also explored.
Exercise 6: Testing CP server caching
CP server caching is examined.
Exercise 7: Configuring back to SCSI3 disk-based only I/O fencing
The west cluster is migrated back to a SCSI3 three-disk I/O configuration.
393 Lab 15: Configuring CPS-based I/O fencing
B387

394 B388
opened.
mgt
Account: student
Password: veritas
cps
Account: root
Password: veritas
sym1
Account: root
Password: veritas
sym2
Account: root
Password: veritas
terminal2.

power on and log into cps.
B389
Exercise 2: Configuring a single node CP server

In this exercise, you configure a single node SFHA6.0 cluster as a CP server.
cps
1
From cps:terminal1, display a summary status of the cluster taking notice of

Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
RUNNING
cps
-- GROUP STATE
-- Group
B
System
ClusterService
cps
Probed
AutoDisabled
State
ONLINE
End of Solution
Display the ports that are open in GAB. Which ports are open?
GAB is not configured for this one node cluster.
Solution
396 B390
gabconfig -a
GAB gabconfig ERROR V-15-2-25022 unknown error
End of Solution
Display the configured status of the LLT links. How many links are
configured?
No links are configured because LLT is not configured for this one node cluster.
Solution

LLT lltstat ERROR V-14-2-15000 open /dev/llt failed: No such file or
directory
End of Solution
Determine if the GAB and LLT configuration files are present.

Note: There should be no GAB or LLT configuration files present.
Solution
cat /etc/gabtab
cat: /etc/gabtab: No such file or directory
cat /etc/llttab
cat: /etc/llttab: No such file or directory
cat /etc/llthosts
cat: /etc/llthosts: No such file or directory
End of Solution
How are GAB and LLT set to behave at system startup and shutdown?
Neither GAB or LLT are set to start on system boot or set to stop on system shutdown.
Solution
more /etc/sysconfig/gab
# This file is sourced from /etc/init.d/gab.
#
#
#
1 = start or stop gab
#
0 = do not start or stop gab
#
GAB_START=0
GAB_STOP=0
B391
more /etc/sysconfig/llt
#
# LLT Configuration file
#
# This file is sourced from /etc/init.d/llt.
#
# LLT module load time tunables
#LLT_MAXNIDS=
#LLT_MAXPORTS=
#LLT_NQTHREAD=
#LLT_BASETIMER=
#
#
1 = start or stop llt
#
0 = do not start or stop llt
#
LLT_START=0
LLT_STOP=0
End of Solution
Use the hastop command to stop the cluster.

Solution
hastop -local
hastatus -sum
End of Solution
398 B392
Use the hastart command to start the cluster. Does the cluster startup? Look in
the engine log to determine or why not.
The cluster will not start with a simple hastart command as it requires GAB to be
configured.
Solution
hastart

hastatus -sum
tail /var/VRTSvcs/log/engine_A.log
...
2011/11/17 08:21:39 VCS
2011/11/17 08:21:39 VCS
restart
2011/11/17 08:21:39 VCS
2011/11/17 08:21:39 VCS
version=6.0.00.0
2011/11/17 08:21:39 VCS
6.0-2011-10-13-16.30.00
2011/11/17 08:21:39 VCS
2011/11/17 08:21:39 VCS
2011/11/17 08:21:39 VCS
2011/11/17 08:21:39 VCS
2011/11/17 08:21:39 VCS
INFO V-16-1-10196 Cluster logger started

NOTICE V-16-1-11027 VCS engine startup arguments=NOTICE V-16-1-11050 VCS engine version=6.0
NOTICE V-16-1-11051 VCS engine join
NOTICE V-16-1-11052 VCS engine pstamp=6.0.000.000NOTICE V-16-1-10114 Opening GAB library
NOTICE V-16-1-10619 'HAD' starting on: cps
INFO V-16-1-10125 GAB timeout set to 30000 ms
ERROR V-16-1-10116 GabHandle::open failed errno = 2
ERROR V-16-1-11033 GAB open failed. Exiting
End of Solution
Attempt to start the cluster using the -onenode switch. Does the cluster
startup?
The cluster starts up.
Solution
hastart -onenode
hastatus -sum

hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
RUNNING
cps
-- GROUP STATE
-- Group
B
ClusterService
System
cps
-- RESOURCES NOT PROBED

-- Group
Type
E
ClusterService
NIC
Probed
AutoDisabled
Resource
csgnic
State
OFFLINE
System
cps
B393
hagrp -wait ClusterService State ONLINE -sys cps
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
RUNNING
cps
-- GROUP STATE
-- Group
B
ClusterService
System
cps
Probed
AutoDisabled
State
ONLINE
End of Solution
Navigate to the /opt/VRTScps/bin directory and perform a long listing of

the contents.
Solution
cd /opt/VRTScps/bin
pwd
opt/VRTScps/bin
400 B394
ls -l
total 13536
-r-xr----- 1
-r-xr----- 1
-r-xr----- 1
-rwxr-xr-x 1
-rwxr-xr-x 1
-r-xr--r-- 1
-rwxr-xr-x 1
-r-xr-xr-x 1
-r-xr-xr-x 1
-rwxr-xr-x 1
-rwxr-xr-x 1
drwxr-xr-x 2
-rwxr-xr-x 1
-rwxr-xr-x 1
-rwxr-xr-x 1
-rwx------ 1
-r-xr----- 1
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root
root 139013 Oct 13 17:03 bmcgen

root 123295 Oct 13 17:03 bmcmap
root 119768 Oct 13 17:03 bmcread
root
88 Oct 13 17:03 client_orbsvc.conf
root
13 Oct 13 17:03 client_version
root
74099 Oct 13 17:03 configure_cps.pl
root 2997552 Oct 13 17:03 cpsadm
root
1481 Oct 13 17:03 cpsat
root
5364 Oct 13 17:03 getcps
root 283300 Oct 13 17:03 osuuidbin
root
5622 Oct 13 17:03 postinstall.sh
root
4096 Nov 17 14:20 Quorum
root 4043770 Oct 13 17:03 vssatbin
root
12431 Oct 13 17:03 vssatcps
root 939169 Oct 13 17:03 vssregctl
root 4887982 Oct 13 17:03 vxcpserv
root 127312 Oct 13 17:03 vxgettext
End of Solution
10 Run the configure_cps.pl script using the following information.
Use option 1: Configure Coordination Point Server on single node VCS

system
CP Server name: cpsclus

Two virtual IPs and NIC resources:

eth0: 10.10.2.91/255.255.255.0 (NIC resource 1)
eth1: 10.10.3.91/255.255.255.0 (NIC resource 2)
Note: It is important to associate the virtual IPs and NICs correctly.
Default port for both VIPs: 14250

No secure communications.
Absolute path of database: /etc/VRTScps/db (default location)
Do not configure NetworkHosts
Solution
./configure_cps.pl
VERITAS COORDINATION POINT SERVER CONFIGURATION UTILITY
=======================================================
Select one of the following:
[1] Configure Coordination Point Server on single node VCS system
[2] Configure Coordination Point Server on SFHA cluster
[3] Unconfigure Coordination Point Server
Enter the option :
1
Proceeding with a single node VCS configuration for CP Server..
(a) Enter the name of the CP Server :
cpsclus
(b) Enter the number of virtual IP(s) to configure :
2
Enter a valid IP address for Virtual IP - 1 which the CP Server process
should
depend on :
10.10.2.91
Enter a valid IP address for Virtual IP - 2 which the CP Server process
should
depend on :
B395
10.10.3.91
(c) Enter a port number for virtual IP 10.10.2.91 in range [49152, 65535],
or
press enter for default port (14250) :
14250
Using default port: 14250
(c) Enter a port number for virtual IP 10.10.3.91 in range [49152, 65535],
or
press enter for default port (14250) :
14250
Using default port: 14250
(d) Veritas recommends secure communication between the CP server and
application clusters. Enabling security requires Symantec Product
Authentication Service to be installed and configured on the cluster.
Do you want to enable Security for the communications? (y/n) (Default:y)
:
n
(e) CP Server uses an internal database to store the client information.
Note: As the CP Server is being configured on a single node VCS,
the database can reside on local file system.
Enter absolute path of the database (Default:/etc/VRTScps/db):
/etc/VRTScps/db
Following is the CP Server configuration information:
------------------------------------------------(a)CP Server Name: cpsclus
(b)CP Server Virtual IP(s): 10.10.2.91, 10.10.3.91
(c)CP Server Port(s): 14250, 14250
(d)CP Server Security : 0

(e)CP Server Database Dir: /etc/VRTScps/db
-------------------------------------------------
402 B396
Press b if you want to change the configuration, <enter> to continue :
Press Enter.
Successfully generated the /etc/vxcps.conf configuration file.
Successfully created directory /etc/VRTScps/db.
Configuring CP Server Service Group (CPSSG) for this cluster
---------------------------------------------Configure NIC resource
NOTE: The supplied network interface should be a public NIC
Enter how many NIC resources you want to configure (1 to 2) :

2
Enter a valid network interface on cps for NIC resource - 1 :
m eth0
Enter a valid network interface on cps for NIC resource - 2 :
eth1
Enter the NIC resource you want to associate with the virtual IP 10.10.2.91
(1 to 2) :
1
Enter the NIC resource you want to associate with the virtual IP 10.10.3.91
(1 to 2) :
2
Symantec recommends configuring NetworkHosts attribute to ensure NIC
resource to be online always.
Do you want to add NetworkHosts attribute for the NIC device eth0 on system
cps ?[y/n] :
n
Do you want to add NetworkHosts attribute for the NIC device eth1 on system
cps ?[y/n] :
n
Enter the netmask for virtual IP 10.10.2.91 :
255.255.255.0
Enter the netmask for virtual IP 10.10.3.91 :
255.255.255.0
Successfully added the CPSSG service group to VCS configuration.
Trying to bring CPSSG service group ONLINE and will wait for 80 seconds.
The Veritas Coordination Point Server has been configured on your system.
Note: It may take several moments before the CPSSG service group is
brought ONLINE and the Veritas Coordination Point Server is
configured.
End of Solution
B397
Exercise 3: Verifying a CP server configuration

In this exercise, you verify the CP server configuration.
sym1
1
From sym1:terminal1, navigate to the /etc directory and display the

contents of the cps configuration file identified during the configuration
process in the previous exercise.
Solution
cd /etc
cat vxcps.conf
## The vxcps.conf file determines the configuration for Veritas CP Server.
cps_name=cpsclus
vip=[10.10.2.91]
vip=[10.10.3.91]
port=14250
security=0
db=/etc/VRTScps/db
End of Solution
Perform a recursive long listing on the (default) path selected for the database
in the recent configuration dialog.
Solution
404B398
ls -lR /etc/VRTScps/db
/etc/VRTScps/db:
total 4
drwx------ 2 root root 4096 Nov 17 14:21 current
/etc/VRTScps/db/current:
total 16
-rw------- 1 root root 11264 Nov 17 11:09 cps_db
-rw------- 1 root root
38 Nov 17 08:50 cps_uuid
End of Solution

Perform a long recursive listing of the /var/VRTScps directory.

Solution
ls -alR /var/VRTScps
/var/VRTScps:
total 24
drwx------ 5
drwxr-xr-x 30
drwx------ 2
drwx------ 3
drwx------ 2
root
root
root
root
root
root
root
root
root
root
4096
4096
4096
4096
4096
Nov
Nov
Nov
Nov
Nov
17
17
17
17
17
08:50
08:50
08:50
08:50
08:50
.
..
diag
ldf
log
/var/VRTScps/diag:
total 8
drwx------ 2 root root 4096 Nov 17 08:50 .
drwx------ 5 root root 4096 Nov 17 08:50 ..
/var/VRTScps/ldf:
total 12
drwx------ 3 root root 4096 Nov 17 08:50 .
drwx------ 5 root root 4096 Nov 17 08:50 ..
drwx------ 2 root root 4096 Nov 17 23:51 ldf
/var/VRTScps/ldf/ldf:
total 6344
drwx------ 2 root root
4096 Nov
4096 Nov
-rw------- 1 root root 2281003 Nov
-rw------- 1 root root 4194329 Nov
17
17
18
17
23:51
08:50
06:49
23:51
.
..
cpserver_A.ldf
cpserver_B.ldf
/var/VRTScps/log:
total 7696
4096 Nov 17 08:50 .
4096 Nov 17 08:50 ..
-rw------- 1 root root 7859049 Nov 18 06:49 cpserver_A.log
End of Solution
Display the contents of the cpserver_A.log and compare and contrast to

the contents of the VCS engine log.
Note: Use the date-time stamps from the cpserever_A.log to correlate
with the VCS engine log.
Solution
more /var/VRTScps/log/cpserver_A.log
#
# Log Name:
# System:
# SysInfo:
194.el5,x86_64
# Created:
#
cpserver
cps
Linux:cps,#1 SMP Tue Mar 16 21:52:39 EDT 2010,2.6.182011/11/17 08:50:22
B399
2011-11-17 08:50:22 CPS INFO

2011-11-17 08:50:22 CPS INFO
vxcps.conf.
2011-11-17 08:50:22 CPS INFO
2011-11-17 08:50:22 CPS INFO
2011-11-17 08:50:22 CPS INFO
2011-11-17 08:50:22 CPS INFO
cpsclus
2011-11-17 08:50:22 CPS INFO
[10.10.3.91]:14250
2011-11-17 08:50:22 CPS INFO
[10.10.2.91]:14250
2011-11-17 08:50:22 CPS INFO
2011-11-17 08:50:22 CPS INFO
VRTScps/db/current/cps_db
2011-11-17 08:50:22 CPS INFO
2011-11-17 08:50:22 CPS INFO
2011-11-17 08:50:22 CPS INFO
2011-11-17 08:50:22 CPS INFO
2011-11-17 08:50:22 CPS INFO
2011-11-17 08:50:22 CPS INFO
2011-11-17 08:50:22 CPS INFO
current/cps_uuid found. Will
2011-11-17 08:50:22 CPS INFO
2011-11-17 08:50:22 CPS INFO
1dd2-11b2-a103-56875d7a7411}
2011-11-17 08:50:22 CPS INFO
11b2-a103-56875d7a7411}]
2011-11-17 08:50:22 CPS INFO
2011-11-17 08:50:22 CPS INFO
2011-11-17 08:50:22 CPS INFO
V-97-1400-69 CPS: read_config_from_file: BEGIN

V-97-1400-70 Opening configuration file /etc/
V-97-1400-74 Scanning configuration file.
V-97-1400-200 Using port 14250
V-97-1400-78 Setting security to DISABLED.
V-97-1400-373 Server name in /etc/vxcps.conf =
V-97-1400-793 IP address in /etc/vxcps.conf =
V-97-1400-793 IP address in /etc/vxcps.conf =
V-97-1400-122 Using db dir: /etc/VRTScps/db
V-97-1400-81 Reading CPS database /etc/
V-97-1400-489 Table clusters is EMPTY
V-97-1400-492 Table nodes is EMPTY
V-97-1400-491 Table nodes_SPV is EMPTY
V-97-1400-495 Table users is EMPTY
V-97-1400-718 Table user_cluster is EMPTY
V-97-1400-86 Done reading CPS cluster database
V-97-1400-362 No UUID file /etc/VRTScps/db/
create a new UUID
V-97-1400-365 CPS: create_uuid: BEGIN
V-97-1400-367 About to store UUID {3d565ca8in file /etc/VRTScps/db/current/cps_uuid
V-97-1400-388 Writing out [{3d565ca8-1dd2V-97-1400-364 CPS: create_uuid: END
V-97-1400-88 CPS: read_config_from_file: END
V-97-1400-0
2011-11-17 08:50:22 CPS INFO V-97-1400-6 Starting CP server
406 B400
2011-11-17 08:50:22 CPS INFO V-97-1400-452 Starting CP server version 2.1

2011-11-17 08:50:22 CPS NOTICE V-97-51-115 Initialized server accepting
connections on host 10.10.3.91 port 14250
2011-11-17 08:50:22 CPS INFO V-97-1400-796 CP Server listening to =
[10.10.3.91]:14250
[10.10.2.91]:14250
[127.0.0.1]:14250
2011-11-17 08:50:22 CPS INFO V-97-1400-20 Setting up security
2011-11-17 08:50:22 CPS INFO V-97-1400-382 Security is turned off,
authentication services are disabled and authorization is enabled.
...
...
2011/11/17 08:48:48 VCS INFO V-16-1-50135
makerw from localhost
2011/11/17 08:48:48 VCS INFO V-16-1-50135
add CPSSG from localhost
2011/11/17 08:48:48 VCS INFO V-16-1-50135
modify ... -add CPSSG SystemList cps 0
2011/11/17 08:48:48 VCS INFO V-16-1-50135
modify CPSSG Parallel 0 from localhost
2011/11/17 08:48:48 VCS INFO V-16-1-50135
User root fired command: haconf User root fired command: hagrp User root fired command: hagrp from localhost
User root fired command: hagrp User root fired command: hagrp -

modify CPSSG AutoStartList cps from localhost

2011/11/17 08:48:48 VCS INFO V-16-1-50135 User root fired command: hares add vxcpserv Process CPSSG from localhost
2011/11/17 08:48:49 VCS INFO V-16-1-50135 User root fired command: haconf dump -makero from localhost
2011/11/17 08:50:25 VCS INFO V-16-1-10298 Resource vxcpserv (Owner:

Unspecified, Group: CPSSG) is online on cps (VCS initiated)
2011/11/17 08:50:25 VCS NOTICE V-16-1-10447 Group CPSSG is online on system
cps
...
End of Solution
Display a summary status of the cluster taking notice of the CPSSG service
group.
Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
RUNNING
cps
-- GROUP STATE
-- Group
B
B
System
CPSSG
cps
ClusterService cps
Probed
Y
Y
AutoDisabled
N
N
State
ONLINE
ONLINE
End of Solution
Display the service group attributes for the CPSSG service group. Where will
the CPSSG fail over if it is declared faulted?
There is no other cluster system where the CPSSG service group can be restarted.
Solution
hagrp -display CPSSG | more

#Group
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
Attribute
AdministratorGroups
Administrators
Authority
AutoFailOver
AutoRestart
AutoStart
AutoStartIfPartial
AutoStartList
AutoStartPolicy
System
global
global
global
global
global
global
global
global
global
Value
0
1
1
1
1
cps
Order
B401
408 B402
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
CPSSG
ClusterList
DisableFaultMessages
Evacuate
ExtMonApp
ExtMonArgs
FailOverPolicy
FaultPropagation
Frozen
GroupOwner
GroupRecipients
Guests
IntentOnline
Load
ManageFaults
ManualOps
NumRetries
OnlineClearParent
OnlineRetryInterval
OnlineRetryLimit
OperatorGroups
Operators
Parallel
PreSwitch
PreSwitching
PreonlineTimeout
Prerequisites
PrintTree
Priority
ProPCV
Restart
SourceFile
SysDownPolicy
SystemList
SystemZones
TFrozen
Tag
TriggerEvent
TriggerPath
TriggerResFault
TriggerResRestart
TriggerResStateChange
TypeDependencies
UserIntGlobal
UserStrGlobal
AutoDisabled
ContainerInfo
Enabled
PCVAllowOnline
PreOnline
PreOnlining
Probed
ProbesPending
State
TriggersEnabled
UserIntLocal
UserStrLocal
VCSi3Info
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
cps
cps
cps
cps
cps
cps
cps
cps
cps
cps
cps
cps
cps
Manual
0
1
Priority
1
0
1
0
ALL
1
0
0
0
0
0
0
0
300
1
0
0
0
./main.cf
cps
0
1
1
0
0
0
0
1
1
0
0
1
0
|ONLINE|
0
End of Solution

List the resources and dependencies that make up the CPSSG service group.
Solution
hagrp -resources CPSSG

vxcpserv
cpsvip1
cpsvip2
quorum
cpsnic1
cpsnic2
hares -dep | grep CPSSG

CPSSG
CPSSG
CPSSG
cpsvip1
cpsvip2
vxcpserv
cpsnic1
cpsnic2
quorum
End of Solution
Display the resource attributes of the quorum resource. Which resource type is
it?
The quorum resource is of type Quorum.
Solution
hares -display quorum | more

#Resource
Attribute
quorum
Group
quorum
Type
quorum
AutoStart
quorum
Critical
quorum
Enabled
quorum
LastOnline
quorum
MonitorOnly
quorum
ResourceOwner
quorum
TriggerEvent
quorum
ArgListValues
Quorum11State12
quorum
ConfidenceLevel
quorum
ConfidenceMsg
quorum
Flags
quorum
IState
quorum
MonitorMethod
quorum
Probed
quorum
Start
quorum
State
quorum
ComputeStats
quorum
Quorum
quorum
QuorumResources
quorum
ResourceInfo
System
global
global
global
global
global
global
global
global
global
cps
cps
cps
cps
cps
cps
cps
cps
cps
global
global
global
global
Value
CPSSG
Quorum
1
1
1
cps
0
0
QuorumResources2cpsvip1cpsvip2
100
not waiting
Traditional
1
1
ONLINE
0
1
cpsvip1cpsvip2
StateValidMsgTS
B403
quorum
quorum
quorum
quorum
quorum
quorum
quorum
ResourceRecipients
TriggerPath
TriggerResRestart
ContainerInfo
MonitorTimeStats
TriggersEnabled
global
global
global
global
cps
cps
cps
0
0
TypeNameEnabled
Avg0TS
End of Solution
Display the resource attributes of the cpsvip1 and cpsvip2 resources. Verify
that the virtual IPs are configured at the operating system level.
Solution
410 B404
hares -display cpsvip1 | more

#Resource
Attribute
System
Value
cpsvip1
Group
global
CPSSG
cpsvip1
Type
global
IP
cpsvip1
AutoStart
global
1
cpsvip1
Critical
global
0
cpsvip1
Enabled
global
1
cpsvip1
LastOnline
global
cps
cpsvip1
MonitorOnly
global
0
cpsvip1
ResourceOwner
global
cpsvip1
TriggerEvent
global
0
cpsvip1
ArgListValues
cps
Device1eth0Address110.10.2.91
NetMask1255.255.255.0PrefixLen11000Options1""IPOptions1""IPRouteOptions1""
cpsvip1
ConfidenceLevel
cps
100
cpsvip1
ConfidenceMsg
cps
cpsvip1
Flags
cps
cpsvip1
IState
cps
not waiting
cpsvip1
MonitorMethod
cps
Traditional
cpsvip1
Probed
cps
1
cpsvip1
Start
cps
1
cpsvip1
State
cps
ONLINE
cpsvip1
Address
global
10.10.2.91
cpsvip1
ComputeStats
global
0
cpsvip1
IPOptions
global
cpsvip1
IPRouteOptions
global
cpsvip1
NetMask
global
255.255.255.0
cpsvip1
Options
global
cpsvip1
PrefixLen
global
1000
cpsvip1
ResourceInfo
global
StateValidMsgTS
cpsvip1
ResourceRecipients
global
cpsvip1
TriggerPath
global
cpsvip1
TriggerResRestart
global
0
cpsvip1
TriggerResStateChange global
0
cpsvip1
ContainerInfo
cps
TypeNameEnabled
cpsvip1
Device
cps
eth0
cpsvip1
MonitorTimeStats
cps
Avg0TS
cpsvip1
TriggersEnabled
cps
hares -display cpsvip2 | more

#Resource
cpsvip2
cpsvip2
cpsvip2
cpsvip2
cpsvip2
cpsvip2
Attribute
Group
Type
AutoStart
Critical
Enabled
LastOnline
System
global
global
global
global
global
global
Value
CPSSG
IP
1
0
1
cps

cpsvip2
MonitorOnly
global
0
cpsvip2
ResourceOwner
global
cpsvip2
TriggerEvent
global
0
cpsvip2
ArgListValues
cps
Device1eth1Address110.10.3.91
NetMask1255.255.255.0PrefixLen11000Options1""IPOptions1""IPRouteOptions1""
cpsvip2
ConfidenceLevel
cps
100
cpsvip2
ConfidenceMsg
cps
cpsvip2
Flags
cps
cpsvip2
IState
cps
not waiting
cpsvip2
MonitorMethod
cps
Traditional
cpsvip2
Probed
cps
1
cpsvip2
Start
cps
1
cpsvip2
State
cps
ONLINE
cpsvip2
Address
global
10.10.3.91
cpsvip2
ComputeStats
global
0
cpsvip2
IPOptions
global
cpsvip2
IPRouteOptions
global
cpsvip2
NetMask
global
255.255.255.0
cpsvip2
Options
global
cpsvip2
PrefixLen
global
1000
cpsvip2
ResourceInfo
global
StateValidMsgTS
cpsvip2
ResourceRecipients
global
cpsvip2
TriggerPath
global
cpsvip2
TriggerResRestart
global
0
cpsvip2
0
cpsvip2
ContainerInfo
cps
TypeNameEnabled
cpsvip2
Device
cps
eth1
cpsvip2
MonitorTimeStats
cps
Avg0TS
cpsvip2
TriggersEnabled
cps
411
ip addr show
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
1000
link/ether 00:0c:29:16:a5:78 brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe16:a578/64 scope link
1000
1000
link/ether 00:0c:29:16:a5:8c brd ff:ff:ff:ff:ff:ff
inet6 fe80::20c:29ff:fe16:a58c/64 scope link
1000
link/sit 0.0.0.0 brd 0.0.0.0
qlen
qlen
qlen
qlen
End of Solution
Lab 15: Configuring CPS-based I/O fencing

B405
10 Display the resource attributes for the vxcpserv resource. Which resource type
is it? Which executable does it run?

The resource is of type Process and it runs /opt/VRTScps/bin/vxcpserv.
Solution
412 B406
hares -display vxcpserv | more

#Resource
Attribute
System
Value
vxcpserv
Group
global
CPSSG
vxcpserv
Type
global
Process
vxcpserv
AutoStart
global
1
vxcpserv
Critical
global
1
vxcpserv
Enabled
global
1
vxcpserv
LastOnline
global
cps
vxcpserv
MonitorOnly
global
0
vxcpserv
ResourceOwner
global
vxcpserv
TriggerEvent
global
0
vxcpserv
ArgListValues
cps
PathName1/opt/VRTScps/bin/
vxcpservArguments1""UserName1rootPriority110PidFile1""
vxcpserv
ConfidenceLevel
cps
100
vxcpserv
ConfidenceMsg
cps
vxcpserv
Flags
cps
vxcpserv
IState
cps
not waiting
vxcpserv
MonitorMethod
cps
IMF
vxcpserv
Probed
cps
1
vxcpserv
Start
cps
1
vxcpserv
State
cps
ONLINE
vxcpserv
Arguments
global
vxcpserv
ComputeStats
global
0
vxcpserv
ConfInterval
global
30
vxcpserv
PathName
global
/opt/VRTScps/bin/vxcpserv
vxcpserv
PidFile
global
vxcpserv
Priority
global
10
vxcpserv
ResourceInfo
global
StateValidMsgTS
vxcpserv
ResourceRecipients
global
vxcpserv
RestartLimit
global
3
vxcpserv
TriggerPath
global
vxcpserv
TriggerResRestart
global
0
vxcpserv
0
vxcpserv
UserName
global
root
vxcpserv
ContainerInfo
cps
TypeNameEnabled
vxcpserv
MonitorTimeStats
cps
Avg0TS
vxcpserv
TriggersEnabled
cps
End of Solution
11 From cps:terminal2, use the tail -f command to monitor the VCS engine
log.
Solution
End of Solution

12 From cps:terminal1, display the value of the Critical and QuorumResources
resource attributes.
Solution
hares -value quorum Critical

1
hares -value quorum QuorumResources

cpsvip1 cpsvip2
End of Solution
13 Display the resource dependencies related to the quorum resource. If the
quorum resource faults, what will happen to the vxcpserv resource? What will
the status of the CPSG service group then be?
The vxcpserv resource will be taken offline. The CPSSG service group will be
declared faulted.
Solution
hares -dep | grep CPSSG

CPSSG
CPSSG
CPSSG
cpsvip1
cpsvip2
vxcpserv
cpsnic1
cpsnic2
quorum
End of Solution
14 Take the cpsvip1 resource offline, probe the quorum resource, and determine
if anything related to the quorum resource is logged in the VCS engine log.
Solution
hares -offline cpsvip1 -sys cps

Note: You can confirm the cpsvip1 resource is offline from the VCS
engine log.
B407
hares -probe quorum -sys cps

Note: The quorum resource probes online.
End of Solution
15 Take the cpsvip2 resource offline, probe the quorum resource, and determine
if anything related to the quorum resource is logged in the VCS engine log.
Solution
hares -offline cpsvip2 -sys cps

Note: You can confirm the cpsvip2 resource is offline from the VCS
engine log.
hares -probe quorum -sys cps

Note: The quorum resource faults. The vxcpserv resource is taken
offline and the CPSG service group is declared faulted.
End of Solution
16 Verify the status of the cluster. What is the effect of the cluster status on CP
server clients? Had the CP server been a multi-node cluster and the CPSG
service group appropriately configured for failover, what would have happened
to the CPSSG service group and the CP server service?
414 B408
As a single node cluster, the CP server is not available to its clients. Had this been a
multi-node cluster with the CPSSG service group appropriately configured for
failover, then the CPSSG service group would likely come online on another cluster
node and the CP server would continue to be available to its clients.

Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
RUNNING
cps
-- GROUP STATE
-- Group
B CPSSG
OFFLINE|FAULTED
B ClusterService
System
Probed
cps
cps
AutoDisabled
State
ONLINE
-- RESOURCES FAILED
-- Group
Type
Resource
System
quorum
cps
CPSSG
Quorum
End of Solution
17 Clear the fault and bring the CPSG service group back online on the cluster
node.
Solution
hagrp -state CPSSG

#Group
CPSSG
hagrp -clear CPSSG
hagrp -state CPSSG

#Group
CPSSG
Attribute
State
Attribute
State
System
cps
Value
|OFFLINE|FAULTED|
System
cps
Value
|OFFLINE|
hagrp -online CPSSG -sys cps
hagrp -wait CPSSG State ONLINE -sys cps
hagrp -state CPSSG

#Group
CPSSG
Attribute
State
System
cps
Value
|ONLINE|
End of Solution
B409
18 From cps:terminal2, terminate the monitoring of the VCS engine log.

Solution
From cps:terminal2, press Ctrl-C.
End of Solution
416 B410

Exercise 4: Configuring for I/O fencing with a CP server

In this exercise, you change the I/O fencing configuration from SCSI3 disk-based
to customized made up of two SCSI3 disks and one CP server.
sym1
1
From sym1:terminal1, display the hosts file and determine the cpsvipa and
cpsvipb virtual hostname and virtual IP information related to cpserver.
Solution
cat /etc/hosts
# Do not remove the following line, or various programs

# that require network functionality will fail
127.0.0.1
localhost.localdomain
localhost
10.10.2.3
10.10.2.4
10.10.2.5
10.10.2.6
10.10.2.7
mgt.example.com
3par.example.com
emc.example.com
vom.example.com
cps.example.com
mgt
3par
emc
vom
cps
10.10.2.11
10.10.2.12
10.10.2.13
10.10.2.14
sym1.example.com
sym2.example.com
sym3.example.com
sym4.example.com
sym1
sym2
sym3
sym4
10.10.2.51
10.10.2.52
10.10.2.61
10.10.2.71
10.10.2.90
10.10.2.91
10.10.3.91
west.example.com
east.example.com
loopyvip.example.com
nfsvip.example.com
cpsclus.example.com
cpsvipa.example.com
cpsvipb.example.com
west
east
loopyvip
nfsvip
cpsclus
cpsvipa
cpsvipb
10.10.2.101
symdb.example.com
symdb
End of Solution
Display the help information for the cpsadm command.

Solution
cpsadm -help
CPS ERROR V-97-1400-620 Please specify an action via -a option
B411
CPS INFO V-97-1400-584 Usage: cpsadm -s <cp server> -a <action> [-p <port
num>] [-c <cluster name>] [-u <uuid>][-e <username@domain>] [-f <user
role>] [-g <domain type>] [-h <hostname>] [-n <nodeid>][-o <victim's
hostname 1[,hostname 2,...]>][-v <victim's nodeid 1[,nodeid 2,...]>] [-i
<ip address> [-r <port number>]]
where:
-a is used to specify action
(one of add_clus, rm_clus, add_node, rm_node, add_user, rm_user
add_clus_to_user, rm_clus_from_user, reg_node, unreg_node,
preempt_node, list_membership, list_nodes, list_users, halt_cps,
db_snapshot, ping_cps, client_preupgrade, server_preupgrade,
list_protocols, list_version, list_ports,
add_port, rm_port)
-c is used to specify cluster name
-e is used to specify user (username@domain)
-f is used to specify user role (cps_admin or cps_operator)
-g is used to specify a domain type (nis, nisplus, ldap, or unixpwd)
-h is used to specify fully qualified hostname
-n is used to specify a host's nodeid
-o is used to specify one or more victim nodes' hostname
-p
-s
-u
-v
-i
-r
is
is
is
is
is
is
used
used
used
used
used
used
to
to
to
to
to
to
specify
specify
specify
specify
specify
specify
the port number to connect to the CP server

the CP server virtual hostname or virtual ip
UUID
one or more victim nodes' nodeids
the ip address which CPS should stop listening on
the port number, in addition to -i
End of Solution
Use the cpsadm command to ping the CP server using various combinations
of both virtual hostnames, fully qualified virtual hostnames, and virtual IPs. Do
any of the pings fail?
All virtual hostname and virtual IP combinations of cpsvipa work in the cpsadm
command.
Solution
418 B412
cpsadm -s cpsvipa -a ping_cps

CP server successfully pinged
cpsadm -s cpsvipa.example.com -a ping_cps

cpsadm -s 10.10.2.91 -a ping_cps

cpsadm -s cpsvipb -a ping_cps

cpsadm -s cpsvipb.example.com -a ping_cps


End of Solution
Display the GAB port status to confirm that Port b is active.

Solution
gabconfig -a
===============================================================
Port a gen
cfca0a membership 01
Port b gen
Port h gen
End of Solution
Display the value of the UseFence cluster attribute.

Solution

SCSI3
End of Solution

Solution
vxfenadm -d
================================
Fencing
Fencing
Fencing
Cluster

Mode: SCSI3
Members:
* 0 (sym1)
1 (sym2)
B413

node
0 in state
node
1 in state
8 (running)
8 (running)
End of Solution
Use the vxfenconfig command to list the coordination points that are in
use.
Solution
vxfenconfig -l
======================================
Single Disk Flag
: 0
Count
: 3
Disk List
Disk Name
Major
201
201
201
Minor
163
35
147
Serial Number
5123456000000008
512345600000000A
512345600000000B
Policy
dmp
dmp
dmp
End of Solution
Display the contents of the three I/O fencing configuration files located in the
/etc directory.
Solution
cat /etc/vxfendg
westfendg
420 B414
cat /etc/vxfentab
#
# /etc/vxfentab:
# VXFEN rc script from the file /etc/vxfendg.
#
cat /etc/vxfenmode
#
# vxfen_mode determines in what mode VCS I/O Fencing should work.
#

# scsi3
- use scsi3 persistent reservation disks
# customized - use script based customized fencing
# disabled
- run the driver but don't do any actual fencing
#
vxfen_mode=scsi3
#
# scsi3_disk_policy determines the way in which I/O Fencing communicates
with
# the coordination disks.
#
# dmp - use dynamic multipathing
# raw - connect to disks using the native interface
#
End of Solution
Determine the I/O fencing disk group members and whether the disk group is
imported or deported.
Solution

emc0_d10
emc0_d11
emc0_d12
auto:cdsdisk
auto:cdsdisk
auto:cdsdisk
(westfendg)
(westfendg)
(westfendg)
online
online
online
End of Solution
10 Display the I/O fencing registrations on the coordinator disks.

Solution

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
Node Name: sym1
Node Name: sym1
Node Name: sym2
Node Name: sym2
B415

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
Node Name: sym2
Node Name: sym1
Node Name: sym2
Node Name: sym1
Node Name: sym1
Node Name: sym1
Node Name: sym2
Node Name: sym2
End of Solution
11 Navigate to the /opt/VRTS/install directory and re-configure I/O
fencing by running the installsfha command using the fencing option.

Use the following information.
422 B416
Confirm that you want to reconfigure I/O fencing on the cluster.

Use option 4 for online fencing migration
Remove the emc0_d10 coordination point.
The total number of NEW coordination points is 1 (one)
The total number of NEW disk coordination points is 0 (zero).
The total number of NEW virtual IPs or fully qualified host names is 2.
The first fully qualified host name is cpvsvipa.example.com and its port is
the default, 14250.
The second fully qualified host name is cpvsvipb.example.com and its
port is also the default, 14250.
Configure the coordination point agent using the name westfensg.
Review the summary file.

Record the log file located in the /var/VRTSvcs/log/vxfen directory.

Log file:
The log file will be in the form of

installsfha-yyyymmddhhmmxxxlog.
Solution
...
Logs are being written to /var/tmp/installsfha-201111171107Hij while
installsfha
is in progress.
Checking communication on sym1 ....................................
Done
Cluster Name: west
Systems: sym1 sym2
Would you like to configure I/O fencing on the cluster? [y,n,q]

Done
Done
Fencing is already started in enabled mode, do you want to reconfigure it?
[y,n,q] (y)
y
Fencing configuration
1 ) Configure Coordination Point client based fencing
2 ) Configure disk based fencing
3 ) Configure fencing in disabled mode
4 ) Online fencing migration
Select the fencing mechanism to be configured in this Application Cluster:
[1-4,q]
B417
4
...
Online fencing migration allows you to online replace coordination points.
Installer will ask questions to get the information of the coordination
points
to be removed or added. Then it will call vxfenswap utility to commit the
coordination points change.
Warning: It may cause the whole cluster to panic if a node leaves
membership
before the coordination points change is complete. (B [0m
Select the coordination points you would like to remove from currently
configured coordination points:
1 ) emc0_d10
2 ) emc0_d11
3 ) emc0_d12
4 ) All
5 ) None
b ) Back to previous menu
Enter the options, separated by spaces: [1-5,b,q,?] (5)
1
You will be asked to give details about Coordination Point Servers/Disks to
be
used as new coordination points. Note that the installer assumes these
values to
be the identical as viewed from all the client cluster nodes.
Enter the total number of new coordination points including both
Coordination Point servers and disks: [b] (1)
1
Enter the total number of disks among these: [b] (0)
Enter the total number of Virtual IP addresses or fully qualified host name
for
the Coordination Point Server #1: [b,q,?] (1)
424B418
2
Enter the Virtual IP address or fully qualified host name #1 for the
Coordination Point Server #1: [b]
cpsvipa.example.com
Enter the port in the range [49152, 65535] which the Coordination Point
Server
cpsvipa.example.com would be listening on or simply accept the default port
suggested: [b] (14250)
14250
Enter the Virtual IP address or fully qualified host name #2 for the
Coordination Point Server #1: [b]
cpsvipb.example.com
Enter the port in the range [49152, 65535] which the Coordination Point
Server
cpsvipb.example.com would be listening on or simply accept the default port
suggested: [b] (14250)
m 14250
Coordination points verification
Current coordination points:
1. emc0_d10
2. emc0_d11
3. emc0_d12
Coordination points to be removed:
1. emc0_d10
Coordination points to be added:
1. [cpsvipa.example.com]:14250,[cpsvipb.example.com]:14250
New set of Coordination points:
2. emc0_d11
3. emc0_d12
Is this information correct? [y,n,q] (y)
y
Removing disks from disk group westfendg
Importing disk group westfendg on sym1 ............................ Done
Removing disk emc0_d10 from disk group westfendg ................. Done
Deporting the disk group westfendg on sym1 ........................ Done
Press [Enter] to continue:
Press Enter.
Updating client cluster information on Coordination Point Server
cpsvipa.example.com
Adding the client cluster to the Coordination Point Server
cpsvipa.example.com Done
Registering client node sym1 with Coordination Point Server
Adding CPClient user for communicating to Coordination Point Server
Adding cluster west to the CPClient user on Coordination Point Server
Registering client node sym2 with Coordination Point Server
B419
Adding CPClient user for communicating to Coordination Point Server

Adding cluster west to the CPClient user on Coordination Point Server
Preparing vxfenmode.test file on all systems
Preparing /etc/vxfenmode.test on system sym1 ...................... Done
Running vxfenswap. Refer to vxfenswap.log under /var/VRTSvcs/log/vxfen on
sym1 for details
Successfully completed the vxfenswap operation
Do you want to configure Coordination Point Agent on the client cluster?
[y,n,q](y)
y
Enter a non-existing name for the service group for Coordination Point
Agent: [b] (vxfen)
westfensg
Adding Coordination Point Agent via sym1 .......................... Done
page:
installsfha log files, summary file, and response file are saved at:
/opt/VRTS/install/logs/installsfha-201111171107Hij
Would you like to view the summary file? [y,n,q] (n)
426 B420
y
installsfha Summary
Old set of coordination points:
1. emc0_d10
2. emc0_d11
3. emc0_d12
2. emc0_d11
3. emc0_d12
ls -l /opt/VRTS/install/logs/installsfhayyyymmddhhmmxxx.
total 128
-rwx------ 1 root root 90674
-rwx------ 1 root root 11049
201111171107Hij.response
-rwx------ 1 root root
286
201111171107Hij.summary
-rwx------ 1 root root 15464
201111171107Hij.tunables
Nov 17 11:09 installsfha-201111171107Hij.log0

Nov 17 11:09 installsfhaNov 17 11:09
installsfha-
Nov 17 11:09
installsfha-
End of Solution

12 Perform a reverse time long listing of the vxfenswap.log files in the
/opt/VRTSvcs/log/vxfen directory and optionally display the contents

of the most recent log file.
Solution
ls -alrt /var/VRTSvcs/log/vxfen/vxfenswap.log*
vxfenswap.log.23371
vxfenswap.log.24233
End of Solution
13 Use the cpsadm command to list the membership, nodes and users now
registered with the CP server related to the west cluster. Mix up the use of both
virtual hostnames, fully qualified virtual host names, and virtual IPs related to
the CP server.
Note: The list_membership option requires an additional c or u switch.
Use the c switch and provide the proper parameter.
Solution
cpsadm -s cpsvipa -a list_membership

CPS ERROR V-97-1400-637 Please specify a cluster name using -c or a UUID
using -u
CPS INFO V-97-1400-614 Usage: cpsadm -s <cp server> -a list_membership {-c
<cluster name> | -u <uuid>}
where:
-a
-c
-s
-u
is
is
is
is
used
used
used
used
to
to
to
to
specify
specify
specify
specify
action
cluster name
the CP server virtual hostname or virtual ip
UUID
cpsadm -s cpsvipa -a list_membership -c west

List of registered nodes: 0 1
cpsadm -s cpsvipb.example.com -a list_nodes

ClusterName
Registered
===========
west
west
UUID
Hostname(Node ID)
===================================
================ ======
{445657e8-1dd2-11b2-9150-8c9a0e6defb3} sym1(0)
1
{445657e8-1dd2-11b2-9150-8c9a0e6defb3} sym2(1)
1
cpsadm -s 10.10.2.91 -a list_users
B421
Username/Domain Type
====================
=======
cpsclient@sym1/vx
Operator
cpsclient@sym2/vx
Operator
Cluster Name/UUID
==================
Role
west/{445657e8-1dd2-11b2-9150-8c9a0e6defb3}
west/{445657e8-1dd2-11b2-9150-8c9a0e6defb3}
End of Solution
14 Display the GAB port status and confirm that port b is active.
Solution
gabconfig -a
===============================================================
Port a gen
cfca0a membership 01
Port b gen
Port h gen
Note: There is no difference.

End of Solution
15 Display the value of the UseFence cluster attribute.

Solution

SCSI3
428 B422
End of Solution

Solution
vxfenadm -d
================================
Fencing Protocol Version: 201

Fencing Mode: Customized

Fencing Mechanism: cps
Cluster Members:
* 0 (sym1)
1 (sym2)
node
0 in state
node
1 in state
8 (running)
8 (running)
Note: The fencing mode is now customized the fencing mechanism is cps.
End of Solution
17 Use the vxfenconfig command to list the Coordination points that are in
use.
Solution
vxfenconfig -l
======================================
security=0
single_cp=0
[cpsvipa.example.com]:14250 {3d565ca8-1dd2-11b2-a103-56875d7a7411}
[cpsvipb.example.com]:14250 {3d565ca8-1dd2-11b2-a103-56875d7a7411}
201
131
522345600000000A
201
147
522345600000000B
dmp
dmp
Note: Two disks and two CP server VIPs are now shown.
End of Solution
18 Display the contents of the three I/O fencing configuration files located in the
/etc/directory.
Solution
cat /etc/vxfendg
westfendg

b
cat /etc/vxfentab
B423
#
# /etc/vxfentab:
# VXFEN rc script from the file /etc/vxfenmode.
#
security=0
single_cp=0
[cpsvipa.example.com]:14250
[cpsvipb.example.com]:14250
201
131
522345600000000A
201
147
522345600000000B
dmp
dmp
Note: Two disks and two CP server VIPs are now shown.
c
cat /etc/vxfenmode
vxfen_mode=customized
vxfen_mechanism=cps
security=0
cps1=[cpsvipa.example.com],[cpsvipb.example.com]
port=14250
vxfendg=westfendg
Note: This configuration file now sets the fencing mode to customize, the
fencing mechanism to cps, and now lists the CP server VIPs.
End of Solution
19 Determine the I/O fencing disk group members and whether the disk group is
imported or deported.
\
Solution
430 B424
emc0_d11
emc0_d12
auto:cdsdisk
auto:cdsdisk
(westfendg)
(westfendg)
online clone_disk
online clone_disk
Note: The two remaining I/O fencing disk are shown.

End of Solution

20 Display the I/O fencing registrations on the coordinator disks.
Note: The /etc/vxfentab file is of a different format now and since more
than just disks are listed, output errors will be introduced if this file is
used as an argument to the vxfenadm command to list fencing keys.
These errors are not harmful and can be ignored.
Optionally, copy the /etc/vxfentab file to /tmp/vxfentab and using
an editor remove all lines other than the disk lines. This file can then be used as
as the argument to the vxfenadm command to list fencing keys.
CAUTION Do not delete the non disk entries from the /etc/vxfentab
file. Never directly edit the /etc/vxfentab file.
Solution
VXFEN vxfenadm ERROR V-11-2-1116

[cpsvipa.example.com]:14250
[cpsvipb.example.com]:14250
Cannot open: security=0

Open of file failed, errno = 2
READ_KEYS failed for: security=0
Error returned
Cannot open: single_cp=0
READ_KEYS failed for: single_cp=0
Error returned
Cannot open: [cpsvipa.example.com]:14250
READ_KEYS failed for:
Error returned
Cannot open: [cpsvipb.example.com]:14250
READ_KEYS failed for:
Error returned

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
Node Name: sym1
Node Name: sym1
Node Name: sym2
Node Name: sym2
B425
Device Name: /dev/vx/rdmp/emc_d12s3

key[0]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[1]:
[Numeric Format]: 86,70,48,48,48,53,48,48
*
Node ID: 0
key[2]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
key[3]:
[Numeric Format]: 86,70,48,48,48,53,48,49
*
Node ID: 1
Node Name: sym1
Node Name: sym1
Node Name: sym2
Node Name: sym2
Note: I/O fencing reservation keys are shown for the remaining two
coordinator disks.
End of Solution
21 Display a summary status of the cluster and note the presence of the westfensg
service group. Display the service group attributes for the westfensg service
group. List the resources that are part of the westfensg service group and
display those resource attributes. Is the westfensg service group a failover or
parallel service group? What is the purpose of the resource of type Phantom?
The westfensg service group is a parallel service group. The coordpoint resource is of
type CoordPoint which is a persistent resource. Therefore, a resource of type
Phantom is needed so that VCS can properly report out service group state.
432 B426
Solution
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
B
B
B
System
ClusterService sym1
ClusterService sym2
appsg
sym1
Probed
Y
Y
Y
AutoDisabled
State
N
N
N
ONLINE
OFFLINE
ONLINE

B
B
B
appsg
westfensg
westfensg
sym2
sym1
sym2
Y
Y
Y
N
N
N
System
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
global
sym1
sym2
sym1
sym2
sym1
sym2
sym1
Value
OFFLINE
ONLINE
ONLINE
hagrp -display westfensg

#Group
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
Attribute
AdministratorGroups
Administrators
Authority
AutoFailOver
AutoRestart
AutoStart
AutoStartIfPartial
AutoStartList
AutoStartPolicy
ClusterList
DisableFaultMessages
Evacuate
ExtMonApp
ExtMonArgs
FailOverPolicy
FaultPropagation
Frozen
GroupOwner
GroupRecipients
Guests
Load
ManageFaults
ManualOps
OnlineClearParent
OnlineRetryInterval
OnlineRetryLimit
OperatorGroups
Operators
Parallel
PreSwitch
PreSwitching
PreonlineTimeout
Prerequisites
PrintTree
Priority
ProPCV
SourceFile
SysDownPolicy
SystemList
SystemZones
TFrozen
Tag
TriggerEvent
TriggerPath
TriggerResFault
TriggerResRestart
TypeDependencies
UserIntGlobal
UserStrGlobal
AutoDisabled
AutoDisabled
ContainerInfo
ContainerInfo
Enabled
Enabled
IntentOnline
0
0
1
1
1
Order
Manual
0
1
Priority
1
0
0
ALL
1
0
0
0
1
0
0
300
1
0
0
./main.cf
sym1
sym2
0
1
1
0
0
0
0
0
1
1
1
B427
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
westfensg
IntentOnline
NumRetries
NumRetries
PCVAllowOnline
PCVAllowOnline
PreOnline
PreOnline
PreOnlining
PreOnlining
Probed
Probed
ProbesPending
ProbesPending
Restart
Restart
State
State
TriggersEnabled
TriggersEnabled
UserIntLocal
UserIntLocal
UserStrLocal
UserStrLocal
VCSi3Info
VCSi3Info
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
1
0
0
1
1
0
0
0
0
1
1
0
0
0
0
|ONLINE|
|ONLINE|
0
0
hagrp -resources westfensg

coordpoint
RES_phantom_westfensg
434B428
hares -display coordpoint

#Resource
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
coordpoint
Attribute
Group
Type
AutoStart
Critical
Enabled
LastOnline
MonitorOnly
ResourceOwner
TriggerEvent
ArgListValues
ArgListValues
ConfidenceLevel
ConfidenceLevel
ConfidenceMsg
ConfidenceMsg
Flags
Flags
IState
IState
MonitorMethod
MonitorMethod
Probed
Probed
Start
Start
State
State
ComputeStats
FaultTolerance
ResourceRecipients
System
global
global
global
global
global
global
global
global
global
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
sym1
sym2
global
global
global
Value
westfensg
CoordPoint
1
1
1
sym1
0
0
FaultTolerance
FaultTolerance
100
100
1
1
0
0
not waiting
not waiting
Traditional
Traditional
1
1
0
0
ONLINE
ONLINE
0
0

coordpoint
coordpoint
coordpoint
coordpoint
Enabled
coordpoint
Enabled
coordpoint
coordpoint
coordpoint
TS
Fri
coordpoint
TS
Fri
coordpoint
coordpoint
TriggerPath
global
TriggerResRestart
global
ContainerInfo
sym1
0
0
Type
Name
ContainerInfo
Type
Name
sym2
MonitorTimeStats
sym1
MonitorTimeStats
sym2
ResourceInfo
sym1
Nov 18 09:53:30 2011
ResourceInfo
sym2
Nov 18 09:53:38 2011
TriggersEnabled
sym1
TriggersEnabled
sym2
Avg
Avg
State
0
0
Valid
TS
TS
Msg
State
Valid
Msg
hares -display RES_phantom_westfensg

#Resource
Enabled
Enabled
TS
TS
Attribute
System
Group
global
Type
global
AutoStart
global
Critical
global
Enabled
global
LastOnline
global
MonitorOnly
global
ResourceOwner
global
TriggerEvent
global
ArgListValues
sym1
ArgListValues
sym2
ConfidenceLevel
sym1
ConfidenceLevel
sym2
ConfidenceMsg
sym1
ConfidenceMsg
sym2
Flags
sym1
Flags
sym2
IState
sym1
IState
sym2
MonitorMethod
sym1
MonitorMethod
sym2
Probed
sym1
Probed
sym2
Start
sym1
Start
sym2
State
sym1
State
sym2
ComputeStats
global
ResourceRecipients
global
TriggerPath
global
TriggerResRestart
global
ContainerInfo
sym1
0
0
Type
Name
ContainerInfo
sym2
Type
Name
MonitorTimeStats
MonitorTimeStats
ResourceInfo
sym1
sym2
sym1
Avg
Avg
State
0
0
Valid
Msg
ResourceInfo
sym2
State
Valid
Msg
TriggersEnabled
TriggersEnabled
Value
westfensg
Phantom
1
1
1
sym2
0
0
""
""
100
100
not waiting
not waiting
Traditional
Traditional
1
1
1
1
ONLINE
ONLINE
0
TS
TS
sym1
sym2
End of Solution
B429
22 Examine the VCS engine log and navigate to the output generated by the
coordpoint resource as it monitors I/O fencing coordination points.

Solution
436 B430
...
2011/11/17 11:09:51 VCS INFO V-16-10061-628 (sym2)
CoordPoint:coordpoint:monitor:Successfully monitored registration with
cp [cpsvipa.example.com]:14250
2011/11/17 11:09:51 VCS INFO V-16-10061-629 (sym2)
CoordPoint:coordpoint:monitor:The agent will not log the message until
next state change in the cp: [cpsvipa.example.com]:14250
2011/11/17 11:09:51 VCS INFO V-16-10061-628 (sym2)
cp [cpsvipb.example.com]:14250
2011/11/17 11:09:51 VCS INFO V-16-10061-629 (sym2)
next state change in the cp: [cpsvipb.example.com]:14250
2011/11/17 11:09:51 VCS INFO V-16-10061-628 (sym2)
cp /dev/vx/rdmp/emc0_d11s3
2011/11/17 11:09:51 VCS INFO V-16-10061-629 (sym2)
next state change in the cp: /dev/vx/rdmp/emc0_d11s3
2011/11/17 11:09:51 VCS INFO V-16-10061-628 (sym2)
2011/11/17 11:09:51 VCS INFO V-16-10061-629 (sym2)
2011/11/17 11:09:51 VCS INFO V-16-10061-628 (sym1)
cp [cpsvipa.example.com]:14250
2011/11/17 11:09:51 VCS INFO V-16-10061-629 (sym1)
2011/11/17 11:09:51 VCS INFO V-16-10061-628 (sym1)
cp [cpsvipb.example.com]:14250
2011/11/17 11:09:51 VCS INFO V-16-10061-629 (sym1)
2011/11/17 11:09:51 VCS INFO V-16-10061-628 (sym1)
2011/11/17 11:09:51 VCS INFO V-16-10061-629 (sym1)
2011/11/17 11:09:51 VCS INFO V-16-10061-628 (sym1)
2011/11/17 11:09:51 VCS INFO V-16-10061-629 (sym1)
2011/11/17 11:14:53 VCS INFO V-16-2-13716 (sym2) Resource(coordpoint):
Output of the completed operation (info)
==============================================
Coordination Point 1: [cpsvipa.example.com]:14250
Coordination Point 2: [cpsvipb.example.com]:14250
Coordination Point 3: /dev/vx/rdmp/emc0_d11s3
the
the
the
the
the
the
the
the
the
the
the
the
the
the
the
the


==============================================
2011/11/17 11:14:54 VCS INFO V-16-2-13716 (sym1) Resource(coordpoint):
Output of the completed operation (info)
==============================================
Coordination Point 1: [cpsvipa.example.com]:14250
Coordination Point 2: [cpsvipb.example.com]:14250
...
End of Solution
23 Navigate to the /etc/vxfen.d/data directory. Perform a long listing of
the contents, then a file command on the cp_uid_db file. Finally, display its
contents.
Solution
cd /etc/vxfen.d/data
pwd
/etc/vxfen.d/data
ls -l
total 4
-rw-r----- 1 root root 160 Nov 17 11:09 cp_uid_db
file cp_uid_db
cp_uid_db: ASCII text
cat cp_uid_db
[cpsvipa.example.com]:14250 server
56875d7a7411}
[cpsvipb.example.com]:14250 server
56875d7a7411}
{3d565ca8-1dd2-11b2-a103{3d565ca8-1dd2-11b2-a103-
End of Solution
B431
Exercise 5: Testing CP server communication failure

In this exercise, you examine the failure of communications to one and then both
of the CP server VIPs. You also explore I/O fencing caching.
sym1
1
From sym1:terminal2, monitor the VCS engine log.

Solution
End of Solution
From sym1:terminal1, display the value of the FaultTolerance resource

attribute value for the coordpoint resource.
Solution
hares -value coordpoint FaultTolerance

0
End of Solution
438 B432
Use the cpsadm command to verify communications to both CP server VIPs.

Solution


End of Solution

Simulate the loss of communications via the 10.10.2.91 CP server VIP by

running the following command.
sh -x /student/labs/vcs/vcs60/cps/block.cpsvipa
Use the cpsadm command to ping the 10.10.2.91 VIP to confirm the loss of
communications.
Solution
+ ssh cps '/opt/VRTSvcs/bin/hares -offline cpsvip1 -sys cps'
+ ssh cps '/opt/VRTSvcs/bin/hares -wait cpsvip1 State OFFLINE -sys cps'

Connection timed out for host 10.10.2.91 on port 14250
Failed to create connection
Error in connecting with CPS
End of Solution
Probe the coordpoint resource on both systems and check the state of the
coordpoint resource.
Solution
hares -probe coordpoint -sys sym1
hares -state coordpoint

#Resource
coordpoint
coordpoint
Attribute
State
State
System
sym1
sym2
Value
FAULTED
FAULTED
End of Solution
From sym1:terminal2, observe the VCS engine log and determine what
happens to the coordpoint resource.
Solution
...
2011/11/18 13:33:17 VCS ERROR V-16-10061-624 (sym1)
CoordPoint:coordpoint:monitor:Unable to reach cp (or) local node not
registered on cp [cpsvipa.example.com]:14250
2011/11/18 13:33:17 VCS INFO V-16-10061-629 (sym1)
CoordPoint:coordpoint:monitor:The agent will not log the message until the
2011/11/18 13:33:18 VCS ERROR V-16-1-54031 Resource coordpoint (Owner:
Unspecified, Group: westfensg) is FAULTED on sys sym1
B433

RES_phantom_westfensg (Owner: Unspecified, Group: westfensg) on System sym1
executed /opt/VRTSvcs/bin/triggers/resfault sym1 coordpoint ONLINE
successfully
2011/11/18 13:33:19 VCS INFO V-16-1-10305 Resource RES_phantom_westfensg
(Owner: Unspecified, Group: westfensg) is offline on sym1 (VCS initiated)
2011/11/18 13:33:19 VCS NOTICE V-16-1-10235 Restart is set for group
westfensg. Group will be brought online if fault on persistent resource
clears. If group is brought online anywhere else from AutoStartList or
manually, then Restart will be reset
2011/11/18 13:33:19 VCS ERROR V-16-1-10205 Group westfensg is faulted on
system sym1
2011/11/18 13:33:19 VCS NOTICE V-16-1-10446 Group westfensg is offline on
system sym1
2011/11/18 13:33:21 VCS ERROR V-16-10061-624 (sym2)
2011/11/18 13:33:21 VCS INFO V-16-10061-629 (sym2)
successfully
system sym2
2011/11/18 13:33:23 VCS NOTICE V-16-1-10446 Group westfensg is ofline on
system sym2
...
End of Solution
440B434
From sym1:terminal1, restore communications to the 10.10.2.91 VIP by

sh -x /student/labs/vcs/vcs60/cps/unblock.all
Use the cpsadm command to ping the 10.10.2.91 VIP to confirm the
communications are restored. Then, attempt to clear the coordpoint resource
faults on both systems. Were you successful?
You cannot clear the coordpoint resource. You can probe it manually and the fault
will, in this case, auto clear. If a monitor had run before you attempted the clear then
the resource is already online.

Solution
sh -x /student/labs/vcs/vcs60/cps/unblock.all

hares -clear coordpoint -sys sym1

VCS WARNING V-\16-1-10263 Cannot clear resource of Operations type 'None'

#Resource
coordpoint
coordpoint
Attribute
State
State
System
sym1
sym2
Value
ONLINE
ONLINE
End of Solution
Change the FaultTolerance resource attribute value to 1 (one) for the

Solution
haconf -makerw
hares -modify coordpoint FaultTolerance 1

1
End of Solution
B435
Simulate the loss of communications via the 10.10.2.91 CP server VIP by

Use the cpsadm command to ping the 10.10.2.91 VIP to confirm loss of
communications.
Solution

End of Solution
10 Probe the coordpoint resource on both systems and check the state of the
Solution

#Resource
coordpoint
coordpoint
Attribute
State
State
System
sym1
sym2
Value
ONLINE
ONLINE
End of Solution
442B436
11 From sym1:terminal2, observe the VCS engine log and determine what
happens to the coordpoint resource. Is there a difference from the last time the
coordpoint resource was probed?
The difference from the last time the coordpoint resource was probed is that the
resource has not faulted.

Solution
...
2011/11/18 13:57:11 VCS ERROR V-16-10061-624 (sym2)
2011/11/18 13:57:11 VCS INFO V-16-10061-629 (sym2)
MSG_RES_PROBE coordpoint sym2 from localhost
2011/11/18 13:57:14 VCS ERROR V-16-10061-624 (sym1)
2011/11/18 13:57:14 VCS INFO V-16-10061-629 (sym1)
2011/11/18 13:57:17 VCS INFO V-16-1-55031 Resource coordpoint in online
state received recurring online message on system sym1
2011/11/18 13:57:18 VCS INFO V-16-1-55031 Resource coordpoint in online
state received recurring online message on system sym2
...
End of Solution
12 From sym1:terminal1, simulate the loss of communications via the 10.10.3.91
CP server VIP by running the following command.

sh -x /student/labs/vcs/vcs60/cps/block.cpsvipb
Use the cpsadm command to ping the 10.10.3.91 VIP to confirm loss of
communications.
Solution
sh -x /student/labs/vcs/vcs60/cps/block.cpsvipb

End of Solution
13 Probe the coordpoint resource on both systems and check the state of the
Solution
B437

#Resource
coordpoint
coordpoint
Attribute
State
State
System
sym1
sym2
Value
FAULTED
FAULTED
End of Solution
14 From sym1:terminal2, observe the VCS engine log and determine what
happens to the coordpoint resource. What happens?
The coordpoint resource has faulted and the westfensg service group has faulted on
both systems.
Solution
444B438
...
system sym2
system sym2
2011/11/18 14:01:14 VCS ERROR V-16-10061-624 (sym1)
registered on cp [cpsvipb.example.com]:14250
2011/11/18 14:01:14 VCS INFO V-16-10061-629 (sym1)
successfully
MSG_RES_PROBE coordpoint sym2 from localhost
system sym1
system sym1
...
End of Solution

15 From sym1:terminal1, restore communications to both CPS VIPs by running
the following command.

sh -x /student/labs/vcs/vcs60/unblock.all
Use the cpsadm command to ping the 10.10.2.91 and 10.10.3.91 VIPs to
confirm the communications are restored. Clear and probe the coordpoint
resource on both systems and display the state of the coordpoint resource.
Solution
sh -x /student/labs/vcs/vcs60/unblock.all



VCS WARNING V-16-1-10263 Cannot clear resource of Operations type 'None'

VCS WARNING V-16-1-10263 Cannot clear resource of Operations type 'None'
#Resource
coordpoint
coordpoint
Attribute
State
State
System
sym1
sym2
Value
ONLINE
ONLINE
End of Solution
16 From sym1:terminal2, terminate the monitoring of the VCS engine log.

Solution

End of Solution
B439
Exercise 6: Testing CP server caching

In this exercise, you examine CP server caching.
sym1
1
From sym1:terminal1, use the vxfenconfig command to confirm the

current active coordination points. Determine if the CP servers listed are in the
/etc/vxfen.d/data/cp_uid_db file.
Solution
vxfenconfig -l
======================================
security=0
single_cp=0
201
131
522345600000000A
201
147
522345600000000B
dmp
dmp
cat /etc/vxfen.d/data/cp_uid_db
56875d7a7411}
56875d7a7411}
{3d565ca8-1dd2-11b2-a103{3d565ca8-1dd2-11b2-a103-
End of Solution
446B440
Shutdown VCS on both cluster systems.

Solution
hastop -all
gabconfig -a
===============================================================
Port a gen
b4b801 membership 01
Port b gen


===============================================================
Port a gen
Port b gen
End of Solution
Shutdown the CP server cluster.

Solution
ssh cps /opt/VRTSvcs/bin/hastop -local
ssh cps /opt/VRTSvcs/bin/hastatus -sum

End of Solution
Use the service vxfen stop command to stop I/O fencing on sym1.
Solution
service vxfen stop

Stopping vxfen.. Done
gabconfig -a
===============================================================
Port a gen
vxfenadm -d
End of Solution
Use the service vxfen start command to start I/O fencing. Why does
I/O fencing startup even though the CP server is shutdown?
I/O fencing starts up because the CP Server information is cached in the

/etc/vxfen.d/data/cp_uid_db file.
B441
Solution
service vxfen start

Starting vxfen..
Loaded 2.6.18-128.el5 on kernel 2.6.18-194.el5
WARNING: No modules found for 2.6.18-194.el5, using compatible modules for
2.6.18-128.el5.
Starting vxfen.. Done
Please see the log file /var/VRTSvcs/log/vxfen/vxfen.loggabconfig a
Note: Wait a few seconds for I/O fencing to register.

b
gabconfig -a
===============================================================
Port a gen
Port b gen
vxfenadm -d
================================
Fencing
Fencing
Fencing
Cluster

Mode: Customized
Mechanism: cps
Members:
* 0 (sym1)
1 (sym2)

node
0 in state
node
1 in state
8 (running)
8 (running)
End of Solution
448B442
Remove the /etc/vxfen.d/data/cp_uid_db file and then stop and

restart I/O Fencing. Does I/O Fencing startup?
Even after an extended wait, I/O fencing does not startup.
Solution
rm /etc/vxfen.d/data/cp_uid_db
rm: remove regular file `/etc/vxfen.d/data/cp_uid_db'? y

ls -l /etc/vxfen.d/data/cp_uid_db
ls: /etc/vxfen.d/data/cp_uid_db: No such file or directory
service vxfen stop

Stopping vxfen.. Done
gabconfig -a
===============================================================
Port a gen
vxfenadm -d
service vxfen start

Starting vxfen..
Loaded 2.6.18-128.el5 on kernel 2.6.18-194.el5
WARNING: No modules found for 2.6.18-194.el5, using compatible modules for
2.6.18-128.el5.
Please see the log file /var/VRTSvcs/log/vxfen/vxfen.log
End of Solution
Examine the /var/VRTSvcs/log/vxfen/vxfen.log and determine

the difference in log entries between the last two stop-start sequences. What
can be determined?
In the first stop-start sequence, the cache file was present and so, even with the CP
server shutdown, I/O fencing could start. In the last stop-start sequence, the cache
information was missing. Since the CP server was shutdown, I/O Fencing could not
start.
Solution
Note: The important log entries are in italics.

...
Fri Nov 18 14:47:37 PST 2011 Invoked vxfen. Starting
Fri Nov 18 14:47:37 PST 2011 calling stop_fun
Fri Nov 18 14:47:37 PST 2011 stopping vxfen..
B443
Fri Nov 18 14:47:37 PST 2011 starting /sbin/vxfen-shutdown

Fri Nov 18 14:47:37 PST 2011 starting retry loop
Fri Nov 18 14:47:37 PST 2011 count is 0
Fri Nov 18 14:47:47 PST 2011 vxfenconfig -U returned 0
Fri Nov 18 14:47:47 PST 2011 vxfenconfig -U output is
Fri Nov 18 14:47:47 PST 2011 exiting normally
Fri Nov 18 14:47:47 PST 2011 /sbin/vxfen-shutdown returned with 0
Fri Nov 18 14:47:47 PST 2011 stopping vxfen.. Done
Fri Nov 18 14:47:52 PST 2011 calling mod_unload.
Fri Nov 18 14:50:35 PST 2011 starting vxfen..
Fri Nov 18 14:50:36 PST 2011 calling start_fun.
Fri Nov 18 14:50:36 PST 2011 found vxfenmode file
Fri Nov 18 14:50:36 PST 2011 calling generate_disklist
Fri Nov 18 14:50:36 PST 2011 calling /sbin/vxfen-startup in bg
Fri Nov 18 14:50:36 PST 2011 starting vxfen.. Done
Fri Nov 18 14:50:36 PST 2011 starting in vxfen-startup
Fri Nov 18 14:50:36 PST 2011 case -m :: fencing mechanism cps
Fri Nov 18 14:50:36 PST 2011 executing local_info.sh: begin
Fri Nov 18 14:50:42 PST 2011 output was security=0
single_cp=0
/dev/vx/rdmp/emc0_d11s3 5102450400000002
/dev/vx/rdmp/emc0_d12s3 5102450400000003
Fri Nov 18 14:50:42 PST 2011 executing local_info.sh: end
Fri Nov 18 14:50:42 PST 2011 calling regular vxfenconfig
Fri Nov 18 14:51:22 PST 2011 return value from above operation is 0
Fri Nov 18 14:51:22 PST 2011 output was Log Buffer: 0xffffffff88e31ac0
...
450 B444

Fri Nov 18 15:00:31 PST 2011 calling stop_fun
Fri Nov 18 15:00:31 PST 2011 stopping vxfen..
Fri Nov 18 15:00:31 PST 2011 starting /sbin/vxfen-shutdown
Fri Nov 18 15:00:31 PST 2011 starting retry loop
Fri Nov 18 15:00:31 PST 2011 count is 0
Fri Nov 18 15:00:41 PST 2011 vxfenconfig -U returned 0
Fri Nov 18 15:00:41 PST 2011 vxfenconfig -U output is
Fri Nov 18 15:00:41 PST 2011 exiting normally
Fri Nov 18 15:00:41 PST 2011 /sbin/vxfen-shutdown returned with 0
Fri Nov 18 15:00:41 PST 2011 stopping vxfen.. Done
Fri Nov 18 15:00:46 PST 2011 calling mod_unload.
Fri Nov 18 15:01:15 PST 2011 starting vxfen..
Fri Nov 18 15:01:16 PST 2011 calling start_fun.
Fri Nov 18 15:01:16 PST 2011 found vxfenmode file
Fri Nov 18 15:01:17 PST 2011 calling generate_disklist
Fri Nov 18 15:01:17 PST 2011 calling /sbin/vxfen-startup in bg
Fri Nov 18 15:01:17 PST 2011 starting vxfen.. Done
Fri Nov 18 15:01:17 PST 2011 starting in vxfen-startup
Fri Nov 18 15:01:17 PST 2011 case -m :: fencing mechanism cps
Fri Nov 18 15:01:17 PST 2011 executing local_info.sh: begin
Fri Nov 18 15:01:20 PST 2011 output was Co-ordination point
[cpsvipa.example.com]:14250 not reachable, and its UUID could not be
fetched from cache. Cannot proceed ahead.
End: local_info.sh returning FAILURE (100)
Fri Nov 18 15:01:20 PST 2011 executing local_info.sh: end
Fri Nov 18 15:01:20 PST 2011 calling regular vxfenconfig
Fri Nov 18 15:01:25 PST 2011 return value from above operation is 1
Fri Nov 18 15:01:25 PST 2011 output was VXFEN vxfenconfig ERROR V-11-2-1036
Unable to configure since VxFEN daemon failed to talk to the driver.

Log Buffer: 0xffffffff88e31ac0

VXFEN vxfenconfig NOTICE Driver will use customized fencing - mechanism cps
Fri Nov 18 15:01:25 PST 2011 exiting with 1
...
End of Solution
Startup the CP server cluster.

Solution
ssh cps /opt/VRTSvcs/bin/hastart -onenode
ssh cps /opt/VRTSvcs/bin/hagrp -wait CPSSG State

ONLINE -sys cps
ssh cps /opt/VRTSvcs/bin/hastatus -sum

-- SYSTEM STATE
-- System
State
Frozen
RUNNING
cps
-- GROUP STATE
-- Group
B
B
System
CPSSG
cps
ClusterService cps
Probed
Y
Y
AutoDisabled
State
N
N
ONLINE
ONLINE
End of Solution
Use the service vxfen start command to start I/O fencing. Confirm
that the /etc/vxfen.d/data/cp_uid_db cache file has been
re-created. Use the vxfenconfig command to confirm that the coordination
points in use.
Solution
service vxfen start

Please see the log file /var/VRTSvcs/log/vxfen/vxfen.log
Note: Wait a few seconds or repeat the gabcongfig -a command until

port b opens.
b
gabconfig -a
===============================================================
Port a gen
B445
gabconfig -a
===============================================================
Port a gen
Port b gen
vxfenadm -d
================================
Fencing
Fencing
Fencing
Cluster

Mode: Customized
Mechanism: cps
Members:
* 0 (sym1)
1 (sym2)

node
0 in state
node
1 in state
8 (running)
8 (running)
ls -l /etc/vxfen.d/data/cp_uid_db
-rw-r----- 1 root root 160 Nov 19 03:23 /etc/vxfen.d/data/cp_uid_db
cat /etc/vxfen.d/data/cp_uid_db
56875d7a7411}
56875d7a7411}
{3d565ca8-1dd2-11b2-a103{3d565ca8-1dd2-11b2-a103-
vxfenconfig -l

======================================
452 B446
security=0
single_cp=0
201
131
522345600000000A
201
147
522345600000000B
dmp
dmp
End of Solution
10 Start VCS on both cluster systems.

Solution
hastart
ssh sym2 /opt/VRTSvcs/bin/hastart

gabconfig -a
===============================================================
Port a gen
Port b gen
Port h gen
b4b80c membership 01
hastatus -sum
-- SYSTEM STATE
-- System
State
Frozen
A
A
RUNNING
RUNNING
0
0
sym1
sym2
-- GROUP STATE
-- Group
System
B
B
B
B
B
B
sym1
sym2
sym1
sym2
sym1
sym2
ClusterService
ClusterService
appsg
appsg
westfensg
westfensg
Probed
Y
Y
Y
Y
Y
Y
AutoDisabled
State
N
N
N
N
N
N
ONLINE
OFFLINE
ONLINE
OFFLINE
ONLINE
ONLINE
End of Solution
B447
Exercise 7: Configuring back to SCSI3 disk-based only I/O fencing

In this exercise, you migrate the west cluster back to a SCSI3 three-disk I/O
configuration.
sym1
1
From sym1:terminal1, navigate to the /opt/VRTS/install directory use

the installsfha command using the fencing option to re-configure I/O fencing.
Use the following information.
Select and confirm that you want to reconfigure I/O fencing on the cluster.
Select option 4 for online fencing migration
Select to remove the CPS coordination point.
The total number of NEW coordination points is 1 (one)
The total number of NEW disk coordination points is 1 (one).
The new disk is emc0_d10
The fencing mode is scsi3
Solution
Logs are being written to /var/tmp/installsfha-201111171107Hij while

installsfha
is in progress.
454B448

Done
Cluster Name: west
Systems: sym1 sym2
Would you like to configure I/O fencing on the cluster? [y,n,q]

y
Checking
Done
Checking
Checking
Checking
Done
Checking
Checking
communication on sym1 ....................................

communication on sym2 ....................................
Fencing is already started in enabled mode, do you want to reconfigure it?

[y,n,q] (y)
y
Fencing configuration
1 ) Configure Coordination Point client based fencing
2 ) Configure disk based fencing
3 ) Configure fencing in disabled mode
4 ) Online fencing migration
Select the fencing mechanism to be configured in this Application Cluster:
[1-4,q]
4
Online fencing migration allows you to online replace coordination points.
Installer will ask questions to get the information of the coordination
points
to be removed or added. Then it will call vxfenswap utility to commit the
coordination points change.
Warning: It may cause the whole cluster to panic if a node leaves
membership
before the coordination points change is complete.
Select the coordination points you would like to remove from currently
configured coordination points:
1) emc0_d11
2) emc0_d12
3) [cpsvipa.example.com]:14250,[cpsvipb.example.com]:14250
4) All
5) None
Enter the options, separated by spaces: [1-5,b,q,?] (5)
3
You will be asked to give details about Coordination Point Servers/Disks to
be
used as new coordination points. Note that the installer assumes these
values to
be the identical as viewed from all the client cluster nodes.
Enter the total number of new coordination points including both
Coordination
Point servers and disks: [b] (1)
1
Enter the total number of disks among these: [b] (0)
B449
1
List of available disks:
1) emc0_dd9
2) emc0_d10
Select 1 disk(s) as coordination points. Enter the disk options, separated
by
spaces: [1-2,b,q]
2
Coordination points verification
Current coordination points:
2. emc0_d11
3. emc0_d12
Coordination points to be removed:
Coordination points to be added:
1. emc0_d10
1. emc0_d11
2. emc0_d12
3. emc0_d10
Is this information correct? [y,n,q] (y)
456 B450
y
All the coordination points are disks. You can configure vxfen in either
scsi3
mode or customized mode, choose 'SCSI3' mode for disk based fencing or
'customized' mode for cps based fencing. Symantec recommends to configure
vxfen
in scsi3 mode.
1) SCSI3
2) Customized
Select the vxfen mode: [1-2,b,q,?] (1)
1
Importing disk group westfendg on sym1 ............................ Done
Adding disk emc0_d10 to disk group westfendg ..................... Done
Deporting the disk group westfendg on sym1 ........................ Done
Press [Enter] to continue:

Press Enter.
Running vxfenswap. Refer to vxfenswap.log under /var/VRTSvcs/log/vxfen on
sym1
for details
Cleaning up client cluster information on removed Coordination Point
servers
Cleaning up on Coordination Point server cpsvipa.example.com ...... Done
page:
installsfha log files, summary file, and response file are saved at:
/opt/VRTS/install/logs/installsfha-201111190639zCT
Would you like to view the summary file? [y,n,q] (n)
m n
End of Solution
Using the vxfenadm and vxfenconfig commands verify the I/O fencing
configuration. Optionally, examine the fencing configuration files located in
the /etc/directory.
Solution
vxfenadm -d

================================
Fencing
Fencing
Fencing
Cluster

Mode: SCSI3
Members:
* 0 (sym1)
1 (sym2)

node
0 in state
node
1 in state
8 (running)
8 (running)
vxfenconfig -l
======================================
Single Disk Flag
: 0
B451
Count
: 3
Disk List
Disk Name
Major
201
201
201
Minor
115
179
83
Serial Number
522345600000000A
522345600000000B
5223456000000009
Policy
dmp
dmp
dmp
End of Solution
End of lab
458 B452

Appendix C
Supplemental Content
459
VCS Simulator configuration

Creating a new Simulator configuration
When you add a Simulator cluster configuration, a new directory structure is
created and populated with sample files based on the criteria you specify.
The Simulator repository is in C:\Program Files\VERITAS\VCS

Simulator.
460 C2
Within the Simulator directory, each Simulator configuration has a directory

corresponding to the cluster name. When the Simulator is installed, several sample
configurations are placed in the sim_dir, such as:
SOL_ORACLE: An two-node Solaris cluster with an Oracle service group
LIN_NFS: A two-node Linux cluster with two NFS service groups
WIN_SQL_VVR_C1: One of two clusters in a global Windows cluster with a
SQL service group
When you add a cluster:
The default types.cf file corresponding to the selected platform is copied
from sim_dir\types to the
sim_dir\cluster_name\conf\config directory.
A main.cf file is created based on the
sim_dir\sample_clus\conf\config\main.cf file, using the
cluster and system names specified when adding the cluster.

Using a customized configuration

You can copy existing main.cf and types.cf files into the conf directory of
a simulated cluster to test or modify that configuration.
For example, if you have a cluster implementation and want to test fault and
failover behavior, you can create a new simulated cluster and copy the
configuration files from the actual cluster.
Note: Configuration files containing the UseFence directive cannot be used in a

Windows simulator. UseFence is discussed in the I/O Fencing lesson.
You can also create new resources or service groups within the Simulated cluster,
and then copy the modified configuration files back into a real cluster. This
requires stopping and restarting VCS, as discussed later in the course.
Using Cluster Manager with the Simulator
After the Simulator is started, you can use the Cluster Manager Java GUI to
connect to the simulated cluster. You can either launch Cluster Manager from the
Simulator GUI or start Cluster Manager from Cluster Monitor.
Note: If you see a message that the GUI is unable to connect to the Simulator,
verify that the Simulator is running and check the port number. Also, verify
that you typed the user name and password specific to that cluster
configuration.
461 Appendix C Supplemental Content
C3
462 C4

Configuring IP addresses
Configuring application IP addresses
C5
464C6

Configuring administrative IP addresses
C7
466 C8

C9
Communication configuration file examples
The llttab file
468 C10

The Solaris example is shown in the Cluster Communication lesson.
C11
Specifying the node number
470 C12

C13
How LLT and GAB are started automatically
472 C14

C15
Adding LLT links
474C16

C17
Dissimilar network interfaces example
The slide shows an example of an llttab file on Solaris where two private
network interfaces are configured on different network interfaces on each of two
nodes.
476 C18

Troubleshooting VCS
Monitoring VCS
VCS provides numerous resources you can use to gather information about the
status and operation of the cluster. These include:
VCS log files
VCS engine log file, /var/VRTSvcs/log/engine_A.log
Agent log files in /var/VRTSvcs/log
hashadow log file, /var/VRTSvcs/log/hashadow-err_A.log
System log files
The hastatus utility
Notification by way of SNMP traps and e-mail messages
Event triggers
Veritas Operations Manager
Cluster Manager Java GUI
The information sources that have not been covered elsewhere in the course are
discussed in more detail in the next sections.
C19
VCS logs
In addition to the engine_A.log primary VCS log file, VCS logs information
for had, hashadow, and all agent programs in these locations:
had: /var/VRTSvcs/log/engine_A.log
hashadow: /var/VRTSvcs/log/hashadow-err_A.log
Agent logs: /var/VRTSvcs/log/AgentName_A.log
Messages in VCS logs have a unique message identifier (UMI) built from product,
category, and message ID numbers. The Symantec Operations Readiness Tools
Web site, sort.symantec.com, provides additioinal information about
common UMI log entries.
478 C20
Each entry includes a text code indicating severity, from CRITICAL entries
indicating that immediate attention is required, to INFO entries with status
information.
The log entries are categorized as follows:
CRITICAL: VCS internal message requiring immediate attention. Contact
Support.
ERROR: Messages indicating errors and exceptions
WARNING: Messages indicating warnings
NOTICE: Messages indicating normal operations
INFO: Informational messages from agents
Entries with CRITICAL and ERROR severity levels indicate problems that require
troubleshooting.
Changing the log level and file size

You can change the amount of information logged by agents for resources being
monitored. The log level is controlled by the LogDbg resource type attribute.
Changing this value affects all resources of that type. Possible values are any
combination of DBG_1 to DBG_21 log tags, and one of the severity levels
represented by DBG_AGINFO, DBG_AGDEBUG, and DBG_AGTRACE.
Use the hatype command to change the LogDbg value and then write the inmemory configuration to disk to save the results in the types.cf file.
Note: Only increase agent log levels when you experience problems. The
performance impacts and disk space usage can be substantial.
You can also change the size of the log file from the default of 32 MB. The
minimum log size is 64k and maximum is 128MB, specified in bytes. This is
defined in the LogFileSize cluster attribute.
When the engine_A.log file reaches the maximum size, the file is renamed
engine_B.log and a new engine_A.log file is created. The next time the
engine_A.log file reaches the maximum size, the engine_B.log file is
renamed to engine_C.log. Upon subsequent rollovers, the contents of the
engine_C.log file are overwritten and are no longer available.
C21
Using the Symantec technical Web sites

The Symantec SPRT Web site contains product and patch information, a
searchable knowledge base of technical notes, access to product-specific news
groups and e-mail notification services, and other information about contacting
technical support staff. You can access SORT from http://
sort.symantec.com.
The Symantec Connect Web site provides a portal for accessing technical
resources, such as product documentation, software, technical articles, and
discussion groups. You can access Symantec Connect from http://
www.symantec.com/connect.
480 C22

Troubleshooting guide
A VCS problem is typically one of three types:
Cluster communication
VCS engine startup
Service groups, resources, or agents
Procedure overview
To start troubleshooting, determine which type of problem is occurring based on
the information displayed by hastatus -sum output.
Cluster communication problems are indicated by the message:
Cannot connect to server -- Retry Later
VCS engine startup problems are indicated by systems in the
STALE_ADMIN_WAIT or ADMIN_WAIT state.
Other problems are indicated when the VCS engine, LLT, and GAB are all
running on all systems, but service groups or resources are in an unexpected
state.
C23
The hastatus display uses categories to indicate the status of different types of
cluster objects. Examples are shown in the following table.
Systems
Service groups
Failed resources
Resources not probed
482 C24

Using the troubleshooting job aid

You can use the troubleshooting job aid provided on the Symantec Education Web
site to assist you in solving problems in your VCS environment.
http://www.symantec.com/training/
To navigate to the job aids, click:

Browse Training by Product A-Z > Cluster Server > Course Supplementary
Materials
This lesson provides the background for understanding the root causes of
problems, as well as the effects of applying solutions described in the job aid.
Ensure that you understand the consequences of the commands and methods you
use for troubleshooting when using the job aid.
C25
Making backups
484C26
Include VCS configuration information in your regular backup scheme. Consider

archiving these types of files and directories:
/etc/VRTSvcs/conf/config/types.cf and any other custom types
files
/etc/VRTSvcs/conf/config/main.cf
main.cmd, generated by:
hacf -cftocmd /etc/VRTSvcs/conf/config
and
hacf -verify
LLT and GAB configuration files in /etc:
llthosts
llttab (unique on each system)
gabtab
Customized triggers in /opt/VRTSvcs/bin/triggers
Agents in /opt/VRTSvcs/bin and /opt/VRTSagents/ha/bin
Note: The VCS software distribution includes the VRTSspt package, which
provides vxexplorer, a tool for gathering system information that may
be needed by Support to troubleshoot a problem.


Appendixes B and C

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Appendixes B and C

Hochgeladen von

Copyright:

Verfügbare Formate

Veritas Cluster

Server 6.0 for UNIX:

LEAD SUBJECT MATTER

Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Appendix B: Lab Solutions

Copyright 2012 Symantec Corporation. All rights reserved.

Exercise 1: Checking lab prerequisites .............................................. B-168

Veritas Cluster Server 6.0 for UNIX: Install and Configure

Exercise 7: Configuring back to SCSI3 disk-based only I/O fencing .. B-448

Copyright 2012 Symantec Corporation. All rights reserved.

Appendix C: Supplemental Content

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Cluster Server 6.0 for UNIX: Install and Configure

Copyright 2012 Symantec Corporation. All rights reserved.

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Cluster Server 6.0 for UNIX: Install and Configure

Copyright 2012 Symantec Corporation. All rights reserved.

The eight virtual machines are as follows:

Copyright 2012 Symantec Corporation. All rights reserved.

cps: A RHEL55 VM with 1 GB RAM and one CPU, which has a

Veritas Cluster Server 6.0 for UNIX: Administration

Virtual machine configuration

Copyright 2012 Symantec Corporation. All rights reserved.

Fully Qualified Host Name

Veritas Cluster Manager Java Console

Virtual machine configuration (continued)

Fully Qualified Host Name

SFHA 5.1 SP1 (VCS 5.1

SFHA 5.1 SP1 (VCS 5.1

SFHA 5.1 SP1 (VCS 5.1

SFHA 5.1 SP1 (VCS 5.1

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Cluster Server 6.0 for UNIX: Administration

Virtual machine account information

Copyright 2012 Symantec Corporation. All rights reserved.

Accessing virtual machines

Lab 2: Intro Lab A: Using VMware Workstation in a classroom

Exercise 2: Logging on to virtual machines (VMware Workstation)

Exercise 3: Adjusting the VMware view (VMware Workstation)

Veritas Cluster Server 6.0 for UNIX: Administration

Copyright 2012 Symantec Corporation. All rights reserved.

Lab 2: Intro Lab A: Using VMware Workstation in a classroom

VMware Workstation in a physical classroom environment

Copyright 2012 Symantec Corporation. All rights reserved.

Veritas Cluster Server 6.0 for UNIX: Administration

VMware Workstation interface

Copyright 2012 Symantec Corporation. All rights reserved.

Lab 2: Intro Lab A: Using VMware Workstation in a classroom

Exercise 1: Starting virtual machines (VMware Workstation)

On the desktop of your host system, double-click the Load Environment

Copyright 2012 Symantec Corporation. All rights reserved.

From the VMware Workstation application, choose a virtual machine by

Notice which virtual network (VMnetx) to which each network adapter is

Copyright 2012 Symantec Corporation. All rights reserved.

Lab 2: Intro Lab A: Using VMware Workstation in a classroom

Start the mgt virtual machine.

From the VMware Workstation application, click the mgt tab.

From the toolbar, click the green Power On button.

Copyright 2012 Symantec Corporation. All rights reserved.

Start the vom virtual machine.

Veritas Cluster Server 6.0 for UNIX: Administration

From the VMware Workstation application, click the vom tab.