Au 546 Xstudhint

V3.1.0.
cover
Front cover
HACMP System
Administration I: Planning and
Implementation
(Course Code AU54)
Student Exercises
with Hints
ERC 6.0
IBM Certified Course Material

Student Exercises with Hints
Trademarks
IBM is a registered trademark of International Business Machines Corporation.
The following are trademarks of International Business Machines Corporation in the United
States, or other countries, or both:
AIX AIX5L DB2
DB2 Universal Database eServer Enterprise Storage Server
HACMP iSeries NetView
Notes POWER4+ POWER5
pSeries Redbooks RS/6000
SP Tivoli TotalStorage
WebSphere
Think is a trademark or registered trademark of Lenovo in the United States, other
countries, or both.
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the
United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft
Corporation in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other
countries.
Other company, product and service names may be trademarks or service marks of others.
October 2005 Edition
The information contained in this document has not been submitted to any formal IBM test and is distributed on an as is basis without
any warranty either express or implied. The use of this information or the implementation of any of these techniques is a customer
responsibility and depends on the customers ability to evaluate and integrate them into the customers operational environment. While
each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will
result elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk.
Copyright International Business Machines Corporation 1998, 2005. All rights reserved.
This document may not be reproduced in whole or in part without the prior written permission of IBM.
Note to U.S. Government Users Documentation related to restricted rights Use, duplication or disclosure is subject to restrictions
set forth in GSA ADP Schedule Contract with IBM Corp.
V3.1.0.1
TOC Contents
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
Exercise Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Exercise 1. Cluster Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1
Exercise 2. Cluster Planning: Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1
Exercise 3. Cluster Planning: Network Setup and Test . . . . . . . . . . . . . . . . . . . . . . 3-1
Exercise 4. Application Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1
Exercise 5. HACMP Software Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1
Exercise 6. Initial Cluster Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-1
Exercise 7. Basic HACMP Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1
Exercise 8. Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-1
Exercise 9. Network File System (NFS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-1
Exercise 10. HACMP Problem Determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-1
Exercise 11. Migration (optional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-1
Appendix A. Cluster Diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1
Copyright IBM Corp. 1998, 2005 Contents iii

Course materials may not be reproduced in whole or in part
without the prior written permission of IBM.
iv HACMP Implementation Copyright IBM Corp. 1998, 2005

V3.1.0.1
TMK Trademarks
The reader should recognize that the following terms, which appear in the content of this
training document, are official trademarks of IBM or other companies:
IBM is a registered trademark of International Business Machines Corporation.
The following are trademarks of International Business Machines Corporation in the United
States, or other countries, or both:
AIX AIX 5L DB2
DB2 Universal Database eServer Enterprise Storage Server
HACMP iSeries NetView
Notes POWER4+ POWER5
pSeries Redbooks RS/6000
SP Tivoli TotalStorage
WebSphere
Think is a trademark or registered trademark of Lenovo in the United States, other
countries, or both.
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the
United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft
Corporation in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other
countries.
Other company, product and service names may be trademarks or service marks of others.
Copyright IBM Corp. 1998, 2005 Trademarks v

vi HACMP Implementation Copyright IBM Corp. 1998, 2005

V3.1.0.1
pref Exercise Description

Exercise instructions - This section contains what it is you are going to
accomplish. See the Lab Setup Guide and the course Lab Guide for
instructions and details pertaining to the labs. You are given the
opportunity to work through each exercise given what you learned in
the unit presentation.
Copyright IBM Corp. 1998, 2005 Exercise Description vii

Text highlighting
The following text highlighting conventions are used throughout this book:
Bold Identifies file names, file paths, directories, user names and
principals.
Italics Identifies links to Web sites, publication titles, is used where the
word or phrase is meant to stand out from the surrounding text,
and identifies parameters whose actual names or values are to
be supplied by the user.
Monospace Identifies attributes, variables, file listings, SMIT menus, code
examples of text similar to what you might see displayed,
examples of portions of program code similar to what you might
write as a programmer, and messages from the system.
Monospace bold Identifies commands, daemons, menu paths, and what the user
would enter in examples of commands and SMIT menus.
viii HACMP Implementation Copyright IBM Corp. 1998, 2005

V3.1.0.1
EXempty Exercise 1. Cluster Planning

(with Hints)
What This Exercise Is About

This exercise is a high-level design of a cluster. It is scenario-based.
This reinforces the lecture material.
What You Should Be Able to Do

At the end of the exercise, you should be able to:
Create a high-level design of a cluster
Interpret the business requirements into a diagram suitable for
creating further HACMP configuration information
Describe how HACMP will assist in creating the design
Verify that your lab environment is set up
Introduction
The scenario that the exercises are based on is a company which is
amalgamating its computer sites to a single location. It is intended to
consolidate computer sites from two cities into one situated roughly in
the middle of the original two. The case study has been designed
around five randomly chosen countries in the world. These countries
and city configurations have been tested in our environment but we
offer the choice to use your own. On to the scenario.
You will also checkout your lab environment in this exercise.
Required Materials
Your imagination.
Paper or a section of a white board.
Lab setup
Copyright IBM Corp. 1998, 2005 Exercise 1. Cluster Planning 1-1

Exercise Instructions with Hints

Preface
(All hints are marked by a sign)
A company had two machines located in two different cities. One computer was located in
Halifax and one in Calgary. The systems have been named by their city designation to
keep them straight. The corporate Web server resides on the Halifax computer. Currently
the systems are running on internal disks, on systems too small for the task. As part of the
consolidation new systems will be used. These new systems are to be configured in such a
manner as to provide as close to 7x24x365 access to the Web server as possible with
pSeries technology. Also corporate marketing is about to launch a major initiative to
promote a second application that they want to be highly available. The corporate
management has insisted that this project be successful, and so both machines would be
moved to the new computer center in Regina to resolve all issues of reliability that thus far
have caused great corporate embarrassment. All eyes are focused on this project.
A project briefing has been called by the senior executive to get an overview of how the
funds for the equipment are applied.
Your task is to prepare for that meeting to present a high level solution.
Part 1 Checkout your lab environment

There will be a more extensive treatment of naming conventions when beginning exercise
2. In this exercise you will only document the lab information that you will need and verify
that the lab environment is setup.
The lab exercise instructions and hints are, in general, designed for labs running with a
Windows PC accessing a p520 with LPARs for nodes. The following diagram shows the
p520 environment (see next page):
1-2 HACMP Implementation Copyright IBM Corp. 1998, 2005

V3.1.0.1
EXempty
p520 Lab Environment
hdisk0 VIO L1 L2 CL L3
en0
rootvg en1
hdisk1
en2
vdisks
ent
en2 = console session
HMC
Intel workstations (windows/xp, websm, vncviewer)
Copyright IBM Corporation 2005
In this environment:
a. The console is now an IP address that will NOT be used by HACMP
b. An X session (full screen) is via the VNC software
c. All adapters (Ethernet and disk) in the HACMP nodes are virtual.
If you are not running in this environment please ask the instructor for any additional
checking such as access to the shared disks. There may be some differences if not running
in this environment

__ 1. You will need the following information from your instructor

- team number _______________________
You will substitute this number for the # symbol in the lab exercises (this number is
actually not required for the p520 environment but may be convenient to use).
- root password ______________________
- IP addresses for the console access (p520 environment)
node1 _________________________
node2 _________________________
client __________________________
node3 _________________________
- location of HACMP code _(/ha53/hacode)_______________________
__ 2. Verify that you have the vncviewer icon on your pc (p520 environment)
__ 3. Start a console (login) session as root to your node1
__ 4. Check to see that you have 2 unused Ethernet interfaces. You will configure them in
exercise 3.
netstat -in (see console address)
lscfg | grep Eth (see 2 additional interfaces)
__ 5. Check to see that you have shared disks on node1 and node2. In the p520 standard
environment you should see a rootvg hdisk and 4 unassigned hdisks. Assign pvids if
necessary
On first node: lspv then chdev -a pv=yes -l hdisk_
On second node: BE CAREFUL rmdev -dl hdisk _ (NOT rootvg) followed
by cfgmgr. The pvids of the shared disks on both nodes should be the same
__ 6. Check to see that you have access to HACMP code (/ha53/hacode/ha_cd)
cd to location of HACMP code and verify
__ 7. Disable DNS on your nodes. You will use only an /etc/hosts file.
# mv /etc/resolv.conf /etc/resolv.conf.bk
__ 8. Check that Java14.sdk is installed your nodes and client using the lslpp -L
command
__ 9. Run the command updtvpkg on your nodes and client to ensure X11 libraries are
known to RPM.
updtvpkg
__ 10. Check COMMUNITY public in /etc/snmpdv3.conf on your nodes and client to ensure
that the line not commented out.
cat /etc/snmpdv3.conf | grep -i community
__ 11. Install JUST the PDF documentation and the Online Planning Worksheets filesets
(HACMP cluster filesets do not have to be installed to do this). Do this on both nodes
or HACMP will complain about it later.

V3.1.0.1
EXempty # smit install_all

Select cluster.doc.en_US.es (just PDF) and cluster.es.worksheets
Accept license
__ 12. Use an X session on your cluster node and start up the online planning worksheets.
Note: In the p520 environment X session means using the vncviewer client. In this
case the format for the destination is IP address followed by a colon followed by the
number 1 (that is, 9.47.88.5:1)
cd /usr/es/sbin/cluster/worksheets
Execute worksheets
__ 13. Browse the planning worksheets to verify that the application comes up and to see
that you can provide information about cluster, nodes, network, storage, and
applications. You need not provide any specific information at this time.
Part 2 Create a cluster diagram based on introduction lecture

__ 1. Draw each of the computer systems as described.
__ 2. Add the applications to the nodes.
__ 3. Add a network connection to each system for access to the outside world.
__ 4. Combine the services of the existing networks resulting in a single network.
__ 5. Add new disks to your drawing, showing SAN or direct connectivity cabling
connections.
__ 6. Make the disks highly available--RAID/mirror, multi-pathing with redundant adapters.
__ 7. Ensure there is no single point of failure for network, application, and data access.
__ 8. Indicate how the resources fail and recover.
__ 9. Make the diagram simple to understand.
END OF LAB


V3.1.0.1
EXempty Exercise 2. Cluster Planning: Storage

(with Hints)

This exercise is going to build on the high-level design. You continue
to build upon the cluster. The next step is to document your hardware
to create an inventory of materials to work with. You use a cluster
planning worksheets and a generic cluster diagram to design and
document your cluster storage resources. The design is based on
either the country scenario provided or the high-level design you
created in the prior exercise.

At the end of the lab, you should be able to:
Create component worksheets and a diagram showing your cluster
topology in detail
Identify the hardware configuration of the classroom equipment
Introduction
Lab scenario
The systems provided are to be the new systems for the consolidated
computer center. You must prepare these systems to be the
replacements of the production systems.
In this exercise, you will be examining and configuring the shared
storage for your high availability solution.
You will be using the On Line Planning Worksheets (OLPW)
application to document your configuration. However, since it is useful
to have a paper copy of the cluster configuration to refer to as you are
doing the exercises and since most classrooms do not have a printer,
you will also be filling out the Cluster Component Worksheets on
page 16.
Lab equipment
There may be differences between the documentation and the real
machines in the classroom environment. The systems have been
selected to provide a consistent experience but a variety of equipment
may be used.
Copyright IBM Corp. 1998, 2005 Exercise 2. Cluster Planning: Storage 2-1
Please check with your instructor if you have any questions.

- Remote equipment
The lab equipment may be located remotely, which means you will be
connecting to the lab systems from computers in the classroom.
- Nodes
Each team will have three systems: two nodes and a client system. In
many cases, the cluster nodes will be LPARs in a POWER5-based
system. There may be a fourth system for a later exercise to add a
third node.
- Networks
The communications path will usually be Ethernet. There will be two
network interfaces for cluster communications (usually en0 and en1).
If your lab equipment is remote, there will be a third interface for
communication to the remote classroom (usually en2). If you are using
POWER5-based LPARs, these will be virtual Ethernet adapters.
There must also be a non-IP serial network -- either RS232, target
mode SSA or heartbeat over disk. If you are using POWER5-based
LPARs, you will be using heartbeat over disk using a disk in an
enhanced concurrent volume group.
You will configure the networks in the next exercise.
- Storage
Each node will have at least one disk for private storage (rootvg). At
least two shared disks will be available for shared storage between the
two nodes. SCSI, Fiber Channel, SSA or virtual SCSI disks may be
used for these shared disks. If you are using POWER5-based LPARs,
the disks will be virtual SCSI disks. If not using POWER5-based
LPARs, you will likely be using SSA disks.
These lab exercises do not require you to create mirrored logical
volumes. Normally, unless you were using a hardware RAID disk
array, you would create mirrored LVs. If you have four shared disks
available in your lab environment, you can create mirrors, if desired.
However this is not required for the exercises.
- Cluster diagram
The picture below shows the generic cluster for this lab.

V3.1.0.1
EXempty
LAB Reference Cluster

classroom
client
systems en0 en1
Network = ________________ (netmask = 255.255.255.0)
en2 en1 en0 en0 en1 en2

Home Node Name = Home Node Name =
Resource Group = Resource Group =
Startup Policy = Startup Policy =
Fallover Policy = Fallover Policy =
Fallback Policy = Fallback Policy =
Service IP Label = Service IP Label =
Application server = Application server =
en0 (non-service) = en0 (non-service) =
en2 (classroom) = en2 (classroom) =
Label = Label =
non-IP
Device = Device =
network
rootvg VG = rootvg
disk =
disk =
VG =
disk =
disk =
Figure 2-1. Lab Reference Cluster AU546.0
Naming Conventions
In this exercise guide, we suggest a naming convention for the class
exercises. Your instructor may ask that you follow a different
convention. For the sake of your instructors sanity, please follow the
naming conventions suggested in the exercises or the naming
convention he or she requests that you use.
- Cluster name
You will use your team number to create your cluster name in the
OLPW application:
team#
where # is replaced by your team number.
- Network names and addresses
Each team will have two cluster nodes and one client system. We use
a naming convention based on three Canadian cities and your team
number:
t#hallifax node1
t#toronto node2
t#regina client
where # is replaced by your team number. The team number is used
for network names and addresses to avoid confusion and duplicate IP
addresses. You will configure the network in the next exercise.
- Storage names
In this exercise, you will be discovering and configuring the shared
storage for two applications: appA and appB.
Each application will be associated with a resource group. The
resource groups will include the following storage components, which
will be configured in this exercise:
appA_group appB_group
Volume group appA_vg appB_vg
FS logical volume appA_lv appB_lv
FS log LV appA_log_lv appB_log_lv
FS mount point /appA_fs /appB_fs

V3.1.0.1
EXempty Exercise Instructions with Hints

Preface
Two versions of the exercise are available: with hints and without hints. Please feel free
to use which ever version is most useful for your learning. Or switch back and forth
between them.
Also, dont forget to ask the instructor if you have questions.
All hints are marked by a sign.
Hints show commands and output for team2. Make the appropriate substitutions for
your team.
Part 1: Examine the Cluster Environment and Complete the Cluster

Component Worksheets with Storage Information
Using the cluster component worksheets (located at the end of this exercise), record the
information as listed in the following steps.
__ 1. Write down your team number here: ____. In these lab exercises you must replace
the symbol # with your team number unless otherwise noted.
__ 2. Log in as root on both of your cluster nodes. The root password will be provided by
your instructor.
__ 3. Identify and record in the cluster components worksheet the device names and
location codes of the disk adapters.
Note: If you are using virtual disks, some of the commands are a little different than
you may be used to. If you are unfamiliar with the virtual SCSI environment, please
see the with Hints version of these instructions.
SSA DISKS
# lsdev -Cc adapter
VIRTUAL SCSI DISKS
For virtual SCSI disks, lsdev does not show the location code. Use lscfg.
t2halifax # lscfg | grep vscsi
* vscsi1 U9111.520.10EE61E-V7-C6-T1 Virtual SCSI Client Adapter
t2toronto # lscfg | grep vscsi

Vx identifies the LPAR ID, in this example, t2halifax is LPAR 7 and

t2toronto is LPAR 8. Both LPARs are in the same p520 frame:
U9111.520.10EE61E.
Cx identifies the virtual I/O slot, in this example, the two virtual SCSI
adapters are in virtual slots 5 and 6 on each node.
location codes of the shared disks (hdisks and pdisk for SSA disks).
Note: The external disks may not have PVIDs on them at this time. Check with your
instructor to find out which disks are private and which are shared. Usually hdisk0 is
private and hdisk1-4 are shared.
SSA DISKS
# lspv
# lsdev -Cc disk
# smitty ssadlog
VIRTUAL SCSI DISKS
t2halifax # lscfg | grep disk
* hdisk4 U9111.520.10EE61E-V7-C6-T1-L840000000000 Virtual SCSI
Disk Drive
Disk Drive
Disk Drive
Disk Drive
Disk Drive
Vx and Cx are the IPAR ID and virtual slot numbers. From this we can see
that:
hdisk1-4 are on the same adapter (which we know from the previous step
to be vscsi1) and
hdisk0 is on vscsi0.
location codes of the internal disks.
SSA DISKS
# lsdev -Cc disk
# lsdev -Cc pdisk
VIRTUAL SCSI DISKS
# lscfg | grep disk

V3.1.0.1
EXempty __ 6. The storage needs to be divided into two volume groups. Size of the volume groups
is not important. In a real environment, disks should be mirrored and quorum issues
addressed. Here the emphasis is on the operation of HACMP not how the storage is
organized. You should have four disks so feel free to set up a mirror on one of the
volume groups. Different methods of configuring the disks are going to be used
through out the exercises. Decide on the organization but only create the volume
groups when directed to.
__ 7. Identify and update the cluster planning worksheets with the names of two shared
volume groups. Use the following names:
appA_vg
appB_vg
__ 8. Identify and update the cluster component worksheets with the LV component
names to have a shared file system in each of the two volume groups. Select names
for the logical volumes, file systems and file system logs. Use the following names.
(Instructions and hints assume JFS file systems, however feel free to use JFS2 if
you prefer.)
data LVs:
appA_lv
appB_lv
jfslog LVs:
appA_log_lv
appB_log_lv
file systems:
/appA_fs
/appB_fs
__ 9. Identify and update the cluster components worksheets the name for your cluster
(any string without spaces, up to 32 characters) using the following or choose your
own.
- cluster name is team#
__ 10. Now, add the networking information to the generic cluster diagram of your cluster.
This diagram can be found in Appendix A (there are two blank ones after the filled in
one. One is for in class and the other is to take home). On the other hand you may
want to just compare the information on your component worksheets to the filled in
worksheet at the beginning of Appendix A.
Only fill in what you know -- the LVM information-- at the bottom of the diagram.
__ 11. Next, you will enter the storage information into the Online Planning Worksheets
(OLPW) application.
Open a graphical connection to one of your nodes and start the Online Planning
Worksheets application.
# /usr/es/sbin/cluster/worksheets/worksheets
__ 12. Enter Cluster Name, Company and Author and click Apply. Use any values you
like.
__ 13. You will need to navigate to Resource Associations to add the storage information,
but first you must add the two nodes and the two resource groups to the OLPW.
__ 14. Select the Requisite tab at the bottom left. Select Nodes and Communication
Paths in the navigation pane. Enter each node name and click Add. Leave the
Communication Path to the Node: field blank for now.

V3.1.0.1
EXempty __ 15. Select Resource Groups in the navigation pane on the left side of the OLPW. Enter
the name of one resource group. Allow the Startup Behavior, Fallover Behavior
and Fallback Behavior fields to default for now. Select one node from Available
Nodes and click >> to add it to Participating Nodes. Repeat for the second node.
Click Add. (You may need to scroll down to find the Add button.)
Repeat for the second resource group.
__ 16. Scroll down to display the two resource groups.
__ 17. Now, you are ready to add the storage information for appA_group. Select
Resource Associations in the navigation pane. Select appA_group from the
Resource Group pull down list. Select HDisks from the Resource Type pull down
list. Type hdisk1 (or whatever your first shared disk name is) in the Resource Name
field and click Add. Repeat for the other disks. Select Volume Group
(non-concurrent) in the Resource Type pull down; enter appA_vg in the
Resource Name field and click Add. Select File System in the Resource Type pull
down; enter appA_lv in the Resource Name field and click Add.

V3.1.0.1
EXempty __ 18. Repeat for appB_group.

__ 19. Save your changes.
__ a. Since you have not yet entered the network information, turn off the Validate
When Saving feature. Select the Settings menu, click Validate When Saving
to uncheck this setting.
__ b. Select File -> Save. Navigate to /home/au54. Accept the default file name
(cluster_name.haw) and click Save.
__ 20. Close the OLPW application. Click File -> Exit.
Part 2: Configure Volume Group for AppA Shared Data

With your cluster planning sheets available, begin the configuration of the volume group for
AppA. We will configure the VG for AppB in a later exercise.
__ 21. Log in to both nodes as root.
__ 22. Verify that both nodes have the same number of disks.
# lspv
# lsdev -Cc disk
__ 23. Identify the internal and shared disks from the cluster worksheet. These disks might
or might not have PVIDs on them.
If they match between the two systems, then you can skip to step __ 28. on page 12.
If they dont match, or if you are not sure, continue with the next step.
SSA DISKS
# lscfg | grep hdisk
VIRTUAL SCSI DISKS
# lspv
__ 24. On both systems, delete only the external hdisks.
Hint assumes hdisk0 is the only internal disk and that hdisk1-4 are shared
disks.
# rmdev -l hdisk1 -d
ALTERNATE METHOD:
# for i in 1 2 3 4
do
>rmdev -l hdisk$i -d
>done
__ 25. On one system, add all of disks back in and give them PVIDs.
Hint assumes hdisk0 is the only internal disk and that hdisk1-4 are shared
disks.
# cfgmgr
# chdev -a pv=yes -l hdisk1
__ 26. On the other system, add the disks back in.

cfgmgr
__ 27. Verify that the disk names and PVIDs agree on both nodes.
On both nodes:
lspv
__ 28. Find a VG major number not used on either node __________.
on both nodes execute:
# lvlstmajor
(the numbers listed are available)

V3.1.0.1
EXempty __ 29. Go to your t#halifax node. Create an Enhanced Concurrent Volume Group called
appA_vg. This will be the volume group for appAs shared data. Remember to use
the major number you discovered in the previous step.
In order to create an Enhanced Concurrent Volume Group, the bos.clvm.enh fileset
must be installed. Check to see if it is installed.
# lslpp -L | grep lvm
If not, you will need to install it. In order to install it, you must install one of the
following prerequisites:
rsct.basic.rte 2.2.0.0
cluster.es.server.rte 4.4.0.0
Since we are not ready to install the HACMP (cluster.es.*) filesets yet, well install
the rsct.basic fileset for now. Install rsct.basic from the /ha53/hacode/ex2/rsct
directory and bos.clvm from the /ha53/hacode/ex2/clvm directory.
# smitty install_latest
Fill in /ha53/hacode/ex2/rsct in the
INPUT device / directory for software field.
Use F4 in the Software to install field and select the rsct.basic filesets.
# smitty install_latest
Fill in /ha53/hacode/ex2/clvm in the
INPUT device / directory for software field.
Use F4 in the Software to install field and select the bos.clvm filesets.
# smitty vg
(or smitty -> system storage -> logical volume manager ->
volume groups)
Select Add a volume group)
Select Add an Original Volume Group
Fill in Volume Group name field
Set partition size (if default value wont work) <-- pop-up list is available
Using F4 select a single physical volume in the PHYSICAL VOLUME names
field (you may select another volume if you have three or more shared disks)
Set Activate volume group AUTOMATICALLY at system restart? to no
Set the Volume group MAJOR NUMBER to the value discovered in the previous
step
Set Create VG Concurrent Capable? to enhanced concurrent
__ 30. Vary on the volume group and create a jfslog logical volume with a name of
appA_log_lv. The type is to be jfslog. Only one LP is required.
# varyonvg appA_vg
# smitty lv
Select Add a Logical Volume
Select the volume group just created, a list is provided (F4)
Fill in the NAME = appA_log_lv,
number of lp = 1,
TYPE = jfslog
Allow other fields to default, or fill in your favorite options
__ 31. Format the jfslog logical volume.
# logform /dev/appA_log_lv
answer y to delete all the information
__ 32. Create a logical volume for data called appA_lv. Set number of LPs to 10.
# smitty lv
Add the logical volume
Enter your favorite options
__ 33. Create a file system on appA_lv using the Add a Journaled File System on a
previously defined logical volume. The mount point should be /appA_fs and
the file system should not be automatically activated on system restart.
# smitty jfs
-> Add a Journaled File System on a previously defined logical
volume
-> Add a Standard Journaled File System
Fill the required fields, making sure to:
use the F4 list to select the logical volume just created
Use /appA_fs as the mount point
Set mount automatically at system restart to No
__ 34. Verify the file system can be mounted manually.
# mount /appA_fs
__ 35. Check the correct log file (appA_log_lv) is active. If you have a loglv00 then you
might not have formatted the jfs log before you created the jfs.
# lsvg -l appA_vg
ALTERNATE METHOD: mount command with no arguments
__ 36. Umount the file system.
# umount /appA_fs
__ 37. Vary off the volume group.

V3.1.0.1
EXempty # varyoffvg appA_vg

__ 38. On your t#toronto node, import the volume group using the major number, hdisk
and volume group information. The VG name must be the same as the system it
was created on.
# importvg -V <major#> -y appA_vg hdiskX
__ 39. Set the autovaryon flag to off for the volume group. You will need to manually vary
it on first.
# varyonvg appA_vg
# chvg -an appA_vg
__ 40. Mount the file system on the second node and verify it functions.
# mount /appA_fs
__ 41. Check the correct log file (appA_log_lv) is active.
# lsvg -l appA_vg
ALTERNATE METHOD: mount command with no arguments.
__ 42. Unmount the file system.
# umount /appA_fs
__ 43. Vary off the volume group.
# varyoffvg appA_vg
Cluster Component Worksheets

Table 1: Non-shared Components Worksheet: FIRST Node
Non-shared Components Description Value
Node Name
***Network Adapter*** **IBM 10/100 Mbps Ethernet *** entX 10-60

Network Adapter IF1
Network Adapter IF2
Network Adapter IF3
Network Adapter IF4
***Ext. Disk Adapter*** ***SSA 160 Serial RAID Adapter*** ssaX 10-90
Ext. Disk Adapter 1
Ext. Disk Adapter 2
***Serial port*** ***Standard I/O Serial Port*** saX 01-S1

Serial port 1
Serial port 2
***TTY device *** ***Asynchronous Terminal*** ttyX 01-S1-00-00

TTY device 1
TTY device 2
***Internal Disk *** 16 Bit LVD SCSI Disk Drive hdiskX 10-80-00-4,0
Internal Disk 1
Internal Disk 2
Internal Disk 3
Persistent Address
***IF IP Label/address*** myname 192.168.#x.yy

IF1 IP Label/address
TMSSA device ------------------N/A-----------------------

V3.1.0.1
EXempty
Table 2: Non-shared Components Worksheet: SECOND Node
Component Description Value
Node Name
***Network Adapter*** **IBM 10/100 Mbps Ethernet *** entX 10-60

Network Adapter IF1
Network Adapter IF2
Network Adapter IF3
Network Adapter IF4
***Ext. Disk Adapter*** ***SSA 160 Serial RAID Adapter*** ssaX 10-90
Ext. Disk Adapter 1
Ext. Disk Adapter 2
***Serial port*** ***Standard I/O Serial Port*** saX 01-S1

Serial port 1
Serial port 2
***TTY device *** ***Asynchronous Terminal*** ttyX 01-S1-00-00

TTY device 1
TTY device 2
***Internal Disk *** 16 Bit LVD SCSI Disk Drive hdiskX 10-80-00-4,0
Internal Disk 1
Internal Disk 2
Internal Disk 3
Persistent Address ------------------N/A-----------------------
***IF IP Label/address*** myname# 192.168.#x.yy

TMSSA device ------------------N/A-----------------------
Table 3: Shared Components Worksheet

Component Description Value
Cluster Name
Cluster ID
Cluster Subnet mask
Network Name
Network Name
***Shared Disk *** *P1.1-I3/Q1-W4AC50A84400D* hdiskX pdiskY

Shared Disk 1
Shared Disk 2
Shared Disk 3
Shared Disk 4
Shared vg 1
Shared jfs log 1
Shared jfs lv 1
Shared file system 1
-mount point
Shared vg 2
Shared jfs log 2
Shared jfs lv 2
Shared file system 2
-mount point
ALIAS: myname# 192.168.#x.yy

Service Label/address
REPLACEMENT node1:
Hardware Address ---------------------N/A-----------------
REPLACEMENT node2:
Hardware Address ---------------------N/A-----------------

V3.1.0.1
EXempty END OF LAB

V3.1.0.1
EXempty Exercise 3. Cluster Planning: Network Setup and

Test
(with Hints)

This exercise guides you through the set up and testing of the
networks required for HACMP.

Configure TPC/IP networking suitable for HACMP
Test the TCP/IP configuration
Configure non-IP communications for HACMP
Test the non-IP communications for HACMP
Configure and test name resolution and authentication
Introduction
This section establishes the communication networks required for
implementing HACMP. Networking is an important component of
HACMP, so all related aspects are configured and tested.
Lab equipment
There may be differences between the documentation and the real
machines in the classroom environment. The systems have been
selected to provide a consistent experience but a variety of equipment
may be used.
Please check with your instructor if you have any questions.
- Networks
The communications path will usually be Ethernet. There will be two
network interfaces for cluster communications (usually en0 and en1).
If your lab equipment is remote, there will be a third interface for
communication to the remote classroom (usually en2).
If you are using POWER5-based LPARs, these will be virtual Ethernet
adapters.
Copyright IBM Corp. 1998, 2005 Exercise 3. Cluster Planning: Network Setup and Test 3-1
There must also be a non-IP serial network -- either RS232, target

mode SSA or heartbeat over disk. If you are using POWER5-based
LPARs, you will likely be using heartbeat over disk using a disk in an
enhanced concurrent volume group.
LAB Reference Cluster

classroom
client
systems en0 en1
Network = ________________ (netmask = 255.255.255.0)
en2 en1 en0 en0 en1 en2

Home Node Name = Home Node Name =
Resource Group = Resource Group =
Startup Policy = Startup Policy =
Fallover Policy = Fallover Policy =
Fallback Policy = Fallback Policy =
Service IP Label = Service IP Label =
Application server = Application server =
en2 (classroom) = en2 (classroom) =
Label = Label =
non-IP
Device = Device =
network
rootvg VG = rootvg
disk =
disk =
VG =
disk =
disk =
Figure 3-2. Lab Reference Cluster AU546.0
- Network names and addresses

Each team will have two cluster nodes and one client system. We use
a naming convention based on three Canadian cities and your team
number:
t#hallifax node1
t#toronto node2
t#regina client
where # is replaced by your team number. The team number is used
for network names and addresses to avoid confusion and duplicate IP
addresses.

V3.1.0.1
EXempty The team number will be used in IP addresses to create separate

subnets for each team. See Appendix A for a summary of the
suggested addressing scheme.
Required Materials
Cluster Planning Worksheets and cluster diagram from exercise 2.

Preface
Two versions of the exercise are available: with hints and without hints. Please feel free
to use which ever version is most useful for your learning. Or switch back and forth
between them.
Also, dont forget to ask the instructor if you have questions.
Hints show commands and output for team2. Make the appropriate substitutions for
your team.
Part 1: Examine the Cluster Environment and Complete the Cluster

Component Worksheets with Networking Information
__ 1. Identify and record in the cluster components worksheet the device names (entX)
and location codes of the network adapters.
Note: If you are using virtual Ethernet, some of the commands are a little different
than you may be used to. If you are unfamiliar with the virtual Ethernet environment,
please see the with Hints version of these instructions.
PHYSICAL NETWORK ADAPTERS
# lsdev -Cc adapter
# lsdev -Cc if
VIRTUAL NETWORK ADAPTERS
# lsdev -Cc adapter
# lscfg | grep Ether
# lsdev -Cc if
As noted in the last exercise: Vx identifies the LPAR ID, Cx identifies the
virtual I/O slot.
__ 2. Identify, and record in the cluster components worksheets, the IP labels/addresses
that will be used for the communication interfaces using the following guidelines:
Ensure that the logical subnet rules are complied with.
- We will NOT be using heartbeating over alias
- We WILL be using IPAT via alias
This means that each communication interface must be on a different logical
subnet.
Ensure that all of the communication interfaces have the same subnet mask.

V3.1.0.1
EXempty You can use the following names/addresses or select your own.
Item Address
netmask 255.255.255.0
t#halifax-if1 192.168.#1.1
t#halifax-if2 192.168.#2.1
t#toronto-if1 192.168.#1.2
t#toronto-if2 192.168.#2.2
If you choose to use your own please verify with the instructor that they will not
conflict with another team.
__ 3. Identify, and record in the cluster components worksheets, the IP labels/addresses
for the service and persistent labels using the following guidelines.
Ensure that the logical subnet rules are complied with.
- We will NOT be using heartbeating over alias
- We WILL be using IPAT via alias
This means that the service labels/addresses and persistent labels/addresses
may not be on the same logical subnet as any one of the communication
interfaces.
The following names/addresses may be used or select your own. If you choose
to use your own please verify with the instructor that they will not conflict with
another team.
Item Address
t#halifax-per 192.168.#3.1
t#toronto-per 192.168.#3.2
t#appA-svc 192.168.#3.10
t#appB-svc 192.168.#3.11
__ 4. The IP network name is generated by HACMP.
Note: If you use the Two-Node Cluster Configuration Assistant (as we will do in a
later exercise), HACMP generates the network name. It is also possible to set the
network name using the HACMP Extended Configuration SMIT menus or the
Online Planning Worksheets. We will use the HACMP generated network name in
these exercises.
__ 5. Identify and update the cluster components worksheet with the device names and
location codes of the serial ports.
Note: If you are using POWER5 LPARs for your nodes, you will probably not have a
serial port available for you non-IP network. You will be using disk heartbeat using
an enhanced concurrent volume group.
# lsdev -C | grep - i serial
__ 6. The non-IP network name is generated by HACMP.
__ 7. At this point in time most of the names for the various cluster components should
have been selected and populated on the cluster component worksheets. It is
important to have a clear picture of the various names of these components as you
progress through the exercises.
__ 8. Now add the networking information to the generic cluster diagram of your cluster.
This diagram can be found in Appendix A (there are two blank ones after the filled in
one. One is for in class and the other is to take home). On the other hand you may
want to just compare the information on your component worksheets to the filled in
worksheet at the beginning of Appendix A.
Only fill in what you know -- cluster name, node names (t#halifax, t#toronto), and
IP information at the top.
__ 9. Next you will enter the network information into the Online Planning Worksheets
(OLPW) application.
Open a graphical connection to one of your nodes and start the Online Planning
Worksheets application.
# /usr/es/sbin/cluster/worksheets/worksheets
__ 10. Open the worksheet file you saved from the previous exercise.
Select File -> Open
Navigate to /home/au54 and select cluster_name.haw
__ 11. Navigate to the Networks screen and add the network name. Use the default
HACMP name for the first Ethernet network: net_ether_01.
Select the Requisite tab in lower left.
Select Networks in the navigation pane.
Enter net_ether_01 in the Network Name: field.
Verify that:
Network Type: = ether
Network Class: = Class C
Enable IP Address Takeover via IP Aliases: is checked
Click Add
__ 12. Navigate to the Interface Names screen and enter the interfaces that HACMP will
be using. If you are accessing the systems remotely, you do not need to add the
interfaces used for classroom access (en2 in our example).
Select Interface Names in the navigation pane.
Enter the IP label for the first interface and the interface device for your first
node; verify that Network Type:, Network Name:, and Node Name: fields
are correct and click Add.
Repeat for the other interface in the first node.

V3.1.0.1
EXempty Repeat for the interfaces in your second node.
__ 13. Navigate to the IP Labels screen and enter the service and persistent IP labels.
Select IP Labels in the navigation pane.
Start with the service labels. Select:
Type: = Service
Shared: = Yes C
Enter the IP label for each service address and click Add.
Next add the persistent labels. Select:
Type: = Persistent
Enter each nodes persistent IP label in the Node IP Label: field;
select the node name in the Node Name: pull-down;
and click Add.
__ 14. Save your worksheet. Since you now have the network information entered, you
should use the Validate When Saving setting.
Select the Settings menu; verify that Validate When Saving is checked.
Select File -> Save. The Validate HACMP Definition window should open
and show No errors. Click Close to continue.
Part 2: Configure TCP/IP Interfaces and Name Resolution

__ 15. With your cluster planning sheets available, begin the configuration.
__ 16. Log in as root to both of the cluster nodes.
__ 17. Check the UNIX Hostname (both the hostname command and the uname -n
command should give you the same desired answer).
To display: hostname and uname -n
To change: smitty hostname and uname -S hostname

V3.1.0.1
EXempty __ 18. Using the component worksheets or configuration diagram for values, configure two
network adapters for use as communication interfaces, remember that each
communication interfaces must use a separate logical subnet. If you are accessing
the system remotely, dont change the remote access interface (usually en2).
Note: Do NOT use the Minimum Configuration & Startup option in SMIT. It
changes the name of the node. Use smit chinet instead.
# smitty chinet - or
# smitty
Communications Applications and Services -> TCP/IP -> Further
Configuration -> Network Interfaces -> Network Interface
Selection -> Change / Show Characteristics of a Network Interface
Select each interface in turn and enter:

INTERNET ADDRESS (dotted decimal),
Network MASK (hexadecimal or dotted decimal) and
Current STATE (up).
__ 19. Recheck the hostname and system name.
# hostname
# uname -n
__ 20. Re-verify that the netmask for each interface is correctly set. For this class they
should be set to 255.255.255.0. You can use smitty chinet or ifconfig -a.
The default could cause errors later depending on what your network address was.
# smitty chinet
Select each interface, verify the netmask settings are the same. For this class
the netmask should be 255.255.255.0
# ifconfig -a
__ 21. Check the IP address configuration against the cluster worksheets. Dont worry
about the IP labels yet.
# netstat -in
# ifconfig -a
__ 22. Repeat for the other node. When both nodes are configured, test the
communications between the nodes. Use the ping command to verify connection
between each set of communication interfaces.
t2halifax # for i in 1 2
> do
> ping -c 2 192.168.21.$i
> ping -c 2 192.168.22.$i
> done
t2toronto # for i in 1 2
> do
> ping -c 2 192.168.21.$i
> ping -c 2 192.168.22.$i
> done
__ 23. Update the /etc/hosts file on both nodes (update one and ftp it to the other node).
(Remember to include your service IP labels and persistent IP labels.)
Notes:
Example assumes using suggested IP labels.
Client labels are covered in a later exercise.
If you are connecting remotely, you will need to supply the correct values
for the classroom network interfaces, as provided by your instructor.
# vi /etc/hosts
127.0.0.1 loopback localhost
192.168.#1.1 t#halifax-if1
192.168.#2.1 t#halifax-if2
192.168.#3.1 t#halifax-per
192.168.#3.10 t#appA-svc
192.168.#1.2 t#toronto-if1
192.168.#2.2 t#toronto-if2
192.168.#3.2 t#toronto-per
192.168.#3.20 t#appB-svc
A.B.C.X t#halifax
A.B.C.Y t#toronto
__ 24. Verify name resolution and connectivity on BOTH nodes for all interface IP labels.
(Service and persistent IP labels are not configured yet, so ping will not succeed for
them, but you should still verify name resolution.)
t2halifax # host t2halifax-if1
t2halifax # ping t2halifax-if1
t2halifax # host t2halifax-if2
t2halifax # ping t2halifax-if2
t2halifax # host t2toronto-if1
t2halifax # ping t2toronto-if1
t2halifax # host t2toronto-if2
t2halifax # ping t2toronto-if2
t2halifax # host t2halifax-per
t2halifax # host t2toronto-per
t2halifax # host t2appA-svc
t2halifax # host t2appB-svc

V3.1.0.1
EXempty t2toronto # host t2halifax-if1

t2toronto # ping t2halifax-if1
t2toronto # host t2halifax-if2
t2toronto # ping t2halifax-if2
t2toronto # host t2toronto-if1
t2toronto # ping t2toronto-if1
t2toronto # host t2toronto-if2
t2toronto # ping t2toronto-if2
t2toronto # host t2halifax-per
t2toronto # host t2toronto-per
t2toronto # host t2appA-svc
t2toronto # host t2appB-svc
Part 3: Configure Non-IP Interface

In this part of the exercise, you will verify the hardware and AIX configuration to support a
non-IP network for HACMP.
HACMP supports several different non-IP networks: Serial (RS232), Disk heartbeat (over
an enhanced concurrent mode disk), Target Mode SSA, and Target Mode SCSI.
Depending on the equipment available for your class, you will have at least one of these
options available for use in this class. Check with your instructor for which type of non-IP
network you will be using.
__ 25. With your cluster planning sheets available, begin the configuration.
__ 26. Log in as root to both of the cluster nodes.
__ 27. Based on the type of non-IP network you will be using, jump to the following section
to complete this exercise.
Note: You may have more than one non-IP network available, or maybe only one. In
any case, we suggest that you read through all of the exercise sections, performing
those steps that are supported by your hardware.
Serial:
Using a RS-232 non-IP network on page 11
Target Mode SSA:
Using a target mode SSA non-IP network on page 12
Disk heartbeat using an enhanced concurrent mode disk:
Using a disk heartbeat non-IP network on page 13
Using a RS-232 non-IP network

__ 28. On both nodes check the device configuration of the unused tty device. If the tty
device does not exist, create it. If it does exist, ensure that a getty is not spawned, or
better still, delete it and redefine.
# smitty tty
-> Change/Show Characteristics of a tty or Add a tty
Enable LOGIN = disable
BAUD rate = 9600
PARITY= none
BITS per character = 8
Number of STOP BITS = 1
__ 29. Test the non-IP communications:
- On one node execute
# stty < /dev/tty#
where # is your tty number.
- The screen appears to hang. This is normal.
- On the other node execute
# stty < /dev/tty#
where # is your tty number.
- If the communications line is good, both nodes return their tty settings.
Using a target mode SSA non-IP network

__ 30. If using target-mode SSA for your non-IP network, check if the prerequisites are
there. A unique node number must be set and the device driver must be installed. If
not add it.
- # lsdev -C | grep ssa
- # lsattr -El ssar
- # lscfg -vl ssa0
- # lslpp -L devices.ssa.tm.rte
(If not installed on both nodes, ask your instructor if it can be added and
where are the needed information or resources.)
- Install missing software.
- Install usable microcode (if required).
- # chdev -l ssar -a node_number=unique_number
(must be different on each node)
- Run cfgmgr
must be run on the first node, then the second node, then the first node
- # ls -l /dev | grep ssa
verify the existence of the .tm and .im device files
__ 31. Test the non-IP communication using SSA.
- Node A:
# cat < /dev/tmssanumber.tm
where number is node number of node B
- Node B:
# cat filename > /dev/tmssanumber.im
where filename is a small text file and number is node number of node A

V3.1.0.1
EXempty Using a disk heartbeat non-IP network

__ 32. Configure a disk for disk heartbeating
In order to use a disk heartbeat non-IP network, you must have at least one
enhanced concurrent volume group imported on each node. You created this in the
last exercise. The volume group does not have to be varied-on by either node.
__ 33. Test the non-IP communication using disk heartbeat.
You can use the /usr/sbin/rsct/bin/dhb_read command to test the disk
heartbeat function. Run the command without any arguments to get a usage
statement (there is no man page).
To test, run the command in receive mode on one node, using the disk you added to
appA_vg. Then run it in transmit mode on the other node, again using the disk you
added to appA_vg. You should get Link operating normally output on both
sides. The receive mode command will time-out in about 1 minute if you dont start
the transmit mode command on the other side - or if they are unable to
communicate.
t2halifax # /usr/sbin/rsct/bin/dhb_read
Usage: dump mode : dhb_read -p device-name
Usage: receive mode : dhb_read -p device-name -r
Usage: transmit mode: dhb_read -p device-name -t
t2halifax # /usr/sbin/rsct/bin/dhb_read -p hdisk1 -r
Receive Mode:
Waiting for response . . .
Link operating normally
Note: hdisk1 is an example. Use the appropriate disk name for your systems.
t2toronto # /usr/sbin/rsct/bin/dhb_read -p hdisk1 -t
Transmit Mode:
Detected remote utility in receive mode. Waiting for response . .
.
Link operating normally
END OF LAB

V3.0
EXempty Exercise 4. Application Planning

(with Hints)

We now know how to plan for adding an application to an HACMP
cluster. Making sure the application and the application start/stop
scripts are running before letting HACMP take control is the next step.

Add the IBM Web server powered by Apache to the nodes
Create the application start and stop script for HACMP
Test the application functionality outside of HACMP
Introduction
The intention is to go through the process of making sure an
application and the start/stop scripts work outside of HACMP before
configuring HACMP to take control.
Required Materials
The AIX 5L Expansion pack
Copyright IBM Corp. 1998, 2005 Exercise 4. Application Planning 4-1


Preface
This exercise will have you do the AIX work of setting up and testing the 2 applications
and start/stop scripts that you will put under HACMP control later in the course. This
way you will know that the applications work before putting them under the control of
HACMP which is the recommended practice.
Part 1: Create Start/Stop scripts for appA

The first application appA will be a simple command to put the start and stop times into
a log in shared storage.
__ 1. login to t#toronto as root and export the appA_vg. This is so that we build appA only
on your halifax node outside the control of HACMP.
login t#toronto
exportvg appA_vg
__ 2. Start a console session (login) to t#halifax as root and change directory to /tmp
# cd /tmp
__ 3. Execute the following commands to create the appA start script (if you know vi you
can use the following skipping the 1st occurrence of # echo on each line):
# echo #!/bin/ksh > /tmp/Astart
# echo /usr/bin/hostname>>/appA_fs/log >> /tmp/Astart
# echo /usr/bin/date +starting.%a%H%M >>/appA_fs/log >>/tmp/Astart
# echo echo "-------------" >> /appA_fs/log >> /tmp/Astart
__ 4. Execute the following commands to create the appA stop script (if you know vi you
can use the following without the 1st occurrence of echo on each line):
# echo #!/bin/ksh > /tmp/Astart
# echo /usr/bin/hostname>>/appA_fs/log >> /tmp/Astop
# echo /usr/bin/date +stopping.%a%H%M >>/appA_fs/log >>/tmp/Astop
# echo echo "-------------" >> /appA_fs/log >> /tmp/Astop
__ 5. Make the scripts executable
# chmod +x /tmp/Astart
# chmod +x /tmp/Astop
Part 2: Test the scripts for appA

__ 6. On t#halifax,varyon and the shared volume group and mount the file system
# varyonvg appA_vg
# mount /appA_fs
Execute the following:

V3.0
EXempty __ 7. # /tmp/Astart
__ 8. # /tmp/Astop
__ 9. # cat /appA_fs/log (you should see entries from the start/stop scripts)
__ 10. # umount /appA_vg
__ 11. # varyoffvg /appA_vg
Part 3: Install Mozilla and the IBM Web server software

Our second application will be a web server. In this part of this exercise we will install an
http server on private storage on your t#halifax node for testing. In a later exercise we
will switch to a shared storage environment. Again, this minimizes problem
determination and allows us to know that the application is working before putting it is
put under the control of HACMP.
__ 12. Stay in your session with your t#halifax node as root.
__ 13. Install Mozilla
# cd /ha53/web-app/mozilla
# geninstall -d . -f ./RPMS/ppc/bundle
# geninstall -Ia -Y -d . Mozilla.base.rte
# lslpp -L | grep Moz (verify install)
__ 14. Install IHS2 (IBM HTTP Server)
# cd /ha53/web-app
# geninstall -Z -d /ha53/web-app J:IHS2
# lslpp -L IHS2 (verify install)
Part 4: Test
__ 15. Start an X session (vnc) to t#halifax and open a terminal window.
__ 16. Bring up the web server using the following command (this command will become
the start script):
# /usr/HTTPServer/bin/apachectl start
__ 17. Bring up mozilla and connect to the web server using the URL:
http://t#halifax
__ 18. Bring down the web server using the following command (this command will become
the stop script).
# /usr/HTTPServer/bin/apachectl stop
Part 5: Add the appA resource to the Planning Worksheets

__ 19. Open your planning worksheets file:
# cd /usr/es/sbin/cluster/worksheets
# ./worksheets

Select file --> open

Navigate to /home/au54 and select cluster_name.haw
__ 20. Click on the Application Servers then:
Set Application Server Name to appA
Set Full path of Start Script to /tmp/Astart
Set Full path of Stop Script to /tmp/Astop
Click on Add
__ 21. Click on Nodes and Communication Paths in the navigation area then fill in as
follows:
For t#halifax, set the Communication Path to Node value to the IP address of
t#halifax-if1
Click on add
Repeat for t#toronto using the IP address of t#toronto-if1
__ 22. Save the file:
Click on File ... Save
The Validate HACMP Definition window should open and show No errors. Click
Close to continue.
END OF LAB

V3.0
EXempty Optional Exercises

For the Web-enabled Candidates
__ 1. Change the Web server pages on the shared disk to prove the location of the data
elements.
END OF LAB


V3.1.0.1
EXempty Exercise 5. HACMP Software Installation

(with Hints)

This exercise installs the components of HACMP for AIX to support all
resource group policies.

Verify the node is prepared for the installation of HACMP
Identify and install the packages to run HACMP
Copyright IBM Corp. 1998, 2005 Exercise 5. HACMP Software Installation 5-1

Preface
This exercise is composed of two parts, system capacity checks and software
installation.
Part 1: System Capacity Checks

__ 1. Log into t#halifax as root (this part can be done in parallel by a second person
working on the other node, t#toronto).
__ 2. Verify the following disk space requirements:
i. 140 MB free space in /usr, although the installation of the software will
automatically increase the size if required.
ii. 100 MB free in /tmp, /var and /
Use df -m to check the filesystem sizes.
__ 3. For most lab environments, check to see that the system paging space is set to
twice the size of main memory. This is the default recommendation for small
memory machines. In this lab you do not need to actually add paging space.
lsps -a to see the paging file space
lsattr -E -l sys0 -a realmem to see the amount of memory
__ 4. Ensure Part 1 is performed for your other node, t#toronto
Part 2: HACMP Node Installation

__ 5. Start a console session (login) to t#halifax as root (this part can be done in parallel
by a second person working on the other node t#toronto).
__ 6. Install the prerequisites from /ha53/hacode/pre_reqs.
cd /ha53/hacode/pre_reqs
smitty install_all
__ 7. Ensure the following bos filesets are installed using the lslpp -L command. If any
are missing, inform your instructor
- bos.adt.lib
- bos.adt.libm
- bos.adt.syscalls
- bos.rte.lvm
- bos.clvm
- bos.clvm.enh
- bos.data (commonly forgotten)

V3.1.0.1
EXempty lslpp -L bos.adt*

lslpp -L bos.data
__ 8. RSCT filesets must be at a minimum, version 2.4.2.0. Check the following using the
lslpp -L command. If any are missing, please inform your instructor.
- rsct.compat.basic
- rsct.compat.clients
lslpp -L rsct.compat*
__ 9. If you will be using SSA storage check for the following filesets:
- devices.ssa.tm
lslpp -L devices.ssa.tm.rte
__ 10. Change directory to the location of the cluster filesets. In most classes they can be
found in a subdirectory of the /ha53 directory. If there are questions, ask the
Instructor.
ls -l /ha53/hacode/ha_cd/installp_ppc
__ 11. Install preview the following HACMP filesets (accept license):
Note: update_all or install software will not show the msg filesets, you must use
install_all or Install..from ALL Available Software.
- cluster.adt.es
- cluster.doc.en_US.es
- cluster.es
- cluster.es.clvm
- cluster.es.cspoc
- cluster.es.worksheets
- cluster.license
- cluster.man.en_US.es.data
- cluster.msg.en_US.cspoc (lower case en)
- cluster.msg.en_US.es
- rsct
Use smitty --> Software Installation and Maintenance
--> Install and Update Software
-->Install and Update Software from ALL Available Software
Enter . for the directory
Press F4 to see the list of filesets.
After choosing the filesets and returning to the install menu, set PREVIEW
to yes and set ACCEPT new license agreements to yes. Then execute the
preview install.
__ 12. If the HACMP packages pass the prerequisite check, set preview to no and install
the HACMP filesets. If there is a prerequest failure, notify your Instructor.
__ 13. Install HACMP maintenance. They should be in the /ha53/hacode/ptf/ha

subdirectory.If you have questions, ask the instructor.
Change directory to the maintenance directory.
smitty install -> Install and Update software
--> Update installed software to latest level (Update All)
Enter . for the directory
Change the commit updates field from yes to no
Change the save replaced files from no to yes
__ 14. Install the efix. You do not use installp for this. You just copy the file to
/usr/es/sbin/cluster/diag and make sure it is executable on both nodes.
__ 15. Reboot the nodes.
shutdown -Fr
__ 16. Verify the SMIT menus. Check to see if the HACMP screens are available.
smitty hacmp
__ 17. (Optional) It would be a good idea to set up your /.profile to include paths to the
HACMP commonly used commands so that you dont have to keep entering full path
names in the later lab exercises.
PATH=$PATH:/usr/es/sbin/cluster:/usr/es/sbin/cluster/utilities
PATH=$PATH:/usr/es/sbin/cluster/etc
PATH=$PATH:/usr/es/sbin/cluster/diag
__ 18. (Very Optional) If the nodes have a tape subsystem attached, now would be a good
time for a mksysb backup.
__ 19. Ensure Part 2 is also performed for t#toronto.
Part 3: Setting Up the Client Communications

__ 20. This part of the exercise requires that there is a Client AIX node for your team. If you
have only a PC then you can (after the application integration lab) add clstat.cgi to
the /usr/HTTPServer/cgi-bin directory on both cluster nodes and then use the pc
browser to go to the service address of the HTTP resource group -- your instructor
can help you with this.
__ 21. Using a console session, login to the client (your 3rd machine) as root. If CDE
comes up then exit CDE for now. CDE will come back again after the reboot later in
this exercise.
__ 22. Execute smit mktcpip to set the hostname and IP address of this machine for an enX
interface that has a cable (virtual) in it. The IP address must be on the same subnet
as one of the node interfaces. The suggested hostname is t#regina and the

V3.1.0.1
EXempty suggested address is 192.168.#1.3. Do not set default route and ensure DNS is
disabled.
__ 23. Create an alias for the interface above to be on the same subnet as the service label
for the web app. The suggested value is 192.168.#3.30 and the subnet mask is
255.255.255.0.
smit inet --> configure Aliases --> Add an IPV4 Network Alias
__ 24. Acquire the /etc/host file from halifax# and ensure that the information in this file
agrees with what you did in the previous 2 steps.
ftp t#halifax-if1
get /etc/hosts
quit
cat /etc/hosts
__ 25. Test to ensure that TCP/IP functions correctly.
ping t#halifax-if1
ping t#toronto-if1
__ 26. Test name resolution of the client and the nodes.
Use the host command to test name resolution.
Part 4: HACMP Client Install and setup

__ 27. Install the AIX prerequisite filesets in /ha53/hacode/pre_reqs
smitty install_all
__ 28. Install the HACMP client filesets:
cluster.adt.es
cluster.es (choose only the 3 client filesets)
cluster.license
cluster.man.en_US.es
cluster.msg.en_US.es (choose only the client fileset)
Change to the directory with the filesets
smitty install_all
__ 29. Install the ptf updates (efix is not required here)
Change to the directory with the filesets
smitty update_all
__ 30. Install Mozilla
see exercise 4
__ 31. Verify browser starts and can display a URL, like
file:///usr/lpp/bos.sysmgt/mkcd.README.html/
The next 3 steps will prepare you to use clinfoES from the client machine after HACMP is
started in the next exercise.
__ 32. Copy the clstat.cgi script from /usr/es/sbin/cluster to the /var/docsearch/cgi-bin
directory.
cd /var/docsearch/cgi-bin
cp /usr/es/sbin/cluster/clstat.cgi ./
__ 33. Verify that the file /var/docsearch/cgi-bin/clstat.cgi is world-executable (755 or
rwxr-xr-x)
chmod +x clstat.cgi
ls -al clstat.cgi
__ 34. Test access to clstat.cgi using the URL
http://localhost/cgi-bin/clstat.cgi
You should get a window with the message Could not initialize clinfo connection.
__ 35. Reboot and do the ping tests to verify that this client machine functions as expected.
END OF LAB

V3.1.0.1
EXempty Exercise Review/Wrap-up

This is a good place to stop for a backup.

V3.1.0.1
EXempty Exercise 6. Initial Cluster Configuration

(with Hints)
What This Exercise is About

This lab covers the configuration and testing of a one sided custom
resource group. The cluster planning worksheets will continue to be
updated as the capabilities of the cluster grow.

Configure a 2 node cluster with the Two-Node Cluster
Configuration Assistant
Add an application to a resource group
Discover network and storage information
Add a resource group to obtain a mutual takeover configuration.
Introduction
The scenario starts with setting up and running the Two-Node Cluster
Configuration Assistant to build a standby environment and then
building a second resource group to obtain a (mutual) takeover
environment.
Required Materials
Cluster planning worksheets.
Copyright IBM Corp. 1998, 2005 Exercise 6. Initial Cluster Configuration 6-1

Remember this?
Overview of the Implementation Process

Plan and configure AIX
Eliminate single points of failure
Storage (adapters, LVM volume group, filesystem)
Networks (ip interfaces, /etc/hosts, non-ip networks and
devices)
Applications start and stop scripts
Install the HACMP filesets and reboot
Configure the HACMP environment
Topology
Cluster, node names, HACMP ip and non-ip networks
Resources and Resource groups:
Identify name, nodes, policies
Resources: Application Server, service label, VG, filesystem
Synchronize then start HACMP
We are now ready to Configure the HACMP environment.

In exercise 4 you setup the application environment so that we will be able to do the
configuration all at once using the Two-Node Cluster Configuration Assistant!!!
NOTE: These steps can only be done on one node. You should choose one of your nodes
to be the administration node. We will assume it is t#halifax
Part 1: Configuring a 2 node takeover cluster environment

With your cluster planning sheets available begin the configuration.
__ 1. Ensure you have a console session (login) to t#halifax. Then run the Two-Node
Cluster Configuration Assistant. You will need an IP address (label) for the 2nd

V3.1.0.1
EXempty node, an application server name unique to your team, start and stop script names,
and a service label.
smitty hacmp --> Initialization and Standard Configuration
--> Configuration Assistants-->Two-Node Cluster Configuration
Assistant
For the Communication Path to Takeover Node, select (F4) t#toronto-if1
For the Application Server Name, type appA
For the Application Server Start script, type /tmp/Astart
For the Application Server Stop Script, type /tmp/Astop
For the Service IP Label, select (F4) t#appA-svc.
__ 2. If you encountered an error then do the cluster remove procedure (see lecture or
ask instructor) on both nodes before retrying. Note If you encounter the error
ERROR: Devices are not in Enhanced Concurrent Mode VG then the efix was not
applied or execute flag not on. Further information, if needed, can be found in the
/var/hacmp/clverify/clverify.log
Lets now look at what happened to you as a result of this command.
__ 3. Look at the smit output to see what the Assistant did. You can also find this output in
the /var/hacmp/log/clconfigassist.log file.
__ 4. Start a console session (login) on to your other node (t#toronto) to prove that the
cluster was created on both nodes.
Were the application start and stop scripts copied over? ________________
Was the volume group imported to the other node? __________________
Use the command /usr/es/sbin/cluster/utilities/cldisp | more to answer the
following questions (ignore warning messages in the output):
What is the cluster name? _______________________________________
What is the resource group name? ________________________________
What is the startup policy? ______________________________________
What is the fallback policy?______________________________________
What is the vg resource name (if any)? _____________________________
What is the non-IP network name (if any)? __________________________
On what enX is t#halifax-if1? _____________________________________
What is the IP network name? ____________________________________
Were the start/stop scripts copied over? _____________________________
Was a network created for your console session address?_____________
Note: You will not see this if you are using an attached console. We will remove
this network later so you do not continue to get verify warnings.
__ 5. So were you impressed? _________________________________
__ 6. You can now add the IP network and non-IP network names, that we promised
would be generated by HACMP, to your component work sheets and/or the cluster
diagram if you want to.
__ 7. Return to your administrative node (t#halifax).
__ 8. Optional depending on lab environment: Define an additional Non-IP RS232 or a

TMSSA network. Note that a network is automatically created when you choose the
pair of devices that form the endpoints of the network. Ask your instructor if you can
do this.
smitty hacmp
Select, Extended Configuration
Select Extended Topology Configuration
Select Configure HACMP Communication Interfaces/Devices
Add Communication Interfaces/Devices
Select Add Discovered Communication Interface and Devices
Select Communication Devices from the list
Select, using F7, the Point-to-Point Pair of Discovered Communication
Devices (either a /dev/tty# pair or a TMSSA# pair).
__ 9. (Only if you did previous step) Execute the command cltopinfo and see that the
additional non-IP network was configured. Add this name to the worksheet and/or
diagram.
__ 10. Remove the console session network (if not using an attached console) from the
HACMP topology and synchronize. You will need the extended path to do this.
smitty hacmp --> Extended Configuration
-->Extended Topology Configuration -->Configure HACMP Networks
--> Remove a Network from the HACMP Cluster
Select the network containing your console session address
Return to (F3) the HACMP for AIX menu
Select Initialization and Standard Configuration
Select Verify and Synchronize HACMP Configuration
__ 11. Add a persistent node address for each node in the cluster. You will need the
Extended path to do this -- select Configure HACMP Persistent Node IP
Label/Addresses from the Extended Topology Configuration menu,
Configure a Persistent Node IP Label/Address
Select Add a Persistent Node IP Label/Address
Select a node form the list, press enter
Select (using F4) the network name and IP Label/Address -- the Network
Name will be the same Network Name that the interfaces and service labels
belong too. The suggested IP Labels will be names of the form XXX-per.
Repeat this step for the other node.
__ 12. Synchronize the changes -- Using the F3 key, traverse back to the Extended
Configuration smit screen.
Review the output upon completion looking for any Errors or Warnings. Errors must
be corrected before continuing, warnings should simply be reviewed and noted.

V3.1.0.1
EXempty __ 13. Check to see that your persistent addresses were created. If not then wait until the
cluster is started in Part 3 below and then check again.
netstat -i
__ 14. Take about 10 minutes to review the Startup, Fallover, and Fallback policies using
the F1 key on the Add a Resource Group menu. When you are ready, proceed to
the next step.
-->Configure HACMP Resource Groups -->Add a Resource Group
Part 2: Starting HACMP

__ 15. With your cluster planning sheets available as reference documentation, it is time to
start the cluster just on your administrative node (t#halifax). OK in smit does not
mean startup completed.
On the t#halifax node, enter smitty clstart (or smit hacmp
-> System Management (C-SPOC)
-> Mange HACMP Service
-> Start Cluster Services <-- choose t#halifax and start clinfo.
__ 16. Prove that cluster services are started (clstrmgrES running is not sufficient in
HACMP 5.3). The supported command for doing this is clcheck_server daemon.
However it answers only with a return code (and at time of course development did
not work correctly). However using grpsvcs as the daemon name is a good indicator.
You can also use one of the following commands, which may not be supported on
future releases, for status information: lssrc -ls clstrmgrES or cldump or cldisp
clcheck_server grpsvcs ; echo $? (1 means up)
try some of the other commands especially:
lssrc -ls clstrmgrES | grep state
__ 17. Check that all resources were acquired successfully on the t#halifax node.
lsvg -o
mount
df -k
netstat -i or netstat -in (also verify persistent addresses are there)
There is a starting message in the /tmp/appA.log file.
__ 18. Check that snmpd is working on your cluster nodes by using the command:
snmpinfo -m next -v risc6000clsmuxpd. This command should return the cluster
id which you can see if you execute the command
odmget HACMPcluseter | grep id .
__ 19. Go to your client machine (t#regina).
__ 20. Copy via ftp the clhosts.client file created by HACMP during synchronization in the
/usr/es/sbin/cluster/etc directory to the client machine in the same directory.
Check that snmpd is working to a cluster node by executing the command:

snmpinfo -h t#halifax -m next -v risc6000clsmuxpd
__ 21. Start the clinfoES subsystem and verify that the /usr/es/sbin/cluster/clstat -a
command works.
startsrc -s clinfoES
clstat -a
__ 22. Now start Netscape and make sure that the URL to clstat.cgi is working properly.
The URL is http://localhost/cgi-bin/clstat.cgi
You should now see a window with cluster information displayed. Be patient if
this window shows that the cluster is unstable.
Take a moment to familiarize yourself with what you are looking at. Click on the
resource group name.
You can use this session to monitor the fallover testing that comes next (or you
can run clstat on one of your cluster nodes)
__ 23. Now go to your administrative node (t#halifax) and stop it graceful. Watch what
happens in the clstat browser (be patient -- it may take 2 minutes).
smitty clstop and stop t#halifax graceful
__ 24. Now start HACMP and clinfo on BOTH nodes
smitty clstart <-- start t#halifax and t#toronto and clinfo
__ 25. Use the lsvg command to see that the shared vg (appA_vg) is varied on in passive
mode on the other node (t#toronto).
lsvg appA_vg
Part 3: Fallover testing

__ 26. Return to your administrative node (t#halifax) with your cluster planning sheets
available for reference.
It is time to test the cluster. Although the fallover testing is a function of planning to
eliminate single points of failure, some basic tests should be performed on any cluster.
__ 27. On both nodes verify the IP labels used on each interface. netstat -i Notice
which interface has which IP label.
__ 28. On the t#toronto node telnet to the appA service address (t#appA-svc).
__ 29. Run the command tail -f /usr/es/adm/cluster.log | grep EVENT. There
should not be any scrolling of the log file.
On your t#halifax node, fail the adapter (enX) that the t#appA-svc address is running on by
executing the command ifconfig enX down (or disconnect the cable from the enX
adapter card--kind of hard though to pull a virtual cable!).

V3.1.0.1
EXempty Watch the reaction on both nodes in the /usr/es/adm/cluster.log file. Also
monitor the clstat window. Notice that the telnet session from the t#toronto node was
not interrupted and that the log information scrolled by during the event processing.
__ 30. When swap adapter has completed, verify that the location of the t#appA-svc
service address is now on another Ethernet adapter.
Use netstat -i
__ 31. Restore the failed adapter. The interface should now be in an UP state.
ifconfig enX up (or connect the network cable)
__ 32. (Optional) - You may wish to swap the service address (and/or) persistent address
back by using C-SPOC.
execute smitty hacmp.
Select Cluster System Management (C-SPOC)
Select HACMP Communication Interface Management
Select Swap IP Addresses between Communication Interfaces
Select t#appA-svc from the Available Service/Communication Interfaces
smit screen.
Select t#halifax-if1 from the Swap onto Communication Interface smit
screen (This hint assumes that t#appA-svc is currently an alias to
t#halifax-if2).
confirm the information displayed and hit enter)
__ 33. Using the console session rather than a telnet session (because you will lose it),
monitor with the command tail -f /usr/es/adm/cluster.log | grep EVENT the
t#halifax-if1 node and virtually disconnect both network cables using the ifconfig
enX down command to both interfaces at the same time.
ifconfig en0 down ; ifconfig en1 down
__ 34. There should be a network down event executed after a short period of time. What
happens to the resource group on the t#halifax node, and why?
The resource group should have moved to the t#toronto node
The reason for this is because selective fallover is invoked when a network
down event was detected on t#halifax. HACMP moves the resource group to
maximize its availability
__ 35. On t#toronto, check using the command tail -f /usr/es/adm/cluster.log | grep
EVENT to see that the t#toronto node should also have detected a network failure.
__ 36. On t#halifax, restore both the network connections using the ifconfig enX up
command. What event do you observe happens?
ifconfig en0 up ; ifconfig en1 up
network up
__ 37. Where is the resource Group at this time? Verify that the IP labels, volume groups,
and file systems and application are available on that node.
The resource group should still be on the t#toronto node because the
Fallback policy is Never Fallback.
netstat -i
lsvg -o
/usr/es/sbin/cluster/utilities/clRGinfo
cat /tmp/appA.log
__ 38. You are now going to move resources back from one node to the other. On the
t#halifax node monitor the cluster.log file. On the t#toronto node execute smit clstop
and stop the cluster services with the mode of takeover. Leave the default value for
the other fields.
tail -f /usr/es/adm/cluster.log | grep EVENT (on t#halifax)
smit clstop (on t#toronto)
__ 39. The clstat.cgi should change colors from green to yellow (substate unstable,
t#toronto leaving) and the state of the t#toronto node and interfaces should change
to red (down).
__ 40. All of the components in the resource group should move over to the t#halifax
node. Verify the IP labels, volume groups, and file systems on the t#halifax node.
netstat -i
lsvg -o
cat /tmp/appA.log
__ 41. On the t#toronto node restart HACMP. Observe the /usr/es/admin/cluster.log file (as
shown above) on the t#halifax node and, of course, the clstat session. The resource
group will stay put.
Part 4: Take a break
Part 5: Mutual Takeover: Adding the LVM resources

In the remaining parts of this exercise, C-SPOC and DARE will be used to make changes
to the cluster while it is running. If all things function as designed, no system reboots or
HACMP restarts will be required. C-SPOC and DARE are then covered in more detail in
the next lecture.
__ 42. Ensure HACMP is running on both nodes and that the appA application is running
on t#halifax
__ 43. Using the lspv command on BOTH nodes, verify that there is a shared disk hdiskX
available with the same PVID. If so skip the next step.
lspv
__ 44. If no matching pvid then
__ a. on the t#halifax node make sure that the hdiskX has no PVID

V3.1.0.1
EXempty lspv
chdev -a pv=clear -l hdiskX
__ b. Create a new PVID for hdiskX
chdev -a pv=yes -l hdiskX
__ c. On the t#toronto node delete the hdisk.
rmdev -dl hdiskX
__ d. Add the disk back in.
cfgmgr
__ e. Verify the hdisk number and PVID agree between the two nodes.
lspv on both nodes
__ 45. On the administrative node (t#halifax) create a shared volume group called appB_vg
using C-SPOC.
smitty hacmp
Select Configure Resources to make Highly Available.
Select Configure Concurrent Volume Groups and Logical Volumes.
Select Concurrent Volume Groups
Select Create a Concurrent Volume Group
Select Select ALL (both) the Node Names that will share the Volume
Group
Select the PVID that you identified in step 43.
Fill out the volume group menu:
-Name= group appB_vg
-Using F4 select the single physical volume you identified above
-Check the Physical partition SIZE and major number (C-SPOC
chooses a valid major number); set enhanced concurrent is true)
Create the volume group.
__ 46. Verify the Volume Group exists on both nodes.
lspv
lsvg
Now that the volume group is created it must be discovered, a resource group must be
created and finally the volume group must be added to the resource group before any
further C-SPOC utilities will access it.
__ 47. Discover the volume group using the command smitty hacmp and selecting
Extended Configuration followed by selecting Discover HACMP-related
Information from Configured Nodes
__ 48. On t#halifax, create a resource group called appB_group with the t#toronto node as
the highest priority and t#halifax node as the next priority.
smitty hacmp
Select Configure HACMP Resource Groups
Select Add a Resource Group
Enter the resource group name appB_group, from the planning worksheets.
The participating node names must also be entered -- enter t#toronto first
Take the defaults for the policies.
__ 49. Add the volume group to the resource group
Return (F3) to the menu Configure HACMP Resource Groups then
Select Change/Show Resources for a Resource Group (standard)
Select appB_group
Enter the volume group name using F4.
__ 50. Synchronize the Cluster.
smitty hacmp
Once synchronized, the Volume Group will be varied online, on the owning node
(t#toronto). Wait for this to happen.
__ 51. Now (still on your administrative node t#halifax) use C-SPOC to add a jfs log shared
logical volume to the appB_vg. The name should be appB_jfslog, the LV type should
be jfslog, and use 1 PP.
smitty hacmp
Select Configure Resources to make Highly Available
Select Configure Volume Groups, Logical Volumes and Filesystems
Select Shared Logical Volumes
Select Add a Shared Logical Volume
From the list provided, choose the entry for appB_group appB_vg
From the list provided, choose Auto-select
Set LP=1, NAME = appB_jfslog, LV TYPE = jfslog
__ 52. On t#toronto (where the volume group is active), format the jfslog so that it can be
used by the filesystem that will be created in the next few steps. If the log is not
formatted, it will not be used.
On t#toronto, execute logform /dev/appB_jfslog (answer yes to the
destroy data question
__ 53. Back to t#halifax and CSPOC. Add a second shared logical volume with number of
LOGICAL PARTITIONS=10, NAME= appB_lv, TYPE=jfs.
See the step 51 hint along with the information for this step.
__ 54. Create a filesystem for the Web documents. Enter smit hacmp.

V3.1.0.1
EXempty Select System Management (C-SPOC)

Select HACMP Logical Volume Management
Select Shared File systems
Select Journaled File Systems
Select Add a Journaled File System on a Previously Defined Logical Volume
Add a standard JFS
Pick the appB_lv entry from the pop-up menu
Enter /usr/HTTPServer/htdocs as the mount point. Leave the defaults for the
other values and press enter.
__ 55. The filesystem should be available on node t#toronto in a few minutes.
tail -f /usr/es/admin/cluster.log | grep EVENT or clstat and wait
for the cluster to become stable.
lssrc -ls clstrmgES | grep state
__ 56. Verify both nodes know about the new file system.
cat /etc/filesystems (on both nodes)
__ 57. Check to see that the filesystem is mounted on the system that currently owns the
resource group (should be t#toronto).
df
Part 6: Mutual Takeover: Install the IBM Web server software

In Exercise 4 we proved that the application would work on txhalifax. Now we install it
on txtoronto to put the docs on shared storage then remove the documents from private
storage on txhalifax.
__ 58. On t#toronto, install IHS2 (IBM HTTP Server)
# cd /ha53/web-app
# geninstall -Z -d /ha53/web-app J:IHS2
# lslpp -L IHS2 (verify install)
__ 59. In the directory /usr/HTTPServer/conf/, edit httpd.conf and change the
ServerName variable to be the same as the service IP label (t#appB-svc).
NOTE: The hostname must be resolvable, that is, host hostname should return a
good answer. If the hostname is not resolvable, add the hostname to the 127.0.0.1
address as an alias. If in doubt, ask the Instructor. Remember to ensure name
resolution on both nodes otherwise successful takeover will not happen!
__ 60. On t#halifax, delete all of the information in the directory /usr/HTTPServer/htdocs
(only on this node!).
cd /usr/HTTPServer/htdocs
rm -r ./*
This is because when the resource group falls over this will be covered by the
shared filesystem /usr/HTTPServer/htdocs.
Part 7: Mutual Takeover: Create the resources

__ 61. Make sure you are on your t#halifax as root
__ 62. Create the Application Server resource
smitty hacmp
Select Configure Resources to Make Highly Available
Select Configure Application Servers
Select Add an Application Server
Enter appB as the application name.
Enter /usr/HTTPServer/bin/apachectl start as the application start script.
Enter /usr/HTTPServer/bin/apachectl stop as the application stop script.
__ 63. Create the Service IP label resource.
Select Configure Resources to make Highly Available
Select Configure Service IP Labels/Addresses
Select Add a Service IP Label/Address
Enter the Service IP Label t#appB-svc (use F4) and the network (use F4)
Part 8: Mutual Takeover: Add resources to the resource group, sync,

and test
__ 64. Add the resources to the resource group
Return (F3) to the menu Initialization and Standard Configuration
Select Configure HACMP Resource Groups
Select Change/Show Resources for a Resource Group (standard)
Select appB_group
Enter the service label and application server name using F4.
__ 65. Synchronize the Cluster. Using the F3 key traverse back to Initialization and
Standard Configuration.
select Verify and Synchronize HACMP configuration
__ 66. Test that the t#appB-svc service IP label and volume group are available
netstat -i
lsvg -o
__ 67. Test to see that the HTTPServer (httpd) started ok.
ps -ef | grep http should show a number of httpd daemons running.
__ 68. From the client, start a new browser window and connect to the URL
http://t#appB-svc. The Web screen Welcome to the IBM HTTP Server window
should pop up.
__ 69. Test the new resource group on the t#toronto node to see if takeover works.

V3.1.0.1
EXempty smitty clstop (set Shutdown mode = takeover).

__ 70. Wait for takeover to complete and verify what happens to the Web server. Use the
page reload button on your Web browser to see if the Web server is really there.
__ 71. Bring up the t#toronto node again and start cluster services (smitty clstart).
__ 72. What has happened to the Resource Group, and why?
cllsres -g appA_group
ps -ef | grep http
lsvg -o
df -k
END OF LAB
Optional Exercises
For the Web-enabled Candidates
__ 1. Change the Web server pages on the shared disk to prove the location of the data
elements.
END OF LAB

V3.1.0.1

The first part of the exercise looked at using C-CSPOC to add a new resource to the
cluster.
END OF LAB
Exercise Review/Wrap-up
You have a running cluster. Congratulations, now the fun really begins. Make sure clstat
shows the cluster as stable with the TCP/IP and non-IP networks up.

V3.1.0.1
EXempty Exercise 7. Basic HACMP Administration
(with Hints)

This lab exercise introduces you to a number of basic HACMP
administration tasks.

Configure Web SMIT
Create a cluster snapshot
Use the C-SPOC functions and DARE capabilities of HACMP to
make changes to the cluster while it is running
Add additional resource groups while the cluster is running
Add additional service aliases
Modify resource group policies
Configure settling and fallback timers
Remove a cluster
Create a new cluster using the Initialization and Standard
Configuration path (not using the Two-Node Cluster
Configuration Assistant)
Configure IPAT via replacement
Describe how the ARP cache on clients can be updated using
either gratuitous ARP or Hardware Address Takeover (HWAT)
Configure HWAT
Restore a cluster from a snapshot
Introduction
This lab exercise introduces you to a number of basic HACMP
administration tasks.
Well start by configuring WebSMIT, so that you can use either
interface during this lab.
Copyright IBM Corp. 1998, 2005 Exercise 7. Basic HACMP Administration 7-1
Next well use the snapshot utility to save the current cluster.
Then well experiment with various startup policies, including using a
settling timer and a delayed fallback timer.
Then well remove the current cluster, create a new cluster using IPAT
via replacement and experiment with IPAT via replacement and HWAT.
Finally, well restore our original cluster.
Required Materials
Cluster planning worksheets.

V3.1.0.1
EXempty Exercise Instructions with Hints

This exercise looks at a variety of basic HACMP administration tasks.
First we must set up the application environment so that we can do the configuration all at
once using the Two-Node Cluster Configuration Assistant.
Note: These steps can only be done on one node. You should choose one of your nodes to
be the administration node. We will assume it is t#halifax
Part 1: Enable WebSMIT

In this part of Exercise 7., we will configure and enable WebSMIT.
For the purposes of this exercise, we will be configuring and enabling WebSMIT on your
administrative node (t#halifax).
Also, for the purpose of introducing you to WebSMIT, and in the interests of time, we will
not be using secure HTTP for WebSMIT in this exercise. For explanations on how to create
a more secure web server installation, see the WebSMIT README file
(/usr/es/sbin/cluster/wsm/README).
__ 1. Since appB_group resource group also uses the HTTP server, stop the appB_group
resource group.
We will restart appB_group at the end of Exercise 7.
# smitty hacmp
System Management (C-SPOC) -> HACMP Resource Group and
Application Management -> Bring a Resource Group Offline ->
__ 2. Make a copy of the /usr/HTTPServer/conf/httpd.conf file so that you can restore it
later, if you wish.
# cd /usr/HTTPServer/conf
# cp httpd.conf httpd.conf.appA
__ 3. Create a symbolic link from the /usr/es/sbin/cluster/wsm directory to a new
directory: /usr/HTTPServer/websmit.
# cd /usr/HTTPServer
# ln -s /usr/es/sbin/cluster/wsm ./websmit
Note: The README file suggests that you can either copy files from
/usr/es/sbin/cluster/wsm to /usr/HTTPServer/websmit or create a link. In our
testing, copying files as described in the README did not work.
__ 4. Set ownership and permissions of the /usr/HTTPServer/conf/websmit/logs
directory to nobody:system and 775.
# chown nobody:system websmit/logs
# chmod 775 websmit/logs
__ 5. Make the following edits to the /usr/HTTPServer/conf/httpd.conf file to configure

the HTTP server for WebSMIT.
Note: In the examples below, we show some comment lines to help you locate the
correct line in the file. The lines you must change are shown in bold.
__ a. Change the ServerName attribute from the appA-svc IP label to one of your
interface IP labels. If you are accessing the lab systems remotely, choose the
classroom interface so that you can access WebSMIT directly from a browser on
your classroom system.
ServerName t#halifax:42267
__ b. Add a Listen line which allows access from any IP to port 42267.
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, in addition to the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#
Listen 0.0.0.0:80
Listen 0.0.0.0:42267
__ c. Change the DocumentRoot attribute to point to the websmit/htdocs path.
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory,
but
# symbolic links and aliases may be used to point to other
locations.
#
DocumentRoot "/usr/HTTPServer/websmit/htdocs/en_US"
__ d. Change the Directory definition for documentation to point to websmit/htdocs.
# This should be changed to whatever you set DocumentRoot to.
#
<Directory "/usr/HTTPServer/websmit/htdocs/en_US">
__ e. Change the ScriptAlias for /cgi-bin/ to point to websmit/cgi-bin.
# ScriptAlias: This controls which directories contain server scripts.
# ScriptAliases are essentially the same as Aliases, except that
# documents in the realname directory are treated as applications and
# run by the server when requested rather than as documents sent to the
client.
# The same rules about trailing "/" apply to ScriptAlias directives as to
# Alias.
#
ScriptAlias /cgi-bin/ "/usr/HTTPServer/websmit/cgi-bin/"

V3.1.0.1
EXempty __ f. Change the Directory definition for cgi-bin to point to websmit/cgi-bin.

# "/usr/HTTPServer/cgi-bin" should be changed to whatever your
ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/usr/HTTPServer/websmit/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
__ 6. WebSMIT runs as user nobody. We need to set or check some permissions to allow
it to run:
__ a. Set permissions for wsm_cmd_exec
# cd /usr/HTTPServer/websmit/cgi-bin
# chmod 4511 wsm_cmd_exec
__ b. Check that permissions on /tmp allow access to user nobody.
# ls -ld /tmp
drwxrwxrwt 14 bin bin 12288 Sep 5 09:08 /tmp
__ 7. We are not using secure HTTP for this exercise, so we need to edit the
wsm_smit.conf. Comment out the REDIRECT_TO_HTTPS line.
# cd /usr/HTTPServer/websmit
# vi wsm_smit.conf
# set this variable to 1 if webSMIT requires secure http access
# comment the line out if standard http access is allowed
# the variable will have no effect if AUTHORIZED_PORT is set
# SECURITY ALERT: if secure http is not used, sensitive data
# will be transmitted in the clear
#REDIRECT_TO_HTTPS=1
Note: The comment that "the variable will have no effect if
AUTHORIZED_PORT is set" does not appear to be correct.
__ 8. WebSMIT provides a convenient way to view both HTML and PDF versions of the
HACMP documentation. Add a link for the HACMP documentation.
# ln -sf /usr/share/man/info/en_US/cluster/HAES ./htdocs/en_US/HAES
Note: Remember, you must have installed the documentation packages in order for
WebSMIT to serve them. Also, note that Acrobat Reader must be installed on the
system where you are running your browser to display the PDF versions of the
documentation.
Check to see if the documentation packages are installed and if not, install them.
# lslpp -L| grep doc

cluster.doc.en_US.es.html 5.3.0.0 C F HAES Web-based HTML
cluster.doc.en_US.es.pdf 5.3.0.0 C F HAES PDF Documentation - U.S.
__ 9. Start the HTTP server:

# /usr/HTTPServer/bin/apachectl start
__ 10. Now were ready to give WebSMIT a try. Point your browser to:
http://IP_label:42267
where IP_label is the ServerName you set in step __ 5. __ a.
__ 11. You should see a login screen. Login as root and experiment with WebSMIT.
Notice the navigation aids in the bottom of the screen. Be sure the check out the
Tree View link, which allows you to quickly jump from one part of WebSMIT to
another.
__ 12. We suggest that you experiment with using both the command line SMIT and
WebSMIT as you perform the rest of this exercise.
Part 2: Create a cluster snapshot

Since we have established the cluster the way we want it for this class, we will take a
snapshot now so that we can restore it to this configuration later.
The Cluster Snapshot utility allows you to save cluster configurations you would like to
restore later.
Notes:
You can create your own custom snapshot methods to store additional system and
cluster information that may be useful.
You also have the option to include cluster log files. Including the log files increases
both the size of the snapshot and the running time of the snapshot utility.
Its good practice to give your snapshot a meaningful name and description, so that you
can easily identify why or when the snapshot was taken should a cluster restore be
required. The snapshot name is required; the description is optional.
__ 13. Use the HACMP Extended Configuration menu to create a cluster snapshot.
For this exercise, do not include the log files and do not create and use a custom
snapshot method.
Give your snapshot a meaningful name, like Exercise6snap, and a short
description, like mutual takeover config.
# smitty hacmp
Extended Configuration -> Snapshot Configuration ->
Add Cluster Snapshot
Enter the Cluster Snapshot Name and Cluster Snapshot Description.
Allow the other fields to default.

V3.1.0.1
EXempty Notes:
The snapshot utility runs the clsnapshot command. Two files are created on the
node where the snapshot utility was run:
- snapshot_name.odm
This file contains the current HACMP for AIX ODM class objects.
- snapshot_name.info
This file contains information useful for problem determination in an HACMP
cluster. To populate this file, clsnapshot runs the cllscf, cllsnw, cllsif,
and clshowres commands on the node where the snapshot utility was run
and clsnapshotinfo on both nodes.
By default, the snapshot files are stored in the /usr/es/sbin/cluster/snapshots
directory. You can change this by setting the environment variable
SNAPSHOTPATH.
__ 14. Read the snapshot_name.info file. Go on to the next step when you are ready.
Part 3: Add an additional service alias and resource group to each

cluster node
We now add two resource groups appC_group and appD_group with different startup
policies.
appC_group will be configured so that it starts on the home node and fallsback to the home
node after a fallover and recovery (like the old inactive takeover).
appD_group will be configured using the Online Using Distribution Policy startup
policy: start on first available node that is not running another resource group using Online
Using Distribution Policy and never fallback.
(This is somewhat like the old rotating. Note that in earlier versions of HACMP version 5, it
was possible to select either a node-based or network-based distribution policy.
Node-based is the only choice in HACMP V5.3.)
__ 15. Log in to the t#halifax node as root.
__ 16. Add two additional service labels to the /etc/hosts file and ftp it to your second
node.
192.168.#3.21 t#appC-svc
192.168.#3.22 t#appD-svc
__ 17. Configure the two additional HACMP Service IP Labels/Addresses as resources in
HACMP.
# smitty HACMP
Extended Configuration -> Extended Resource Configuration ->
HACMP Extended Resources Configuration ->
Configure HACMP Service IP Labels/Addresses ->
Add a Service IP Label/Address -> Configurable on Multiple Nodes
Select the network name.

Use F4 (ESC-4) to select the t#appC-svc IP label in the IP Label/Address
field and execute.
Repeat for t#appD-svc.
Note: A value for the Alternate Hardware Address to accompany IP
Label/Address field may not be selected. Remember that HWAT is not supported
for networks using IPAT via IP Aliasing.
__ 18. Add an additional resource group called appC_group with a home node of
t#halifax and a startup policy of Online on First Available node.
Return (F3) to the Extended Resource Configuration menu
Select HACMP Extended Resource Group Configuration -> Add a
Resource Group
Enter:
Resource Group Name:
appC_group
Participating Nodes (Default Node Priority):
both nodes with t#halifax first
Startup Policy:
Online On First Available Node
__ 19. Add another resource group called appD_group with a home node of t#toronto and
a startup policy of Online Using Distribution Policy.
Return (F3) to the HACMP Extended Resource Group Configuration menu
Select Add a Resource Group
Enter:
Resource Group Name:
appD_group
Participating Nodes (Default Node Priority):
both nodes with t#toronto first
Startup Policy:
Online Using Distribution Policy
Fallback Policy:
Never Fallback
Note: A fallback policy of Never Fallback is required if using a startup policy of
Online Using Distribution Policy.
__ 20. Now add the t#appC-svc and t#appD-svc IP Labels to the new resource groups.
Using the F3 key, traverse back to the Extended Resource Group Configuration
menu and select Change/Show Resources and Attributes for a Resource
Group.

V3.1.0.1
EXempty Return (F3) to the HACMP Extended Resource Group Configuration

menu.
Select Change/Show Resources and Attributes for a Resource Group.
Select appC_group from the list.
Use F4 to select t#appC-svc in the Service IP Labels/Addresses field and
press Enter.
Repeat for appD_group and t#appD-svc.
__ 21. Stop the cluster so that we can observe startup behavior.
Exit SMIT
# smitty clstop
Select both nodes and press Enter
Exit SMIT
__ 22. Verify that the cluster is stopped on both nodes.
It may take several minutes for the cluster to stop. Its a good idea to verify the
status of cluster services before proceeding.
To verify that cluster services are stopped, we need to get the status of the cluster
manager daemon (clstrmgrES). In earlier versions of HACMP, you could use
lssrc -s clstrmgrES or
lssrc -g cluster
to verify if the cluster was running. When clstrmgrES was active, the cluster was
running; when it was inoperative, cluster services were stopped.
In HACMP version 5.3, this doesnt work since the clstrmgrES daemon is always
running. There are now several way to get cluster status. Try each of them.
# lssrc -ls clstrmgrES | grep state
This command must be run on each node where you want to check status. You
are looking for the ST_INIT state of the cluster manager daemon. It takes a
minute or so for clstrmgrES to reach the ST_INIT state. You may see some
other states on one or both nodes as the daemon stops services.
# lssrc -a | grep svc
This command must be run on each node. It is a quick way to get status on the
group services daemon (grpsvcs). The Cluster Manager is a client of Group
Services, and uses it for reliable inter-daemon communication. If grpsvcs is
inoperative, the cluster manager is inactive.
# clcheck_server daemon
The clcheck_server command returns the status of the named daemon on the
node where it is run. It returns 0 if the daemon is running. This command is
intended for use within shell scripts which need to reliably determine the status of
a daemon. Note that this utility makes additional checks and retries beyond what
is done by the lssrc query command provided by SRC.
To check the status of the cluster manager, you can use the following script (from
the clcheck_server man page) to check the state of grpsvcs, which the Cluster
Manager uses. If grpsvcs is down, the Cluster Manager is inactive. This script
must be run on each node where you want to check status.
if ! clcheck_server grpsvcs
then
echo "HACMP cluster services are active"
else
echo "HACMP cluster services are inactive"
fi
Note: At the time of this writing, clcheck_server clstrmgrES always returns 0
(active), so you must run this command on grpsvcs to determine cluster status
at this time.
# cldump
This command can be run on any node, even if the cluster is down on that node.
If cluster services are running on any node, cldump returns detailed status on all
cluster nodes, networks, interfaces and resource groups.
If cluster services are not running on any node, cldump returns an error.
__ 23. Synchronize the cluster using the Extended Verification and Synchronization
menu.
Using the F3 key, traverse back to the Extended Configuration menu.
Select Extended Verification and Synchronization
Notice the menu option Automatically correct errors found during
verification. You only see this option when the cluster is down.
Part 4: Test startup policy behavior

Now well start the cluster one node at a time to see the difference in the startup policies
weve selected.
__ 24. Start HACMP only on your t#toronto node.
# smitty clstart -- or use the C-SPOC menu Manage HACMP Services
Select t#toronto only.
__ 25. Once the node is stable, check the status of the resource groups. Does this look
normal? If not, what is wrong -- Should appC_group be online on t#toronto?
# clRGinfo
# clRGinfo -v appC_group
Note: You can also use cldump to get information about resource group
status.

V3.1.0.1
EXempty Actually this is normal, but it might not be what you expect. Since t#halifax
is the highest priority node, you might expect appC_group to not start until
t#halifax comes up. But we set the startup policy to Online on First
Available node, so it starts on t#toronto if it comes up first.
__ 26. Start HACMP on the t#halifax node.
Refer to step __ 24. hints
__ 27. Once the node is stable, check the status of the resource groups. Which node has
the appC_group resource group now? Why?
The resource group should have moved to t#halifax.
Since appC_group has the default fallback policy (Fallback To Higher
Priority Node In The List), HACMP will transition the resource group to
t#halifax when it comes up.
To better understand the startup policies, well bring the cluster services up and down a few
more times.
__ 28. Stop the nodes and bring up t#halifax only. What happens to appC_group and
appD_group?
# smitty hacmp
System Management (C-SPOC) -> Manage HACMP Services -> Stop
Cluster Services
Select both nodes.
Return to the Manage HACMP Services menu.
Select Start Cluster Services.
Select t#halifax only.
The resource groups should have started on t#halifax.
__ 29. Start HACMP on t#toronto. What happens to appC_group and appD_group?
Select Start Cluster Services.
Select t#toronto only.
appC_group: Since t#halifax is the higher priority node, the resource group
does not transition.
appD_group: Since the fallback policy is Never Fallback, the resource group
does not transition.
__ 30. Stop t#halifax with Shutdown mode of takeover. Verify that appC_group and
appD_group are running on t#toronto.
Select Stop Cluster Services.
Select takeover in the Shutdown mode field.
Both resource groups should have moved to t#toronto.

__ 31. Restart t#halifax. What happens to appC_group and appD_group?
Return to the Manage HACMP Services menu. Select Start Cluster
Services.
Since t#halifax is the higher priority for appC_group, appC_group should
have moved to t#halifax.
appD_group should have stayed on t#toronto (fallback policy = Never
Fallback).
Part 5: Gratuitous ARP

In Part 5:, well verify that gratuitous ARP does indeed work when using IPAT via
replacement with AIX clients.
From the AIX 5L command reference for ifconfig:
"Gratuitous ARP is supported for Ethernet, token-ring, and FDDI interfaces. This means
when an IP address is assigned, the host sends an ARP request for its own address (the
new address) to inform other machines of its address so that they can update their ARP
entry immediately. It also lets hosts detect duplicate IP addresses."
__ 32. Log on the client machine. Verify that clinfo has not been started.
# ps -ef | grep -i clinfo | grep -v grep
__ 33. In order to observe the action of gratuitous ARP, there must already be an ARP
entry for the server on the client system. If there is no entry for the IP address in the
ARP cache, the gratuitous ARP packet is ignored. By default, entries in the ARP
cache time-out in 20 minutes from when the entry was first created.
To make sure that you have time for this test, use arp -d, on the client, to delete the
ARP cache entry for t#appA-svc, if it exists. Then use ping to recreate the entry
(restart the 20 minute time-out). Finally use arp -a to verify that there is an entry for
t#appA-svc.
# arp -a | grep appA
# arp -d t#appA-svc
# ping t#appA-svc
__ 34. Generate a swap adapter event on the t#halifax node.
Use ifconfig to bring the interface which has the service label down. HACMP will
move the service label to the other interface and AIX will send a gratuitous ARP
packet. This should cause the ARP cache on the client to be updated.

V3.1.0.1
EXempty # ifconfig enX down

where enX is the interface on which the t#appA-svc service label is
configured
__ 35. Check the contents of the ARP cache on the client, compare the results with the
previous iteration of the command.
Note: If using POWER-5 based LPARs: The hardware addresses of virtual Ethernet
adapters for LPARs in the same POWER-5 based frame typically only differ by one
byte.
__ 36. The hardware address should have been updated in the ARP cache on the client
without any intervention.
Part 6: Re-create a cluster from a snapshot

__ 37. Use the HACMP Extended Configuration menu to create a cluster snapshot.
For this exercise, do not include the log files and do not create and use a custom
snapshot method.
Give your snapshot a meaningful name, like Exercise7snap, and a short
description, like appC-appD config.
# smitty hacmp
Extended Configuration -> Snapshot Configuration ->
Add Cluster Snapshot
Enter the Cluster Snapshot Name and Cluster Snapshot Description.
Allow the other fields to default.
__ 38. Stop the http server on t#halifax.
# /usr/HTTPServer/bin/apachectl stop
__ 39. OPTIONAL: At this point, the HTTP servers on your two nodes are configured
differently. If you want them to be the same, you can:
Configure t#toronto so that it will run WebSMIT (see steps __ 3. on page 3
through __ 8.).
Alternatively, on t#halifax, you can restore the httpd.conf file that you saved in
step __ 2. on page 3
__ 40. Ensure the cluster services are stopped on both clusters nodes.
# smitty clstop
choose both nodes
__ 41. Apply the snapshot that you created in Part 2:, Create a cluster snapshot on page
6 (Exercise6snap), which contains all the cluster definitions you created in exercise
6.
Login to t#halifax (snapshots are stored on the node where they were
created)
# smitty hacmp
Extended Configuration -> Snapshot Configuration -> Apply a
Cluster Snapshot
__ 42. Start HACMP
# smitty clstart
choose both nodes
__ 43. For each resource group, verify to yourself that you understand how the online node
was chosen.
# clRGinfo
__ 44. Fail the t#halifax node in your favorite manner.
Note: If you use the halt command, make sure that you will be able to restart the
node. (If you are using an HMC-connected system, this means you must be able to
connect to the HMC.) Alternatives, which simulate a failed node from the point of
view of HACMP, are:
shutdown -Fr (fast reboot)
smitty clstop, selecting takeover as the Shutdown mode
# halt -q
OR
# shutdown -Fr
OR
# smitty clstop
select takeover.
__ 45. Restart the failed node and observe the re-integration. Verify that you understand
how the online node was chosen for each of the resource groups.
END OF LAB

V3.1.0.1
EXempty Part 7: OPTIONAL: Using a settling timer

In this part, well investigate using a settling timer. Using a settling timer can be useful if you
are using Online on First Available Node together with Fallback to Higher
Priority Node in the List.
You might want to use a startup policy of Online on First Available Node with a
fallback policy of Fallback to Higher Priority Node in the List if you have a
preferred node, but want your application to start on the other node if the preferred node is
down. But what about the case when you are bringing up both your nodes and the
secondary node comes up first? In this case, there will be two online events. The resource
group will come up on the secondary node and then fallover when the primary node comes
up. (See steps __ 25. through __ 27. above.)
Configuring a settling timer allows you to modify the behavior of the startup policy of
Online on First Available Node so that there are not two online operations if you bring
up the secondary node first.
__ 1. Apply the snapshot that you created in step __ 37. on page 13 (Exercise7snap),
which contains all the cluster definitions you created in exercise 7.
created)
# smitty hacmp
Cluster Snapshot
__ 2. Ensure that you are on your administration node (t#halifax) and configure a
Settling Timer (can only be used if the startup policy is Online On First Available
Node).
# smitty hacmp
Configure Resource Group Run-Time Policies -> Configure Settling
Time for Resource Groups
Set a value (in seconds) for the settling time. For this lab use 360.
__ 3. Synchronize the Cluster. Notice the messages about the settling time value in the
SMIT output.
Using the F3 key, traverse back to the Extended Configuration SMIT
screen.
Select Extended Verification and Synchronization.
__ 4. On your administrative node (t#halifax), stop cluster services on both nodes.
# smitty clstop
choose both nodes
__ 5. Wait 2 minutes and then start HACMP -- only on the t#toronto node.
# smitty clstart
choose only t#toronto
__ 6. Using the clRGinfo command, wait until you can see that appB_group is online and
appC_group is still offline on t#toronto. Then start HACMP on the t#halifax node.
Note: The clRGinfo command can be run from either node as long as HACMP is
started on any one of the nodes.
In another window on t#halifax or on t#toronto:
# while true
do
clRGinfo
sleep 5
done
Back in first t#halifax window:
# smitty clstart
choose only t#halifax
__ 7. Verify that the appC_group comes online on t#halifax (without first being online on
t#toronto). As you can see, the purpose of the settling timer is to prevent the
resources from being immediately acquired by the first active node.
__ 8. OPTIONAL -- Repeat this part but wait for settling time to expire after starting the
cluster on t#toronto. Verify that appC_group comes online on t#toronto. Stop the
cluster manager on both nodes, wait 2 minutes, start the cluster manager on both
nodes.
__ 9. If you ARE going on the next OPTIONAL exercise part, you can skip this step.
If you are NOT going on to the other OPTIONAL exercises, restore the snapshot
that you created in Part 2:, Create a cluster snapshot on page 6 (Exercise6snap),
created)
# smitty hacmp
Cluster Snapshot
Part 8: OPTIONAL: Using a delayed fallback timer

The delayed fallback timer lets a resource group fall back to the higher priority node at a
time that you specify. The resource group that has a delayed fallback timer configured and
that currently resides on a non-home node falls back to the higher priority node at the
recurring time (daily, weekly, monthly or yearly), or on a specified date.
In this part we will create and test a delayed fallback timer.

V3.1.0.1
EXempty __ 10. Cluster should be started on both nodes and appC_group should be online on
t#halifax.
# cldump | grep "Node Name"
# clRGinfo
__ 11. On your administrative node (t#halifax), create a delayed fallback timer policy for
30 minutes from now (instructor may modify this time)
Make sure both nodes are using the same time:
# setclock t#toronto-if1
Write down the current time ______________
# smitty hacmp
Configure Resource Group Run-Time Policies -> Configure Delayed
Fallback Timer Policies -> Add a Delayed Fallback Timer Policy
use the following values:
daily
Name of the Fallback Policy = my_delayfbt
HOURS and MINUTES = 30 min from current time
__ 12. Add the fallback timer policy to the resource group appC_group
# smitty hacmp
HACMP Extended Resource Group Configuration -> Change/Show
Resources and Attributes for a Resource Group
Select appC_group
Fill in the field below as indicated:
Fallback Timer Policy [my_delayfbt] (use F4)
__ 13. Synchronize.
Next well test the delayed fallback timer.
__ 14. Verify that appC_group is online on t#halifax using the clRGinfo command
# clRGinfo appC_group
__ 15. Stop cluster services only on t#halifax with takeover.
# smitty clstop
__ 16. Verify that appC_group is now online on t#toronto (clRGinfo).
__ 17. Wait 2 minutes (required before a restart)
__ 18. Start the cluster manager on t#halifax
# smitty clstart
__ 19. Monitor the cluster from t#toronto. In /tmp/hacmp.out at the event summary for
check_for_site_up_complete t#halifax, there is now a message stating the
fallback time. Make sure appC_group is still on t#toronto before the fallback, then
use tail -f hacmp.out file and wait for the fallback to occur.
# vi /tmp/hacmp.out
Look for the fallback time message-- should be very near the bottom
# clRGinfo
verify appC_group is online on t#toronto -- before the fallback time
# tail -f /tmp/hacmp.out
and wait
__ 20. At the time set for the Delayed Fallback Timer, appC_group should move back to
t#halifax (you should see activity from the tail command). Verify that
appC_group has moved to its home node (t#halifax).
Execute clRGinfo to verify that appC_group is online on t#halifax.
__ 21. On your administrative node (t#halifax), remove the name of the Delayed Fallback
Timer (my_delayfbt) from the resource group appC_group (you can keep the policy
definition if you want).
# smitty hacmp
__ 22. Reset the Settling time to 0 (from the Configure Resource Group Run-Time
Policies menu)
# smitty hacmp
Configure Resource Group Run-Time Policies -> Configure Settling
Time for Resource Groups
__ 23. Synchronize.
created)
# smitty hacmp
Cluster Snapshot

V3.1.0.1
EXempty Part 9: OPTIONAL: Recreate the cluster and configure IPAT via
replacement
In a previous exercise, you created a cluster using the Two-Node Cluster Configuration
Assistant. In this part, we will remove the current cluster (after first taking a snapshot) and
then create a new cluster manually. Well configure the new clusters network for IPAT via
replacement, which we will be investigating in Part 10: of this exercise.
__ 25. On your administration node (t#halifax), stop both the cluster nodes.
# smitty clstop
choose both nodes
__ 26. Create a snapshot of the current cluster.
Note: If you just made a snapshot in Step __ 37. on page 13, you can skip this step.
# smitty hacmp
Extended Configuration -> Snapshot Configuration -> Add Cluster
Snapshot
Enter snapshot name and description.
Example:
Cluster Snapshot Name = exercise 7
Cluster Snapshot Description = added C & D resource groups
__ 27. Remove the current cluster configuration.
# smitty hacmp
Extended Configuration -> Select Extended Topology Configuration
-> Configure an HACMP Cluster -> Remove an HACMP Cluster
REPEAT on the other node
Clear the HACMP rhosts file
# echo > /usr/es/sbin/cluster/etc/rhosts
(double quotes with no space)
REPEAT on the other node
__ 28. Add a replacement service address to your /etc/hosts files:
192.168.#1.10 t#appR-repl
(must be on the same subnet as one of the t#hostname-ifX interfaces).
# vi /etc/hosts
add
192.168.#1.10 t#appR-repl
(# is your team number - remember?)
Document this address in you component worksheets (exercise 2).
ftp to your other node and to the client.
__ 29. Configure a new cluster on t#halifax. Go to the HACMP for AIX SMIT panel and
select Initialization and Standard Configuration -> Add nodes to an
HACMP Cluster. Add your two nodes using the two -if1 interfaces.
# smitty hacmp
Initialization and Standard Configuration -> Add Nodes to an
HACMP Cluster
Enter a cluster name (for example, team#).
Using F4, select the appropriate communications paths for BOTH the nodes
(that is, the two -if1 interfaces).
__ 30. Use Extended Configuration to remove the classroom network (t#toronto and
t#halifax interfaces).
# smitty hacmp
Extended Configuration -> Extended Topology Configuration ->
Configure HACMP Networks -> Remove a Network from the HACMP Cluster
Select the classroom network (not the network with the 192.168.21 and
192.168.22 subnets)
__ 31. Use Extended Configuration to set the network to turn off IPAT via aliases.
Configure HACMP Networks -> Change/Show a Network in the HACMP
Cluster
Select the 192.168 network, press Enter
Change Enable IP Address Takeover via IP Aliases to No;
press Enter.
__ 32. Use Extended Configuration to configure a non-IP network by choosing the pair
of devices that will make up the network.
Using the F3 key, traverse back to the Extended Topology Configuration SMIT
screen.
Configure HACMP Communication Interfaces/Devices -> Add
Communication Interfaces/Devices
From the list, select, Add Discovered Communication Interface and
Devices
From the list select, Communication Devices
Select the appropriate pair of devices:
hdisk/hdisk or tty/tty or tmssa/tmssa

V3.1.0.1
EXempty __ 33. Redo the Persistent Addresses from your planning worksheet.
Using the F3 key, traverse back to the Extended Topology Configuration SMIT
screen.
Extended Topology Configuration -> Configure HACMP Persistent
Node IP Label/Addresses -> Add a Persistent Node IP Label/Address
Select t#halifax from the list
Use F4 to select both the Network Name and Persistent IP Label/Address
for this node and press Enter
REPEAT for your other node
__ 34. Create the new Service IP Label resource: t#appR-repl.
Using the F3 key, traverse back to the Extended Configuration SMIT screen.
Select Extended Resource Configuration. Select Configurable on Multiple
Nodes when asked to Select a Service IP Label/Address type.
HACMP Extended Resources Configuration -> Configure HACMP
Service IP Labels/Addresses -> Add a Service IP Label/Address
Select Configurable on Multiple Nodes
Select the Network Name.
Finally, use F4 to select the Service IP Label (t#appR-repl) to be used.
__ 35. Create the appR_group resource group. Set t#halifax as the home node and use
the default policies.
Using the F3 key, traverse back to the Extended Resource Configuration SMIT
screen.
Extended Resource Configuration -> HACMP Extended Resource Group
Configuration -> Add a Resource Group
Enter the Resource Group name (appR_group) and set the participating
nodes (use F4 to choose the nodes). Remember the priority order, the first
node listed, is considered the home or owner node.
Use the default policies
__ 36. Add Resources to the resource group. Add the t#appR-repl Service IP Label and a
Volume Group (either appA_vg or appB_vg). For the purposes of this lab, application
servers are not required. You may add them if you wish.
Using the F3 key, traverse back to the HACMP Extended Resource Group
Configuration SMIT screen.
Resources and Attributes for a Resource Group ->
a Resource Group
Select appR_group from the list
In the appropriate fields, use F4 to choose the t#appR-repl Service IP Label

and a Volume Group.
__ 37. Synchronize the cluster.
Extended Configuration -> Extended Verification and
Synchronization
Choose the defaults and press Enter.
Review the results for any errors.
__ 38. Start HACMP on the t#toronto node and start clinfo.
Monitor the /tmp/hacmp.out file during startup.
# smitty clstart
choose only t#toronto and start clinfo
__ 39. Verify the appR_group did not come online because of the startup policy.
# clRGinfo
__ 40. Start HACMP on the t#halifax node and start clinfo.
# smitty clstart
choose only t#halifax and start clinfo
__ 41. Verify that the appR_group is online on t#halifax
# clRGinfo
__ 42. Since we are now using IPAT via replacement, verify that the t#appR_repl service
IP label/address has replaced one of the interface addresses on t#halifax, instead
of being aliased onto the interface.
# netstat -i
created)
# smitty hacmp
Cluster Snapshot
Part 10: OPTIONAL: Hardware address takeover

In this part, well look at hardware address takeover (HWAT).

V3.1.0.1
EXempty As discussed in Unit 3, HWAT is one of several options if you have clients (or routers)
which do not support gratuitous ARP. Since AIX clients actually do support gratuitous ARP,
it is not possible to create this type of problem on the client in this exercise. However, we
will configure the cluster network for HWAT and observe that the hardware address does
indeed follow the service label when the resource group fallsover.
For details about HWAT, see Appendix C in the student guide.
In this scenario, we assume the router in Regina is a bit of an antique and does not support
gratuitous ARP. It was highlighted as a problem since the ARP cache retention is 20
minutes. This problem was discovered during the preliminary cluster testing.
__ 44. On the t#halifax node, log in as root.
__ 45. Identify the interface that is reconfigured with the t#appR-repl service address and
write the MAC address here:
_____________________________________.
For our purposes, there should be 12 digits and no periods. When using the
netstat command, leading 0s before each period are omitted. You must put them
back in.
# netstat -i
__ 46. Identify an alternate mac address. To specify an alternate hardware address for the
Ethernet interface, add 4 to the 1st byte (xy) to [x+4}y:
_________________________________________
__ 47. Change the t#appR-repl service IP label to add an alternate hardware address.
Using Extended Configuration, navigate to Configure Service IP
Labels/Addresses.
# smitty hacmp
HACMP Extended Resources Configuration -> Configure Service IP
Labels/Addresses -> Change/Show a Service IP Label/Address
Select t#appR-repl
Enter the Alternate Hardware Address using the answer to the previous
step.
__ 48. Synchronize the cluster.
This will cause the resource group to go off line: notice the following message in the
SMIT command output window:
cldare: Detected changes to service IP label t2appR-repl. Please note
that changing parameters of service IP label via a DARE may result in
releasing resource group appR_group.
Extended Configuration -> Extended Verification and

Synchronization
# clRGinfo
Shows appR_group offline. This will take a couple of minutes.
__ 49. Bring the appR_group online using the C-SPOC menu. If, on the client, there is no
ARP cache entry for the t#appR-repl service address, then ping the t#appR-repl
service address.
# smitty hacmp
System Management (C-SPOC) -> HACMP Resource Group and
Application Management -> Bring a Resource Group Online
Select appR_group OFFLINE
BE CAREFUL -- Select Restore_Node_Priority_Order
Accept all fields in the next menu
__ 50. Verify that the alternate hardware address is now configured on the interface for the
t2#appR-repl service address.
Note: It will take a couple of minutes for HACMP to make the change.
# netstat -i
__ 51. Fail the t#halifax node in your favorite manner.
# halt -q
OR
# shutdown -Fr
# smitty clstop
select takeover.
__ 52. Logon to t#toronto and check that the t#appR-repl service address is now on the
t#toronto node and observe that the alternate hardware address associated with
that service address has moved to t#toronto.
# netstat -i
__ 53. Apply the snapshot that you created in Part 2:, Create a cluster snapshot on page
6 (Exercise6snap), which contains all the cluster definitions you created in exercise
6.
created)
# smitty hacmp
Cluster Snapshot
END OF LAB

V3.1.0.1

In this exercise we looked at:
Enabling WebSMIT
Creating a cluster snapshot
Adding additional service addresses
Testing startup policy behaviors
Testing gratuitous ARP
Re-creating a cluster from a snapshot
OPTIONAL: Using a settling timer
OPTIONAL: Using a delayed fallback timer
OPTIONAL: Configuring IPAT via replacement
OPTIONAL: Configuring HWAT

V3.0
EXempty Exercise 8. Events

(with Hints)

This lab covers the adding of error notifications into AIX through the
HACMP smit screens.

Describe the event flow for starting a cluster
List the automatic events that HACMP creates
Optionally add another error notification based on filesystems full
Copyright IBM Corp. 1998, 2005 Exercise 8. Events 8-1


Preface
This exercise looks at Event flow and Error Notification. Remember that Error
Notification is a function of AIX - HACMP just gives you the smit screens that make it
easier to enter error notification methods.
Part 1: Looking at the event flow.

__ 1. Establish a console session (login) as root on the t#halifax node.
__ 2. Stop cluster services on both nodes (graceful).
smitty clstop
__ 3. Remove the /usr/es/adm/cluster.log and /tmp/hacmp.out files.
cd /usr/es/sbin/cluster/events
__ 4. There is a node_up.rp file. Take a look at its contents. This file coordinates the
running of node up on all the cluster nodes. Now lets look more closely at the
node_up process
more node_up.rp
__ 5. Start cluster services on both nodes.
smitty clstart
__ 6. Examine the event flow on t#halifax using the cluster.log file (grep for EVENT) then
answer the next 3 questions.
cat /usr/es/adm/cluster.log | grep EVENT | more
__ a. During what event was the acquire_service_address event run?
Answer:
node_up
__ b. During what event was the start_server event run?
Answer:
node_up_complete
__ c. What event indicates that cluster services is finished starting?
Answer:
node_up_complete
__ 7. Examine the /tmp/hacmp.out file and answer the following question:
During what event were the disks acquired? (use hint to search hacmp.out easily)
more /tmp/hacm.out
/disk

V3.0
EXempty /EVENT
See the event, the event summary and the search string
__ 8. There is a summary log file for hacmp.out. Take moment to see how you can go
from an entry in the hacmp.out summary to the location of the event in the
hacmp.out file using the search argument.
smitty hacmp
Problem Determination Tools ->HACMP Log Viewing and Management
->View/Save/Delete/HACMP Event Summaries
Part 2: Looking at error notification

__ 9. Configure Automatic Error Notification. In HACMP 5.3 Automatic Error configuration
is automatically configured during synchronization. Use odmget errnotify.
When you run automatic error notification, it assigns two error methods for all the error
types noted:
cl_failover is assigned if a disk or network interface card is determined to be a
single point of failure, and that failure would cause the cluster to fall over. If there
is a failure of one of these devices, this method logs the error in hacmp.out and
shuts the cluster node down. A graceful stop is attempted first, if this is
unsuccessful, cl_exit is called to shut down the node.
cl_logerror is assigned for any other error type. If there is a failure of a device
configured with this method, they are logged in hacmp.out.
__ 10. List Error Notification Methods (use Change/Show Error Methods under Error
Notification under Problem Determination in smitty hacmp).
END OF LAB
Copyright IBM Corp. 1998, 2005 Exercise 8. Events 8-3


V3.1.0.1
EXempty Exercise 9. Network File System (NFS)

(with Hints)

This lab covers a couple of different methods for configuring network
file systems with HACMP. It also demonstrates how to set various NFS
options in HACMP exported file systems.

Have HACMP export a file system as part of a resource group
Have HACMP import a file systems as part of resource group
Modify the NFS export options for the exported file system
Add an NFS cross-mount
Modify the NFS cross-mount for performance and flexibility
Copyright IBM Corp. 1998, 2005 Exercise 9. Network File System (NFS) 9-1

Preface
All exercises of this chapter depend on the availability of specific equipment in your
classroom.
Part 1: NFS exports in a resource group

__ 1. Assumptions: You need to start this exercise off with HACMP up on both nodes, and
identify two resource groups -- one whose home node is t#halifax (that is,
appA_group) and the other whose home node is t#toronto (that is, appB_group).
Each group should have a shared file system defined to it (that is, /appA_fs and
/appB_fs). On each node, verify that NFS is running (lssrc -g nfs) after HACMP
is started (biod, rpc.statd and rpc.lockd should be running).
__ 2. Modify the appA_group resource group to add /appA_fs as a file system/directory to
NFS export and set the option Filesystems mounted before IP configured to
true.
# smitty hacmp
Select the appA_group resource group from the list, press Enter
Set Filesystems mounted before IP configured to true
Set Filesystems/Directories to Export to /appA_fs
__ 3. Modify the resource group appB_group to add /appB_fs as a file system/directory to
NFS export.
Using the F3 key, traverse back to the HACMP Extended Resource Group
Configuration.
See the previous step.
__ 4. Synchronize the resources.
screen.
Select Extended Verification and Synchronization
__ 5. When the reconfiguration of resources has completed on each node, check the
directories are exported through NFS.
# lsnfsexp
You see only what is exported from THIS node

V3.1.0.1
EXempty Look at the contents of /etc/xtab. It should look like the output from lsnfsexp.
Note: /etc/exports is not used by HACMP-- see what happens when you try to cat
/etc/exports
# cat /etc/xtab
# cat /etc/exports
__ 6. Log in on the client as root.
__ 7. Create two directories: /halifax and /toronto.
# mkdir /halifax
# mkdir /toronto
__ 8. On the client, using the service address for the appA_group, mount the NFS
exported directory /appA_fs on the local directory /halifax.
# mount t#appA-svc:/appA_fs /halifax
OR
# smitty mknfsmnt
__ 9. On the client, using the service address for the appB_group, mount the NFS
exported directory /appB_fs on the local directory /toronto.
Choose sys in the security method field.
# mount t#appB-svc:/appA_fs /halifax
OR
# smitty mknfsmnt
__ 10. Verify the NFS directories are mounted where intended.
# mount
# df -k
__ 11. Back on a cluster node -- fail one of the nodes in your favorite manner. Verify that
both NFS directories are exported on the remaining node and still mounted on the
client system.
# halt -q
# lsnfsexp
on the remaining cluster node
# df -k
on the client system
__ 12. Try to create a file in the /halifax directory. It should not work. Lets see how this can
be addressed.
# touch /halifax/junk
Part 2: Modifying the NFS Export Options

__ 13. Check output of the lsnfsexp command on the nodes. Notice that only the cluster
nodes can use user root. To change this we create an override file. Its name is
/usr/es/sbin/cluster/etc/exports. You can use this file to specify specific NFS
options for file systems NFS exported by HACMP.
__ 14. On the running node, use the lsnfsexp command to copy the current /etc/xtab file
to the HACMP file and then modify the HACMP file using the following commands:
# lsnfsexp > /usr/es/sbin/cluster/etc/exports
Edit /usr/es/sbin/cluster/etc/exports and add the client to the list of hosts. Save
the file.
If you are accessing the lab systems remotely, the client will have two
interfaces. Be sure to use the client IP label that is on the same subnet as the
service IP labels.
ftp the file to the other node
__ 15. Restart the failed node. The HACMP exports file should be used when NFS
remounts the directory.
__ 16. From the client try to create a file in the NFS directory on the client of the node you
have just restarted.
Use the touch command
Part 3: NFS Cross-mount within the Cluster

__ 17. On both nodes create a directory /hanfs.
# mkdir /hanfs
__ 18. Edit the appA_group resource group so that the /appA_fs NFS file system will be
NFS mounted on the /hanfs mount point for all nodes in that resource group.
# smitty hacmp
Select the appA_group resource group from the list, press Enter
Set
Filesystems/Directories to NFS mount:/hanfs;/appA_fs
__ 19. Synchronize the resources and verify that /appA_fs is NFS mounted on /hanfs on
both nodes.
screen.

V3.1.0.1
EXempty Select Extended Verification and Synchronization

__ 20. Fail the t#toronto node in your favorite manner.
# halt -q
__ 21. Confirm that t#halifax node has all the resource groups, and that the NFS mounts
are OK.
# clRGinfo
# df -k
END OF LAB
This exercise looked at various methods of implementing NFS in an HACMP cluster.

V3.0
EXempty Exercise 10. HACMP Problem Determination

(with Hints)

This lab exercise will explore some of the problem determination tools.

Set up automatic cluster configuration monitoring
Use the Cluster Test Tool
Configure Automatic Correction
Introduction
This exercise will give you practice at implementing features
discussed in the Problem Determination Unit of the course.
Copyright IBM Corp. 1998, 2005 Exercise 10. HACMP Problem Determination 10-1

Preface
Part 1: Use the HACMP Cluster Test Tool

__ 1. Use console session to log in to the t#halifax node as root.
__ 2. Stop cluster services on both nodes.
smitty clstop
__ 3. Ensure cluster services are stopped on both nodes.
/usr/es/sbin/cluster/utilities/clcheck_server grpsvcs ; $?
__ 4. Using smitty hacmp, navigate to Problem Determination Tools --> HACMP
Cluster Test Tool --> Execute Automated Test Procedure. Start HACMP on
the t#Halifax. Note: Remember that these tests are disruptive and will crash one of
the nodes.
Part 2: Setup Automatic Cluster Configuration Monitoring

__ 5. Start cluster services on t#halifax and create a new 50M file system /myfs in
appA_vg not using CSPOC and only on t#halifax
smitty jfs
__ 6. Use the date command to determine the current time.
__ 7. Use the HACMP Verification menu under Problem Determination Tools to set
automatic cluster configuration monitoring to the next possible hour.
NOTE If the next hour is more than 10 minutes from now, then AFTER setting this
value dont wait. Continue with part 3 in this exercise and you will finish waiting at
the end of this exercise.
smitty hacmp
Select, Problem Determination Tools
Select, HACMP Verification
Select, Automatic Cluster Configuration Monitoring
Set the hour for the next possible hour (use the date command to determine
this).
__ 8. When automatic verification runs you will see a console message on t#halifax. It
should look like Broadcast message from root@t#halifax (tty) at hh:mm:26 ...
clverify has detected cluster configuration errors on node t#halifax. Detailed clverify
output is available in standard clverify log on node t#halifax. You will examine the
clverify file in the Part 3 section below.

V3.0
EXempty Part 3: Use Extended Configuration Automatic Correction

__ 9. Ensure cluster services are stopped on both nodes.
/usr/es/sbin/cluster/utilities/clcheck_server grpsvcs ; $?
__ 10. Bring up the Extended Configuration ->Verification and Synchronization menu
and ensure Automatically correct errors found during verification is set
to interactively
__ 11. Do the synchronization. You will be prompted to import /myfs but say no so you can
examine the clverify log in the next step. Leave smit session now.
__ 12. Change directory to /var/hacmp/clverify and execute the command
more clverify.log Then search forwards for FAIL (upper case). Then search
backwards for NAME. This will bring you to the start of the routine that was doing the
check. Scroll forwards until you get to the end of this check.
cd /var/hacmp/clverity
more clverify.log
/FAIL
?NAME
When finished, end your more session
__ 13. If you did not wait for the automatic verification time to occur then wait now. When
automatic verification runs you will see a console message on t#halifax. It should
look like Broadcast message from root@hanode1 (tty) at hh:mm:26 ...
clverify has detected cluster configuration errors on node t#halifax. Detailed clverify
output is available in standard clverify log on node t#halifax.
END OF LAB
Copyright IBM Corp. 1998, 2005 Exercise 10. HACMP Problem Determination 10-3
This exercise looked at how to run the test tool and show that it is disruptive as well as how
to work with Automatic Verification and Auto Correct.

V3.1.0.1
EXempty Exercise 11. Migration (optional)

This exercise is optional. You can
- Take a snapshot and save it to an online planning worksheets file
- Remove the cluster, uninstall HACMP software
- Run the command rm -r /usr/es/*
- Install HACMP 5.2 from /ha53/hacode/ex11
- Configure it with the 2 node assistant
- Use the rolling method described in unit 11 to migrate to HACMP 5.3.
Copyright IBM Corp. 1998, 2005 Exercise 11. Migration (optional) 11-1

V3.1.0.1
AP Appendix A. Cluster Diagrams
Copyright IBM Corp. 1998, 2005 Appendix A. Cluster Diagrams A-1

Cluster Planning Diagram Client:

AU54 lab teams IP Label IP Address
if1 t#regina-if1 192.168.#1.3
REPLACE # with team number
alias 192.168.#3.30
team number= ______ if2 t#regina ___________
IP Label IP Address HW Address IP Label IP Address HW Address

if1 t#halifax-if1 192.168.#1.1 _____________ if1 t#toronto-if1 192.168.#1.2 _____________
if2 t#halifax-if2 192.168.#2.1 _____________ if2 t#toronto-if2 192.168.#2.2 _____________
if3 t#halifax ___________ _____________ if3 t#toronto ___________ _____________
Persist t#halifax-per 192.168.#3.1 Persist t#toronto-per 192.168.#3.2
Classroom network = ________________

classroom
Cluster network = 192.168.XX
systems
(netmask = 255.255.255.0
Home Node Name = t#halifax Home Node Name = t#toronto

Resource Group = appA_rg Resource Group = appB_rg
Startup Policy = OHNO Startup Policy = OHNO
Fallover Policy = FONP Fallover Policy = FONP
Fallback Policy = FBNF Fallback Policy = FBHP
Service IP Label = t#appA-svc Service IP Label = t#appB-svc
192.168.#3.10 192.168.#3.20
Application server = appA Application server = appB
serial
Label = t#halifax_hdiskX_01 Label = t#toronto_hdiskY_01
Device = /dev/hdiskX Device = /dev/hdiskX
serial
Label = t#halifax_tty0_01 Label = t#toronto_tty0_01
Device = /dev/tty0 Device = /dev/tty0
rootvg rootvg
VG =
VG =
Resource Group appA_group contains Resource Group appB_group contains

Volume Group = appA_vg Volume Group = appB_vg
hdisks = ______________ hdisks = ______________
Major # = ______________ Major # = ______________
JFS Log =appA_log_lv JFS Log =appB_log_lv
Logical Volume =appA_lv Logical Volume =appB_lv
FS Mount Point =/appA_fs FS Mount Point =/appB_fs
A-2 HACMP Implementation Copyright IBM Corp. 1998, 2005

V3.1.0.1
AP
Client:
Cluster Planning Diagram IP Label IP Address
if1 ________ ___________
alias ___________
if2 ________ ___________

if1 __________ __________ _____________ if1 __________ __________ _____________
if2 __________ __________ _____________ if2 __________ __________ _____________
if3 __________ __________ _____________ if3 __________ __________ _____________
Persist __________ __________ Persist __________ __________
Network = ________________
(netmask = ___.___.___.___)
Home Node Name = ______________ Home Node Name = ______________

Resource Group = ______________ Resource Group = ______________
Startup Policy = ______________ Startup Policy = ______________
Fallover Policy = ______________ Fallover Policy = ______________
Fallback Policy = ______________ Fallback Policy = ______________
Service IP Label = ______________ Service IP Label = ______________
______________ ______________
Application server = ______________ Application server = ______________
serial
Label = ______________ Label = ______________
Device = ______________ Device = ______________
serial
Label = ______________ Label = ______________
Device = ______________ Device = ______________
rootvg rootvg
VG =
VG =
Resource Group __________ contains Resource Group __________ contains

Volume Group = ______________ Volume Group = ______________
hdisks = ______________ hdisks = ______________
Major # = ______________ Major # = ______________
JFS Log = ______________ JFS Log = ______________
Logical Volume = ______________ Logical Volume = ______________
FS Mount Point = ______________ FS Mount Point = ______________
Copyright IBM Corp. 1998, 2005 Appendix A. Cluster Diagrams A-3

Client:
Cluster Planning Diagram IP Label IP Address
if1 ________ ___________
alias ___________
if2 ________ ___________

if1 __________ __________ _____________ if1 __________ __________ _____________
if2 __________ __________ _____________ if2 __________ __________ _____________
if3 __________ __________ _____________ if3 __________ __________ _____________
Persist __________ __________ Persist __________ __________
Network = ________________
(netmask = ___.___.___.___)
Home Node Name = ______________ Home Node Name = ______________

Resource Group = ______________ Resource Group = ______________
Startup Policy = ______________ Startup Policy = ______________
Fallover Policy = ______________ Fallover Policy = ______________
Fallback Policy = ______________ Fallback Policy = ______________
Service IP Label = ______________ Service IP Label = ______________
______________ ______________
Application server = ______________ Application server = ______________
serial
Label = ______________ Label = ______________
Device = ______________ Device = ______________
serial
Label = ______________ Label = ______________
Device = ______________ Device = ______________
rootvg rootvg
VG =
VG =
Resource Group __________ contains Resource Group __________ contains

Volume Group = ______________ Volume Group = ______________
hdisks = ______________ hdisks = ______________
Major # = ______________ Major # = ______________
JFS Log = ______________ JFS Log = ______________
Logical Volume = ______________ Logical Volume = ______________
FS Mount Point = ______________ FS Mount Point = ______________
A-4 HACMP Implementation Copyright IBM Corp. 1998, 2005

V3.1.0.1
backpg
Back page

Au 546 Xstudhint

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Au 546 Xstudhint

Hochgeladen von

Copyright:

Verfügbare Formate

V3.1.0.

Front cover

IBM Certified Course Material

October 2005 Edition

Exercise Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Exercise 1. Cluster Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1

Exercise 2. Cluster Planning: Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1

Exercise 3. Cluster Planning: Network Setup and Test . . . . . . . . . . . . . . . . . . . . . . 3-1

Exercise 4. Application Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1

Exercise 5. HACMP Software Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1

Exercise 6. Initial Cluster Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-1

Exercise 7. Basic HACMP Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1

Exercise 8. Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-1

Exercise 9. Network File System (NFS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-1

Exercise 10. HACMP Problem Determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-1

Exercise 11. Migration (optional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-1

Appendix A. Cluster Diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1

Copyright IBM Corp. 1998, 2005 Contents iii

iv HACMP Implementation Copyright IBM Corp. 1998, 2005

Copyright IBM Corp. 1998, 2005 Trademarks v

vi HACMP Implementation Copyright IBM Corp. 1998, 2005

pref Exercise Description

Copyright IBM Corp. 1998, 2005 Exercise Description vii

viii HACMP Implementation Copyright IBM Corp. 1998, 2005

EXempty Exercise 1. Cluster Planning

What This Exercise Is About

What You Should Be Able to Do

Copyright IBM Corp. 1998, 2005 Exercise 1. Cluster Planning 1-1

Exercise Instructions with Hints

Part 1 Checkout your lab environment

1-2 HACMP Implementation Copyright IBM Corp. 1998, 2005

p520 Lab Environment

Intel workstations (windows/xp, websm, vncviewer)

Copyright IBM Corporation 2005

Copyright IBM Corp. 1998, 2005 Exercise 1. Cluster Planning 1-3

__ 1. You will need the following information from your instructor

1-4 HACMP Implementation Copyright IBM Corp. 1998, 2005

EXempty # smit install_all

Part 2 Create a cluster diagram based on introduction lecture

Copyright IBM Corp. 1998, 2005 Exercise 1. Cluster Planning 1-5

1-6 HACMP Implementation Copyright IBM Corp. 1998, 2005

EXempty Exercise 2. Cluster Planning: Storage

What This Exercise Is About

What You Should Be Able to Do

Please check with your instructor if you have any questions.

2-2 HACMP Implementation Copyright IBM Corp. 1998, 2005

LAB Reference Cluster

Network = ________________ (netmask = 255.255.255.0)

en2 en1 en0 en0 en1 en2

Figure 2-1. Lab Reference Cluster AU546.0

2-4 HACMP Implementation Copyright IBM Corp. 1998, 2005

EXempty Exercise Instructions with Hints

Part 1: Examine the Cluster Environment and Complete the Cluster

t2toronto # lscfg | grep vscsi

Vx identifies the LPAR ID, in this example, t2halifax is LPAR 7 and

2-6 HACMP Implementation Copyright IBM Corp. 1998, 2005

2-8 HACMP Implementation Copyright IBM Corp. 1998, 2005

__ 16. Scroll down to display the two resource groups.

2-10 HACMP Implementation Copyright IBM Corp. 1998, 2005

EXempty __ 18. Repeat for appB_group.

__ 20. Close the OLPW application. Click File -> Exit.

Part 2: Configure Volume Group for AppA Shared Data

__ 26. On the other system, add the disks back in.

Network Adapter IBM 10/100 Mbps Ethernet * entX 10-60

Serial port Standard I/O Serial Port saX 01-S1

TTY device Asynchronous Terminal ttyX 01-S1-00-00

IF IP Label/address myname 192.168.#x.yy

Network Adapter IBM 10/100 Mbps Ethernet * entX 10-60

Serial port Standard I/O Serial Port saX 01-S1

TTY device Asynchronous Terminal ttyX 01-S1-00-00

IF IP Label/address myname# 192.168.#x.yy

Shared Disk P1.1-I3/Q1-W4AC50A84400D hdiskX pdiskY