Beruflich Dokumente
Kultur Dokumente
. Ethernet Switch or HUB for public network Client access to the cluster. 3. Ethernet Switch or HUB for private network to communicate between cluster nodes and other cluster hardware. For exp: Network power switches and Fibre Channel switches. 4. Network Power Switch To perform fencing in an enterprise-level cluster. 5. Fibre Channel Switch Access to Fibre channel storage. Other options: iSCSI and GNBD. This switch also can do fencing. 6. Storage Some type of storage required depend on purpose. Configuration Tools available: 1. Conga:- Install, configure, management of RH clusters, computers, storages attached to clusters and computers. 2. system-config-cluster 3. Command line tools. Conga :- Primary components are: luci :- It is a server runs on one computer and communicates with multiple clusters and computers via ricci. (Port Number: 8084) ricci :- ricci is an agent that runs on each computer managed by Conga. (Port Number: 11111) luci is accessible through browser and provides three major functions. Homebase:- Provides tools for adding and deleting computers, adding and deleting users, and configuring user privileges. Cluster :- Provides tools for creating and configuring clusters. Storage :- Provides tools for remote administration of storage. luci and ricci communicate with each other via XML. Pictures: Page 15 Cluster configuration file: /etc/cluster/cluster.conf GNBD:- Global Network Block Device (Port Number: 14567) DLM:- Distributed Lock Manager (Port Number: 21064) If a cluster node is configured to be fenced by an integrated fence device, disable ACPI Soft-Off for that node. Disabling ACPI Soft-Off allows an integrated fence device to turn off a node immediately and completely rather than attempting a clean shutdown. Otherwise it will take 4 more seconds. ACPI:- Advanced Configuration and Power Interface. Page 1
Untitled Use chkconfig management and verify that the node turns off immediately when fenced. If this does not happen then. 2 more processes: 1. Changing the BIOS setting to "instant-off" or an equivalent setting that turns off the node without delay. 2. Appending acpi=off to the kernel boot command line of the /boot/grub/grub.conf file. (NP: It does permanently) Disable ACPI Soft-Off with chkconfig management at each cluster node as follows: 1. chkconfig --del acpid This command removes acpid from chkconfig or chkconfig --level 2345 acpid off This command turns off acpid. 2. Reboot the node. 3. When the cluster is configured and running, verify that the node turns off immediately when fenced. Configuring MAX_LUNS: It is not necessary to configure max_luns in Red Hat Enterprise Linux 5 Enable max_luns (or max_scsi_luns for 2.4 kernels) in the /etc/modprobe.conf Considerations for Using Quorum Disk: Quorum Disk is a disk-based quorum daemon, qdiskd, that provides supplemental heuristics to determine node fitness. Heuristic is an adjective for experience-based techniques that help in problem solving, learning and discovery. qdiskd set up heuristics that allow the one node to win based on access to a critical resource If cluster requires additional methods of determining node health, then you should configure qdiskd to meet those needs. Considerations before using qdiskd:1. Cluster node votes: - Each cluster node should have the same number of votes. 2. CMAN membership timeout value: - the time a node needs to be unresponsive before CMAN considers that node to be dead, and not a member. Default is 10 secs. 3. Fencing: - To ensure reliable fencing when using qdiskd, use power fencing. Other exps: Watchdog timers and software based solutions These are not compatible with qdiskd. 4. Maximum nodes: - A cluster configured with qdiskd supports a maximum of 16 nodes. The reason for the limit is because of scalability; increasing the node count increases the amount of synchronous I/O contention on the shared quorum disk device. 5. Quorum disk device: - A quorum disk device should be a shared block device with concurrent read/write access by all nodes in a cluster. The minimum size of the block device is 10 Megabytes. Examples of shared block devices that can be used by qdiskd are a multi-port SCSI RAID array, a Fibre Channel RAID SAN, or a Page 2
Untitled RAID-configured iSCSI target. You can create a quorum disk device with mkqdisk, the Cluster Quorum Disk Utility. Multicast Addresses:switch and associated and support IGMP. Without multicast and IGMP Internet Group Red Hat Cluster nodes communicate among each other using multicast addresses. Each network networking equipment in a Red Hat Cluster must be configured to enable multicast addresses IGMP, not all nodes can participate in a cluster, causing the cluster to fail. Management Protocol
Ethernet Channel bonding :Cluster quorum and node health is determined by communication of messages among cluster nodes via Ethernet. With Ethernet channel bonding, multiple Ethernet interfaces are configured to behave as one, reducing the risk of a single-point-of-failure in the typical switched Ethernet connection among cluster nodes and other cluster h/w.
Configuration CLUSTER with CONGA:#yum install ricci #service ricci start #yum install luci #luci_admin init #service luci restart To open at a GUI. Enter the url as: https://luci_server_hostname:8084 Create a cluster under cluster tab. Create a New Cluster Enter Cluster Name Enter Node Host Name & Root Password Support checkbox if clustered storage is required Submit It causes the following actions: a. b. c. d. Check the Enable Shared Storage
Cluster software packages to be downloaded onto each cluster node. Cluster software to be installed onto each cluster node. Cluster configuration file to be created and propagated to each node in the cluster. Starting the cluster.
Untitled When a cluster is created or selected to configure a cluster, it provides an interface with tabs as: General, Fence, Multicast, and Quorum Partition Fencing is the component of cluster project that cuts off access to a resource (hard disk, etc.) from a node in your cluster if it loses contact with the rest of the nodes in the cluster. The most effective way to do this is commonly known as STONITH, which is an acronym that stands for "Shoot The Other Node In The Head" In other words, it forces the system to power off or reboot. Fencing is often accomplished with a network power switch, which is a power switch that can be controlled through the network. This is known as power fencing. Fencing can also be accomplished by cutting off access to the resource, such as using SCSI reservations. This is known as fabric fencing. Fence :- Fence Daemon Properties parameters: Post-Fail Delay and Post-Join Delay. Post-Fail Delay :- It is the number of seconds the fence daemon (fenced) waits before fencing a node. Default is 0. Post-Join Delay :- It is the number of seconds the fence daemon (fenced) waits before fencing a node after the node joins the fence domain. Quorom partition tab :- 2 options: 1. Use a Quorum Partition, Interval, Votes, TKO, Minimum Score, Device, Label, and Heuristics 2. Do not use a Quorum Partition. Clicking Apply on the Quorum Partition tab propagates changes to the cluster configuration file (/etc/cluster/cluster.conf) in each cluster node. However, for the quorum disk to operate, you must restart the cluster. TKO The number of cycles a node must miss to be declared dead. Interval The frequency of read/write cycles, in seconds. Votes The number of votes the quorum daemon advertises to CMAN when it has a high enough score. Minimum Score The minimum score for a node to be considered "alive". If omitted or set to 0. Device The storage device the quorum daemon uses. The device must be the same on all nodes. Label Specifies the quorum disk label created by the mkqdisk utility. Heuristics a. Path to program - The program used to determine if this heuristic is alive. This can be anything that can be executed by /bin/sh -c. A return value of 0 indicates success; anything else indicates failure. b. Interval - The frequency (in seconds) at which the heuristic is polled. The default interval for every heuristic is 2 seconds. c. Score - The weight of this heuristic. The default score for each heuristic is 1. Configuring Fence Devices:Page 4
Untitled Creating a fence device consists of selecting a fence device type and entering parameters for that fence device (for example, name, IP address, login, and password). With Conga you can create shared and non-shared fence devices. The procedures apply only to creating shared fence devices. You can create non-shared (and shared) fence devices while configuring nodes. Configuring Cluster Members:Creating a cluster consists of selecting a set of nodes (or members) to be part of the cluster. Once you have completed the initial step of creating a cluster and creating fence devices, you need to configure cluster nodes. Configuring a failover domain:A failover domain is a named subset of cluster nodes that are eligible to run a cluster service in the event of a node failure. NP: If you are adding a Samba-service resource, connect a Samba-service resource directly to the service, not to a resource within a service. NP: To verify the existence of the IP service resource used in a cluster service, you must use the /sbin/ip addr list command on a cluster node. The following output shows the /sbin/ip addr list command executed on a node running a cluster service. Configuring Cluster Storage:After you select a computer to administer, a general properties page is displayed for the computer: Hard Drives Partitions Volume Groups Shared storage for use in Red Hat Cluster Suite requires that you be running the cluster logical volume manager daemon (clvmd) or the High Availability Logical Volume Management agents (HA-LVM). APACHE SERVER SETUP The cluster software ensures that only one cluster system runs the Apache HTTP Server at one time. The example configuration consists of installing the httpd RPM package on all cluster nodes (or on nodes in the failover domain, if used) and configuring a shared GFS shared resource for the Web content. When installing the Apache HTTP Server on the cluster systems, run the following command to ensure that the cluster nodes do not automatically start the service when the system boots: Page 5
Untitled # chkconfig --del httpd When adding an httpd service, a floating IP address must be assigned to the service so that the IP address will transfer from one cluster node to another in the event of failover or service relocation. The cluster infrastructure binds this IP address to the network interface on the cluster system that is currently running the Apache HTTP Server. This IP address ensures that the cluster node running httpd is transparent to the clients accessing the service. The cluster software must mount and unmount the file system as the httpd service is started and stopped. Configuring Shared Storage:On one cluster node, use fdisk/parted to create the file system: #mkfs.ext3 /dev/sde3 #mount /dev/sde3 /var/www/html Installing and Configuring the Apache HTTP Server The Apache HTTP Server must be installed and configured on all nodes in the assigned failover domain, if used, or in the cluster: On all node in the cluster: # rpm -Uvh httpd-<version>.<arch>.rpm To configure the Apache HTTP Server as a cluster service: 1. Edit the /etc/httpd/conf/httpd.conf configuration file and customize the file accordingly. a. Specify the directory that contains the HTML files. Also specify this mount point when adding the service to the cluster configuration. Exp: DocumentRoot "/mnt/httpdservice/html" 2. Specify a unique IP address to which the service will listen for requests. Exp: Listen 192.168.1.100:80 This IP address then must be configured as a cluster resource for the service using the Cluster Configuration Tool. 3. when configuring the HTTPD service, specify this script by adding it as a Script resource in the Cluster Configuration Tool.
Page 6
Untitled
A lock manager is a traffic cop who controls access to resources in the cluster, such as access to a GFS file system. You need it because without a lock manager, there would be no control over access to your shared storage, and the nodes in the cluster would corrupt each other's data. DLM GULM CMAN Distributed Lock Manager Grand Unified Lock Manager. redundant server-based cluster and lock manager (alternative to CMAN and DLM) Cluster Manager
To check the status of cluster: #clustat Proper way to shutdown the cluster: Use the "cman_tool leave remove" command before shutting down each node. That will force the remaining nodes to adjust quorum to accomodate the missing node and not treat it as an error: for i in rgmanager gfs2 gfs; do service ${i} stop; done fence_tool leave cman_tool leave remove
Checking status of the cluster: # clustat # clustat -m <member> Display status of <member> and exit # clustat -s <service> Display status of <service> and exit # clustat -l Use long format for services # cman_tool status Show local record of cluster status # cman_tool nodes Show local record of cluster nodes # cman_tool nodes -af # ccs_tool lsnode List nodes # ccs_tool lsfence List fence devices # group_tool displays the status of fence, dlm and gfs groups Page 7
Untitled # group_tool ls displays the list of groups and their membership Resource Group Control Commands: clusvcadm -d <group> Disable <group> clusvcadm -e <group> Enable <group> clusvcadm -e <group> -F Enable <group> according to failover domain rules clusvcadm -e <group> -m <member> Enable <group> on <member> clusvcadm -r <group> -m <member> Relocate <group> to member> clusvcadm -R <group> Restart a group in place. clusvcadm -s <group> Stop <group> Resource Group Locking (for cluster Shutdown / Debugging): clusvcadm -l Lock local resource group manager. This prevents resource groups from starting on the local node. clusvcadm -S Show lock state clusvcadm -Z <group> Freeze group in place clusvcadm -U <group> Unfreeze/thaw group clusvcadm -u Unlock local resource group manager. This allows resource groups to start on the local node. clusvcadm -c <group> Convalesce (repair, fix) resource group. Attempts to start failed, non-critical resources within a resource group.
VCS Veritas Cluster Server RHCS Red Hat Cluster Suite Cluster status hastatus -sum clustat Start service hagrp -online service -sys node clusvcadm -e service Stop service hagrp -offline service -sys node clusvcadm -d service Switch service hagrp -switch service -to node clusvcadm -m node -r service Freeze/Lock Unfreeze/Unlock hagrp -freeze service hagrp -unfreeze service clusvcadm -l clusvcadm -u Stop cluster hastop -all clushutdown Show configuration hasys -display hagrp -display hares -display more /etc/cluster/cluster.conf Page 8
Untitled Configuration Files (Format) /etc/VRTSvcs/conf/config/main.cf (Proprietary, text) /etc/cluster/cluster.conf (XML) Logging halog -info ; more log_file more /var/log/messages HB/CM status lltstat -nvv cman_tool status Terminology (sample status) system [resource] group resource system list system state (online offline faulted) group state (online offline partial) resource state (online offline faulted) member service -NAfailover domain member status (online offline faulted) service status (started stopped disabled failed) -NA-
root@ncs-db-1 ~]# cman_tool status Version: 6.1.0 Config Version: 21 Cluster Name: ncs_cluster Cluster Id: 27444 Cluster Member: Yes Cluster Generation: 1796 Membership state: Cluster-Member Nodes: 2 Expected votes: 1 Page 9
Untitled Total votes: 2 Quorum: 1 Active subsystems: 9 Flags: 2node Dirty Ports Bound: 0 11 177 Node name: ncs-db-n1 Node ID: 1 Multicast addresses: xxx.xxx.xxx.xxx Node addresses: xxx.xxx.xxx.xxx [root@ncs-db-1 ~]# ccs_tool lsnode Cluster name: ncs_dbcluster, config_version: 21 Nodename ncs-db-n1 ncs-db-n2 Votes Nodeid Fencetype 1 1 ncs-db-1-ilo1 1 2 ncs-db-2-ilo2
[root@ncs-db-1 ~]# clusvcadm -r clusvc -m ncs-db-n2 Trying to relocate service:clusvc to ncs-db-n2...Success service:clusvc is now running on ncs-db-n2 Stop cluster resource group Resource group can failover if current node due to fencing.
[root@ncs-db-1 ~]# clusvcadm -s clusvc Local machine stopping service:clusvc...Success Restart cluster resource group Resource group can failover if current node due to fencing.
[root@ncs-db-1 ~]# clusvcadm -R clusvc Freeze cluster resource group After freeze resource group, it will not being monitored by the cluster manager. But current resource group state Page 10
[root@ncs-db-1 ~]# clusvcadm -Z clusvc Local machine freezing service:clusvc...Success Unfreeze cluster resource group
[root@ncs-db-1 ~]# clusvcadm -U clusvc Local machine unfreezing service:clusvc...Success Disable cluster status After disable, resource group shutdown and will not affect by server reboot and/or failover.
[root@ncs-db-1 ~]# clusvcadm -e clusvc Local machine trying to enable service:clusvc...Success service:clusvc is now running on ncs-db-n1
[root@ncs-db-1 ~]# clustat Cluster Status for ncs_dbcluster @ Wed Sep Member Status: Quorate Member Name ------ ---ncs-db-n1 ncs-db-n2
1 15:22:08 2009
Untitled Service Name ------- ---service:clusvc Owner (Last) ----- -----ncs-db-n1 State ----started
Resource Group Locking (for cluster Shutdown / Debugging): clusvcadm -l Lock local resource group manager. This prevents resource groups from starting on the local node. clusvcadm -S Show lock state clusvcadm -u Unlock local resource group manager. This allows resource groups to start on the local node. Updating cluster.conf To update the config file in a running cluster: Have all nodes running as cluster members using the original cluster.conf. On one node, update /etc/cluster/cluster.conf, incrementing config_version. On the same node run ccs_tool update /etc/cluster/cluster.conf This instructs the system to propagate the new cluster.conf to all nodes in the cluster. Verify that the new cluster.conf exists on all nodes. On the same node run cman_tool version -r <new config_version> It is necessary that ricci be running in each cluster node to be able to propagate updated cluster configuration information.This informs the cluster manager of the version number change. Check cman_tool status to verify the new config version.
Page 12