Author: Matt Reid

Creating Highly Available Database Services (MySQL + DRBD + LVS)

Date: 2011-05-05 Version: 0.1.A

Table of Contents
Creating Highly Available Database Services ...........................................................................................1 (MySQL + DRBD + LVS).........................................................................................................................1 Overview...............................................................................................................................................1 Requirements ........................................................................................................................................1 Hardware ..........................................................................................................................................1 Rules to Follow ................................................................................................................................2 Installation Process ...............................................................................................................................2 Install DRBD....................................................................................................................................2 Configure DRBD..............................................................................................................................2 Install Heartbeat Services ................................................................................................................3 Test Failover Services.......................................................................................................................4 Administrative Command Reference ..............................................................................................5

Overview
MySQL lacks a rather important built in feature: high availability services. By default we have replication or, when permitted by schema and other limitations, MySQL Cluster with the NDB storage engine. However, when we need to use InnoDB or other engines we are limited to a single master server or problematic ring-replication on dual-master servers. For various reasons that make those solutions less than ideal, we are left without a high availability architecture. To solve the issue we can use LVS (linux virtual server) to create a two node cluster. Generally we would use SAN for a highly available data storage layer. In the event that we do not have SAN we can use DRBD (disk replication block device) to provide the equivalent of a network RAID-1 storage layer for the database cluster. DRBD handles block level data replication between the active and passive nodes while LVS provides a floating virtual IP address (VIP) for a single point of contact for the application to communicate with the database cluster. Once setup we now have the equivalent of a RAID-1 database system functioning as a logical single node that is comprised of two physical nodes. At any given time only one of the nodes is active and can accept read and write query traffic. In the event of a service failure, either due to software or hardware failure, the cluster services will disable the active node and enable services on the passive node. The result is a cost effective and reliable solution to our high availability needs.

Requirements
Hardware
1. Bonded primary interfaces for database traffic (eth0, eth1) 2. Bonded secondary interfaces for heartbeat and sync traffic (eth2, eth3) 3. Minimum of one empty, unformated partition for DRBD data storage. Cannot be shared. 4. Set the network sync rate in drbd.conf to a maximum percentage of 30% network bandwidth if not using a private network or cross connect for the sync traffic. If using a private or cross connect network you can use up to 75% for sync. In general, a gigE network can be assume to support ~110MB/sec when the underlying storage layer also supports write speeds of that amount. As of 2011-05 a sync rate on a gigE

network has been tested to support 50MB sync (raid-10 10K SAS array) but this should not be taken as the maximum supported speed on all boxes since storage layers differ.

Rules to Follow
1. A DRBD disk device cannot be mounted if it is in 'secondary' state, it must be set to primary mode to be able to mount on the operating system level. To set a node's data storage to primary mode run the following command: "shell> drbdadm primary all" 2. If you manually stop MySQL on the active node the cluster services will not initiate a failover, so don't expect it to. If you need to stop services on the active node to initiate a failover to the passive server please use the following administrative commands for failover: hb_standby on the active node, or hb_takeover on the passive node.

Installation Process
Install DRBD
shell> yum update shell> yum install gnutls lm_sensors net-snmp ncurses libgcrypt glib2 openssl glib arptables_jf shell> yum install drbd82 kmod-drbd82

Configure DRBD
1. Copy the following into /etc/drbd.com where each node section is edited to have the active and passive node hostnames and corresponding IP, disk, devices.
#####START global { usage-count yes; } common { syncer { rate 50M; verify-alg sha1; } handlers { outdate-peer "/usr/lib64/heartbeat/drbd-peer-outdater"; } } resource r0 { protocol C; disk { on-io-error detach; fencing resource-only; no-disk-barrier; no-disk-flushes; } net { cram-hmac-alg sha1; shared-secret "<create a md5 hash here that is unique but shared between both nodes"; sndbuf-size 512k;

max-buffers 8000; max-epoch-size 8000; after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; data-integrity-alg sha1; } on <node1-hostname> { device disk <destination logical device like /dev/drbd0>; <original physical partition for data, like /dev/cciss/c0d1p1>;

address <ip_address-node1>:7789; meta-disk internal; } on <node2-hostname> { device disk <destination logical device like /dev/drbd0>; <original physical partition for data, like /dev/cciss/c0d1p1>;

address <ip_address-node2>:7789; meta-disk internal; } } ####END

2. Ensure the destination partition is not formatted, if so delete the partition and create a raw partition in its place. Do not mount the new partition or create a filesystem on it (yet). 3. Initial creation of drbd (run on both nodes)
shell> drbdadm create-md all

4. Start DRBD services (run on both nodes)

shell> /etc/init.d/drbd start

5. Initialize the data store (active node only)

shell> drbdadm -- --overwrite-data-of-peer primary all

6. Create filesystem on data store (active node only)

shell> mkfs.xfs <logical device name from config, like /dev/drbd0>

7. Watch syncronization on the passive node via this command (sync will take a while to finish (5+ hours on a 1TB FS using 50MB sync))
shell> watch -n 10 'cat /proc/drbd'

8. Create destination directory on both nodes

shell> mkdir -p <mount point for drbd partition like /data/drbd0>

9. Mount the DRBD partition on active node

shell> mount /dev/drbd0 /data/drbd0

10. Enable the deadline scheduler for the DRBD physical device (not /dev/drbd0)
shell> echo deadline > /sys/block/<DRBD PHYSICAL DEVICE>/queue/scheduler

You may also make a global I/O scheduler selection by passing the elevator option via your kernel command line. To do so, edit your boot loader configuration (normally found in /boot/grub/menu.lst if you are using the GRUB bootloader) and add elevator=deadline to your list of kernel boot options.

11. Setup DRBD online verification schedule in /etc/crontab 42 0 * * 0 root /sbin/drbdadm verify all 12. Setup MySQL to use the drbd data partition (out of scope of document)

Install Heartbeat Services

1. Install the necessary packages from the yum repository
shell> yum install heartbeat heartbeat-gui heartbeat-ldirectord heartbeat-pils heartbeat-stonith

2. Copy heartbeat (ha.d directory) config files from repo and set IPs where needed. If you don't have the repo files use the following settings. /etc/ha.d/ha.cf
logfacility local0 debugfile /var/log/ha-debug logfile /var/log/ha-log

keepalive 500ms deadtime 10 warntime 5 initdead 30 mcast bond0 225.0.0.1 694 2 0 ping 10.20.30.1

respawn hacluster /usr/lib64/heartbeat/ipfail apiauth ipfail gid=haclient uid=hacluster respawn hacluster /usr/lib64/heartbeat/dopd apiauth dopd gid=haclient uid=hacluster auto_failback off

node laxdbload01 node laxdbload02

/etc/ha.d/haresources
<active_hostname> \ drbddisk \ IPaddr2::<VIP_address>/24/bond0 \ Filesystem::<DRBD device name>::<DRBD mount point>::xfs \ mysqld

/etc/ha.d/authkeys
auth 3 3 md5 <random md5 authentication string that is the same on both nodes>

3. Ensure mysql server is not set for boot time

shell> chkconfig mysql off

4. Copy the /etc/init.d/mysql file to /etc/ha.d/resource.d/mysqld

shell> cp /etc/init.d/mysql /etc/ha.d/resource.d/mysqld

5. Ensure heartbeat starts at boot

shell> chkconfig heartbeat on

6. Ensure /etc/ha.d directory is chown'd by root

shell> chown -R root:root /etc/ha.d

7. Start heartbeat on the active node

shell> /etc/init.d/heartbeat start

Watch the debug log for errors: /var/log/ha-debug

8. Make sure the active node correctly added the VIP

shell> ip addr sh bond0

you should see something like: inet 10.20.30.119/24 brd 10.20.30.255 scope global secondary bond0 , and MySQL should be running, and the DRBD mount should be mounted.

9. You can check the status of the completed work via the following commands
shell> /etc/init.d/heartbeat status

10. start heartbeat on the passive node

shell> /etc/init.d/heartbeat start

Test Failover Services

1. Test failover services as follows (on active). This will force failover, which then will enable the DRBD disk and start mysql on the passive node. Run the command and then (using the commands reference section of this document) check services and look at the heartbeat log to see if services migrated correctly.
shell> /usr/lib64/heartbeat/hb_standby

2. You can reclaim services from active to passive via

shell> /usr/lib64/heartbeat/hb_takeover

3. Run SANITY CHECKS before going into production. Heartbeat must be stopped on both nodes to run the sanity checks. Fix any errors that come up.
shell> /usr/lib64/heartbeat/BasicSanityCheck

Administrative Command Reference

1. Report Heartbeat status for cluster
shell> /usr/lib64/heartbeat/heartbeat -s

2. Fail over services from active to passive node:

shell> /usr/lib64/heartbeat/hb_standby

3. Recalim services from active node and migrate to passive node:

shell> /usr/lib64/heartbeat/hb_takeover

4. Reload heartbeat configs

shell> /etc/init.d/heartbeat reload

5. You can set state of all local device to be the primary using this command:
shell> drbdadm primary all

6. Or switch the local device to be the secondary using:

shell> drbdadm secondary all

7. To change only a single DRBD resource, specify the resource name instead of all. You can temporarily disconnect the DRBD nodes:

shell> drbdadm disconnect all

8. Reconnect them using connect:

shell> drbdadm connect all

9. Check current VIP address:

(active server) shell> ip addr sh bond0

10. Watch DRBD sync state in realtime

shell> watch -n 10 'cat /proc/drbd'