Beruflich Dokumente
Kultur Dokumente
High Availability
PHP Quebec, 2009
You?
How do you measure availability?
• Availability is measured in “nines”;
– Two, three, four, five and six nines.
• Is it the time your application is running minus
unpredicted outages?
• Does scheduled maintenance count?
• How about network downtime?
Comparing you options
Availability % Downtime per year Downtime per month*
90% 36.5 days 72 hours
95% 18.25 days 36 hours
98% 7.30 days 14.4 hours
99% 3.65 days 7.20 hours
99.9% 8.76 hours 43.2 min
99.99% 52.6 min 4.32 min
99.999% 5.26 min 25.9s
99.9999% 31.5 s 2.59s
* For monthly calculations, a 30-day month is used.
Source: http://en.wikipedia.org/wiki/High_availability
How do you do this?
• You likely have a boss that demands 100%
availability.
– Part of Implementing HA is a social problem, here’s how
to deal with this....
1. Identify all failure scenarios
• Example Problem:
RAID controller thinks it’s April fools day and starts
writing 1s instead of 0s and 0s instead of 1s.
You’re lucky you can get back online within the day. You
probably have to abandon that server and restore the
backup on a new one.
Failure Scenarios
<?php
/*
Change from 60 seconds (way too long) to five seconds.
If a connection fails, reduce functionality slightly.
*/
ini_set('mysql.connect_timeout', 5);
mysql_connect($hostname, $username, $password)
or handle_mysql_failure();
..
Example (cont.)
..
/*
Check if we have a database link
before rendering optional functionality.
*/
if (!degrade_gracefully()) {
show_add_comment_box();
} else {
print "Comments is under maintenance, check back soon!";
}
?>
A simple example, but..
• There are probably many cases like this in your
application.
High Availability Tools
mysql-master-
master
MySQL Replication
• Asynchronous
– I’ll explain that in a second.
• Statement-Based (until 5.1 at least)
• Spreads read load.
• Does not spread writes.
How replication works...
How replication works...
How replication works...
How replication works...
How replication works...
DRBD
• A Linux Kernel Module
• Think of it as “Network Level RAID1”.
• Not just for MySQL!
– Can be used to mirror other files.
• Doesn’t spread reads.
• Doesn’t spread writes.
• Can be Synchronous.
How DRBD works
How DRBD works
How DRBD works
How DRBD works
How DRBD works
How DRBD works
How DRBD works
How DRBD works
How DRBD works
Comparing Replication to DRBD
Feature Replication DRBD
Syncronous No Yes
Secondary hardware requirements Either faster/slower than master At least as fast as master
192.168.1.1 192.168.1.2
192.168.1.3
#2 #1
Webservers
Database Servers
Heartbeat
192.168.1.1 192.168.1.2
192.168.1.3
#2 #1
Webservers
Database Servers
Heartbeat
192.168.1.1 192.168.1.2
192.168.1.3
#2
Webservers
Database Servers
Heartbeat
192.168.1.1 192.168.1.2
192.168.1.3 Webservers
Database Servers
Heartbeat
192.168.1.1 192.168.1.2
#1 192.168.1.3 Webservers
Database Servers
MMM
• Similar concept to Heartbeat with a floating
“application facing” IP address.
– But more MySQL Aware than heartbeat.
• Under active development by Percona.
Memcached
• Dead simple front-end caching.
• Access data from cache, alleviate database load
server.
Cacti
• A graphing tool.
• Monitoring health important:
Cacti (cont.)
• When we combine our powers together, we have
topologies.....
Starting small...
Web Servers
Web Servers
Web Servers
Web Servers
Web Servers
Front End
Caching (Memcached) Database Slave
Database Server
Clusters aren’t all fun and games
• Split Brains
– A situation where both nodes think that they are
the primary.
I’m
I’m
Primary
Primary
Another issue...
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
An “inconsistent” backup
Users
id name email
user_friendship
1 Fred emo_kid5000@hotmail.com
user_1 user_2
2 Mary daddys_princess@hotmail.com
1 2
2 1
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
An “inconsistent” backup
Users
id name email
user_friendship
1 Fred emo_kid5000@hotmail.com
user_1 user_2
2 Mary daddys_princess@hotmail.com
1 2
2 1
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
An “inconsistent” backup
Users
id name email
user_friendship
1 Fred emo_kid5000@hotmail.com
user_1 user_2
2 Mary daddys_princess@hotmail.com
1 2
3 Barry bazza@somewhere.com
2 1
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
An “inconsistent” backup
Users
id name email
user_friendship
1 Fred emo_kid5000@hotmail.com
user_1 user_2
2 Mary daddys_princess@hotmail.com
1 2
3 Barry bazza@somewhere.com
2 1
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
An “inconsistent” backup
Users
id name email
user_friendship
1 Fred emo_kid5000@hotmail.com
user_1 user_2
2 Mary daddys_princess@hotmail.com
1 2
3 Barry bazza@somewhere.com
2 1
3 2
user_blog_posts 2 3
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
An “inconsistent” backup
Users
id name email
user_friendship
1 Fred emo_kid5000@hotmail.com
user_1 user_2
2 Mary daddys_princess@hotmail.com
1 2
3 Barry bazza@somewhere.com
2 1
3 2
user_blog_posts 2 3
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
An “inconsistent” backup
Users
id name email
user_friendship
1 Fred emo_kid5000@hotmail.com
user_1 user_2
2 Mary daddys_princess@hotmail.com
1 2
3 Barry bazza@somewhere.com
2 1
3 2
user_blog_posts 2 3
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
An “inconsistent” backup
Users
What’s
id name email
user_friendship
1 Fred emo_kid5000@hotmail.com
user_1 user_2
2 Mary daddys_princess@hotmail.com
1 2
3 Barry bazza@somewhere.com
wrong
2 1
3 2
user_blog_posts 2 3
id user_id post_contents
here?
1 1 I hate my parents.
2 2 Ponnies! lol. :D
What’s wrong (cont.)
Users
id name email
user_friendship
1 Fred emo_kid5000@hotmail.com
user_1 user_2
2 Mary daddys_princess@hotmail.com
1 2
3 Jim i_hate_mary@hotmaiil.com
2 1
3 2
user_blog_posts 2 3
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
A “consistent backup”
Users
id name email
user_friendship
1 Fred emo_kid5000@hotmail.com
user_1 user_2
2 Mary daddys_princess@hotmail.com
1 2
2 1
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
A “consistent backup”
Users
id name email
user_friendship
1 Fred emo_kid5000@hotmail.com
user_1 user_2
2 Mary daddys_princess@hotmail.com
1 2
2 1
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
A “consistent backup”
Users
id name email
user_friendship
1 Fred emo_kid5000@hotmail.com
user_1 user_2
2 Mary daddys_princess@hotmail.com
1 2
2 1
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
A “consistent backup”
Users
id name email
user_friendship
1 Fred emo_kid5000@hotmail.com
user_1 user_2
2 Mary daddys_princess@hotmail.com
1 2
2 1
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
InnoDB with MVCC
Users
id name email
1 Fred emo_kid5000@hotmail.com
2 Mary daddys_princess@hotmail.com
3 Barry bazza@somewhere.com
4 Jim i_hate_mary@hotmail.com
InnoDB with MVCC
Users
id name email
1 Fred emo_kid5000@hotmail.com
2 Mary daddys_princess@hotmail.com
3 Barry bazza@somewhere.com
4 Jim i_hate_mary@hotmail.com
InnoDB with MVCC
Users
id name email
1 Fred emo_kid5000@hotmail.com
2 Mary daddys_princess@hotmail.com
3 Barry bazza@somewhere.com
4 Jim i_hate_mary@hotmail.com
InnoDB with MVCC
Users
id name email
1 Fred emo_kid5000@hotmail.com
2 Mary daddys_princess@hotmail.com
3 Barry bazza@somewhere.com
4 Jim i_hate_mary@hotmail.com
InnoDB with MVCC
Users
id name email
1 Fred emo_kid5000@hotmail.com 3 Barry
2 Mary daddys_princess@hotmail.com bazza@hotmail.com
3 Barry bazza@somewhere.com
4 Jim i_hate_mary@hotmail.com
InnoDB with MVCC
Users
id name email
1 Fred emo_kid5000@hotmail.com 3 Barry
2 Mary daddys_princess@hotmail.com bazza@hotmail.com
3 Barry bazza@somewhere.com
4 Jim i_hate_mary@hotmail.com
MySQL Backup Overview
• mysqldump
• Text based backups
• Easier to recover individual rows
• Raw Backup
• Copy the data files
• Filesystem snapshot
• InnoDB Hot Backup (commercial)
• Backup from a Slave
Comparisons
InnoDB Hot
mysqldump Raw backups Snapshots
Backup*
Recovery
Slow Fast Medium Fast
Speed
Recovery
Best Depends Not Good Good
Granularity
Point in Time Recovery
• Really slick, just enable binary logging.
• An example backup:
Recovery:
mysql < mybackup.sql
mysqlbinlog mylog.000005 | mysql
Need more info?
• We have a blog on everything about
MySQL Performance:
www.mysqlperformanceblog.com
Pictures used with CC Attribution
• Hot Rod:
http://www.flickr.com/photos/januszbc/1353696160/
• Hyundai:
http://www.flickr.com/photos/daveseven/
1538984615/
The End
• Questions?