MySQL 优化

MySQL Optimization
(A couple of different ways.)

Morgan Tocker morgan@percona.com
1
Monday, March 7, 2011
Show of Hands

InnoDB? MyISAM? XtraDB? MySQL 4.x? MySQL 5.0.x? MySQL 5.1? MySQL 5.5? Something else?
2
About this presentation

Q: Would you like to talk about MySQL optimization? A: Sure. Optimization can be made at many levels. Im going to show 2-3 parts of the puzzle:

Query Optimization. How InnoDB works. (Time Permitting) Our work at Percona on improving MySQL.
3
It all starts with EXPLAIN
Bookmark this manual page: http://dev.mysql.com/doc/refman/5.1/en/usingexplain.html It is the best source for anyone getting started.
4
Examples come from:
IMDB database loaded into InnoDB tables (~5G). Download it and import it for yourself using imdbpy2sql.py: http://imdbpy.sourceforge.net/
5
First Example
6
Find the Title Bambi
3.09s
ALL means tablescan Anticipated number of rows to be examined In this case a sort is required because of the order by
Additional filtering may be possible before passing to sort. 7

Aha! Now add an index:
8
We must revisit...
0.00s
Using = for comparison, but not primary key lookup. Identified title as a candidate index, chose to use it. Size of the index used (in bytes) Anticipated number of rows to be examined dropped considerably. 9
Other ways of accessing
0.00s
At most one We can Better type of const. matching row. stop when we find one row. In InnoDB the primary key is often much faster than all other keys.
10
And..
0.00s
Type is range. BETWEEN, IN() and < > are also ranges.
Number of rows to be examined has increased - we are not specific enough.
Ignore the time with EXPLAIN. Only look at the time for a query.
11
Whys that a range?

We're looking for titles between BambA and BambZ* When we say index in MySQL, we mean trees.

That is, B-Tree/B+Tree/T-Tree. Pretend they're all the same (for simplication). There's no radically different indexing methods in MySQL unless you play storage engine Bingo**.
12
* In reality the range is a little wider ** The memory storage engine supports hash indexes
Whats that?
13
Could this be a range?
3.2s
14
No, we cant traverse.

Do we head left or right here?
15
LIKE Z%
0.05s
16
LIKE T%
3.13s
17
LIKE The %
3.07s
18
MySQL is (reasonably) smart.
It dynamically samples the data to choose which is the better choice - or in some cases uses static statistics*. This helps the optimizer choose:

Which indexes will be useful. Which indexes should be avoided. Which is the better index when there is more than one.
19 * To refresh statistics run ANALYZE TABLE table_name;

Why avoid indexes?
B-Trees work like humans search a phone book;

Use an index if you want just a few rows. Scan cover-to-cover if you want a large percentage.
20
Why avoid indexes (cont.)
We benchmarked this on a different schema:

Table scan has a relatively fixed cost (red line). The index has completely different effectiveness depending on how much it can filter. Hopefully MySQL switches at the right point.
21
What you should take away:
Data is absolutely critical.
Development environments should contain sample data exported from production systems. Between two seemingly identical queries, execution plans may be very different.
Input values are absolutely critical.
22
See also: http://www.mysqlperformanceblog.com /2009/10/16/how-not-to-find-unused-indexes/
Improve this Query
3.41s
This number of rows is a guess. It keeps changing between examples.
23
Were Spoiled for Choice.
24
Index on production_year
3.53s
25
Might work if...
0.92s
26
Index on title(50)
0.02s
27
Comparing the two:
mysql> EXPLAIN SELECT * from title WHERE title = 'Pilot' AND production_year BETWEEN 2006 and 2009\G
28
Composite Indexes.
What is better?
INDEX py_t (production_year, title) INDEX t_py (title, production_year)
29
Index on py_t
0.02s
30
http://www.mysqlperformanceblog.com/2010/01/09/getting-aroundoptimizer-limitations-with-an-in-list/
Index on t_py
0.00s
31
Recommendations
Index over multiple columns if it can improve ltering. i.e.

GOOD: Only some pilots made between 2006-2009. BAD: All pilots made between 2006-2009.
32
Recommendations (cont.)
Don't know what order to specify the columns?
RULE: Think how to lter the fastest. Use that order left to right. EXCEPTION: If there's a range (><, BETWEEN, %). Those always go to the RIGHT.
33
http://www.mysqlperformanceblog.com/2010/01/09/getting-aroundoptimizer-limitations-with-an-in-list/
Query Tuning
How InnoDB Works
Numbers everyone should know

L1 cache reference Branch mispredict L2 cache reference Mutex lock/unlock Main memory reference Compress 1K bytes with Zippy Send 2K bytes over 1 Gbps network Read 1 MB sequentially from memory Round trip within same datacenter Disk seek Read 1 MB sequentially from disk Send packet CA->Netherlands->CA 0.5 ns 5 ns 7 ns 25 ns 100 ns 3,000 ns 20,000 ns 250,000 ns 500,000 ns 10,000,000 ns 20,000,000 ns 150,000,000 ns
36
See: http://www.linux-mag.com/cache/7589/1.html and Google http:// www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf
About Disks.
10,000,000 ns = 10ms = 100 operations/second.
This is about the average for a 7200RPM drive. The variation is because disks are mechanical. We can much write faster sequentially than randomly.
The actual time has dramatic variation.

37
[default] Everything is buffered!
When you write to a le, heres what happens in the Operating System:
Block 9, 10, 1, 4, 200, 5.
Block 1, 4, 5, 9, 10, 200
What happens to this buffer if we loose power?
38
The OS provides a way!
$ man fsync
Synopsis #include <unistd.h> int fsync(int fd); int fdatasync(int fd); Hint: MyISAM just writes to the OS buffer and has no durability.
Description fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent storage device) where that file resides. The call blocks until the device reports that the transfer has completed. It also flushes metadata information associated with the file (see stat(2)).
39
http://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/
Knowing this:

InnoDB wants to try and reduce random IO. It can not (safely) rely on the operating systems write buffering and be ACID compliant.
.. and InnoDB algorithms have to compensate.
40
Basic Operation (High Level)
Log Files
SELECT * FROM City WHERE CountryCode=AUS
Buffer Pool
Tablespace
41
Log Files
Buffer Pool
Tablespace
41
Log Files
Buffer Pool
Tablespace
41
Log Files
Buffer Pool
Tablespace
41
Log Files
Buffer Pool
Tablespace
41
Log Files
Buffer Pool
Tablespace
41
Basic Operation (cont.)
Log Files
UPDATE City SET name = 'Morgansville' WHERE name = 'Brisbane' AND CountryCode='AUS'
Buffer Pool
Tablespace
42
Log Files
Buffer Pool
Tablespace
42
Log Files
Buffer Pool
Tablespace
42
Log Files
Buffer Pool
Tablespace
42

01010
Log Files
Buffer Pool
Tablespace
42

01010
Log Files
Buffer Pool
Tablespace
42

01010
Log Files
Buffer Pool
Tablespace
42

01010
Log Files
Buffer Pool
Tablespace
42
Why dont we update?
This is an optimization!
The log le IO is sequential and much cheaper than live updates. The IO for the eventual updates to the tablespace can be optimized as well.
Provided that you saved enough to recover, this shouldnt matter should it?
43
More on Logs...
Logs are only used during recovery.
Not even read when we need to write down dirty pages!
Log Files
To gure out which pages need to be evicted we have two lists - the ush list and the LRU. Log activities are all assigned a LSN (log sequence number).
44
Log Files, Checkpoints, etc.
Most database systems work this way:
In Oracle the transaction logs are called Redo Logs.
The background process of syncing dirty pages is normally referred to as a Checkpoint. InnoDB has fuzzy checkpointing.
45
Log Writing
You can change increase innodb_log_file_size. This will allow InnoDB to smooth out background IO for longer.
Tip: Optionally you could change innodb_log_files_in_group as well. Be aware that your effective log le is innodb_log_file_size * innodb_log_files_in_group
46
Log Writing (cont.)
You can also change innodb_flush_log_at_trx_commit to 0 or 2 to reduce the durability requirements of this write.
Requires less ushing - particularly helpful on systems without writeback caches!
innodb_log_buffer_size may also help buffer changes for longer before writing to the logs.
Very workload dependent - tends to be more helpful for writing big TEXT/BLOB changes.
47
In summary
On commit, the log has to be ushed to guarantee changes.
Nothing else has to be done.
What visibility do we have into these operations? How do we decide how much background work to do per second? What happens if we fall behind in background work?
48
Enhancements via Percona
Terminology
Oracle Product MySQL Server License GPL Percona Equivalent Product Percona Server The XtraDB Storage Engine XtraBackup License GPL GPL GPL
The InnoDB Storage GPL Engine (Plugin edition) InnoDB Hot Backup Commercial
(GPL = Completely free for you to use. Support not included.)
50
Performance Improvements
Improved Buffer Pool Scalability Insert buffer controls Separate purge thread Completely disable the query cache. Faster page checksums* Strip comments before using query cache Improved Rollback Segment Scalability* Separate location of double write buffer* Improved IO Path + adaptive checkpointing Transaction logs larger than 4G supported.
Remove excess fcntl calls
Data dictionary memory consumption controls Support for different page sizes*
Increased number of undo slots* Per session configuration of innodb_flush_log_at_trx_commit
51
* Changes on disk format (not backwards compatible)
Improved Buffer Pool Scalability
Additional patches to what is already available in the InnoDB Plugin.
Splits the buffer pool mutex into smaller mutexes:

Flush list mutex LRU mutex Free mutex hash mutex
52
Data Dictionary control
Once an InnoDB table is opened it is never freed from the in-memory data dictionary (which is unlimited in size). XtraDB introduces:
--innodb_dict_size_limit - a conguration item in bytes. innodb_dict_tables - a status variable for number entries in the cache.
53
Undo Slots
In the built-in InnoDB, the number of undo slots is limited to 1024.
This means the number of open transactions is limited to 1023. See: http://bugs.mysql.com/bug.php?id=26590 Some statements require 2 undo slots.
In XtraDB, this is expanded to 4072 with -innodb_extra_undoslots=1.
54
Warning: This is binary format incompatible!
Rollback Segments
In XtraDB, its also possible to have more than one rollback segment.
Each segment contains undo slots.
Conguration is --innodb-extra-rsegments=N This has the added effect of reducing mutex contention on the rollback segment:
Mutex at 01b3e3e78 created le trx/trx0rseg.c line 167

http://www.percona.com/docs/wiki/perconaxtradb:patch:innodb_extra_rseg http://www.mysqlperformanceblog.com/2009/10/14/tuningfor-heavy-writing-workloads/
55
Fast Checksums
The InnoDB page checksum computation is slower than it needs to be. XtraDB has the option to use a faster checksum format.
56
Different Page Sizes
XtraDB now has support for different page sizes - 4K, 8K, 16K.
57
Separate Purge Thread
Cleans up a long history list length faster:
58
See: http://www.mysqlperformanceblog.com/2009/10/14/ tuning-for-heavy-writing-workloads/
Usability Enhancements
Show contents of the buffer pool Save index statistics between restarts Import / export of buffer pool contents Show data dictionary User / Index / Table statistics Disable automatic statistics regeneration Transactional Replication Advise in processlist when waiting on Query cache mutex. Improved slow query log Log connection errors Import / export of innodb_file_per_table tables Deadlock counter Retain query response time distribution.
Store buffer pool in shared memory segment
Better handling of corrupted tables
Show Temporary Tables
59
Show Buffer Pool Contents

mysql> SELECT d.*,round(100*cnt*16384/(data_length+index_length),2) fit FROM (SELECT schema_name,table_name,count(*) cnt,sum(dirty),sum(hashed) FROM INNODB_BUFFER_POOL_PAGES_INDEX GROUP BY schema_name,table_name ORDER BY cnt DESC LIMIT 20) d JOIN TABLES ON (TABLES.table_schema=d.schema_name AND TABLES.table_name=d.table_name); +-------------+---------------------+---------+------------+-------------+--------+ | schema_name | table_name | cnt | sum(dirty) | sum(hashed) | fit | +-------------+---------------------+---------+------------+-------------+--------+ | db | table1 | 1699133 | 13296 | 385841 | 87.49 | | db | table2 | 1173272 | 17399 | 11099 | 98.42 | | db | table3 | 916641 | 7849 | 15316 | 94.77 | | db | table4 | 86999 | 1555 | 75554 | 87.42 | | db | table5 | 32701 | 7997 | 30082 | 91.61 | | db | table6 | 31990 | 4495 | 25681 | 102.97 | | db | table7 | 1 | 0 | 0 | 100.00 | +-------------+---------------------+---------+------------+-------------+--------+ 7 rows in set (26.45 sec)
60
Source: http://www.mysqlperformanceblog.com/ 2010/03/26/tables-fit-buffer-poo/ * Command may differ slightly between versions.
Save Buffer Pool Contents
Export the contents of the buffer pool to a le called ib_lru_dump in the data directory:
SELECT * FROM information_schema.XTRADB_ADMIN_COMMAND /*! XTRA_LRU_DUMP*/; SELECT * FROM information_schema.XTRADB_ADMIN_COMMAND /*! XTRA_LRU_RESTORE*/;
Restored ib_lru_dump:
61
Note: Not the actual contents - it takes 8 bytes to remember the address of a 16K page.
Import/Export tables
Because --innodb-file-per-table still has information (data dictionary, undo) in the global tablespace you cant just back it up by itself. With a new setting, --innodb_expand_import=1, this is no longer the case. Tip: The import/export still has to be done with XtraBackup. Documentation available here:
http://www.percona.com/docs/wiki/percona-xtradb:patch:innodb_expand_import
62
The Slow Query Log

$ tail /var/log/mysql.slow .. MySQL Server # Time: 100924 13:58:47 # User@Host: root[root] @ localhost [] # Query_time: 399.563977 Lock_time: 0.000110 Rows_sent: 1 Rows_examined: 46313608 SET timestamp=1285336727; select STRAIGHT_JOIN count(*) as c, person_id FROM cast_info FORCE INDEX(person_id) INNER JOIN title ON (cast_info.movie_id=title.id) WHERE title.kind_id = 1 GROUP BY cast_info.person_id ORDER by c DESC LIMIT 1;
$ mysql -e SET GLOBAL log_slow_verbosity = full; $ tail /var/log/mysql.slow Percona Server .. # Time: 100924 13:58:47 # User@Host: root[root] @ localhost [] # Thread_id: 10 Schema: imdb Last_errno: 0 Killed: 0 # Query_time: 399.563977 Lock_time: 0.000110 Rows_sent: 1 Rows_examined: 46313608 Rows_affected: 0 Rows_read: 1 # Bytes_sent: 131 Tmp_tables: 1 Tmp_disk_tables: 1 Tmp_table_sizes: 25194923 # InnoDB_trx_id: 1403 # QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: Yes # Filesort: Yes Filesort_on_disk: Yes Merge_passes: 5 # InnoDB_IO_r_ops: 1064749 InnoDB_IO_r_bytes: 17444847616 InnoDB_IO_r_wait: 26.935662 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 65329 SET timestamp=1285336727; select STRAIGHT_JOIN count(*) as c, person_id FROM cast_info FORCE INDEX(person_id) INNER JOIN title ON (cast_info.movie_id=title.id) WHERE title.kind_id = 1 GROUP BY cast_info.person_id ORDER by c DESC LIMIT 1;
63
User Statistics
( Not Possible )
MySQL Server
mysql> SET GLOBAL userstat_running = 1; Percona mysql> SELECT DISTINCT s.TABLE_SCHEMA, s.TABLE_NAME, s.INDEX_NAME FROM information_schema.statistics `s` LEFT JOIN information_schema.index_statistics IS ON (s.TABLE_SCHEMA = IS.TABLE_SCHEMA AND s.TABLE_NAME=IS.TABLE_NAME AND s.INDEX_NAME=IS.INDEX_NAME) WHERE IS.TABLE_SCHEMA IS NULL; +--------------+---------------------------+-----------------+ | TABLE_SCHEMA | TABLE_NAME | INDEX_NAME | +--------------+---------------------------+-----------------+ | art100 | article100 | ext_key | | art100 | article100 | site_id | | art100 | article100 | hash | | art100 | article100 | forum_id_2 | | art100 | article100 | published | | art100 | article100 | inserted | | art100 | article100 | site_id_2 | | art100 | author100 | PRIMARY | | art100 | author100 | site_id | ... +--------------+---------------------------+-----------------+ 1150 rows IN SET (1 min 44.23 sec)
Server
64
(Related) Xtrabackup Features
Report on fragmentation of indexes:

$ xtrabackup --stats --tables=art.link* -datadir=/mnt/data/mysql/ ... table: art/link_out104, index: PRIMARY, space id: 12, root page 3 leaf pages: recs=25958413, pages=497839, data=7492026403 bytes, data/pages=91% ...
65
http://www.mysqlperformanceblog.com/2009/09/14/statisticsof-innodb-tables-and-indexes-available-in-xtrabackup/
Basic Operation (again.)
Log Files
Buffer Pool
Tablespace
66
Log Files
Buffer Pool
Tablespace
66
Log Files
Buffer Pool
Tablespace
66
Log Files
Buffer Pool
Tablespace
66

01010
Log Files
Buffer Pool
Tablespace
66

01010
Log Files
Buffer Pool
Tablespace
66

01010
Log Files
Buffer Pool
Tablespace
66

01010
Log Files
Buffer Pool
Tablespace
66

Set innodb_buffer_pool_size to 50-80% of memory. 01010
Log Files
Buffer Pool
Tablespace
66

Set innodb_buffer_pool_size to 50-80% of memory. 01010
Log Files
Increase the size of the log files (innodb_log_file_size and innodb_log_files_in_group)
Buffer Pool
Tablespace
66

Set innodb_buffer_pool_size to 50-80% of memory. 01010 Set innodb_flush_log_at_trx_commit=2 if durability is not as important.
Log Files
Buffer Pool
Tablespace
66

Set innodb_buffer_pool_size to 50-80% of memory. 01010 Set innodb_flush_log_at_trx_commit=2 if durability is not as important. Typically use innodb_flush_method=O_DIRECT if using a Hardware RAID controller.
Log Files
Buffer Pool
Tablespace
66

(Possibly) Move the log files to separate spindles (sequential IO) Set innodb_buffer_pool_size to 50-80% of memory. 01010 Set innodb_flush_log_at_trx_commit=2 if durability is not as important. Typically use innodb_flush_method=O_DIRECT if using a Hardware RAID controller.
Log Files
Buffer Pool
Tablespace
66

(Possibly) Move the log files to separate spindles (sequential IO) If innodb_log_waits > 0 the buffer being filled UPDATE City SET (innodb_log_buffer_size) name = 'Morgansville' before writing to the log files WHERE name = 'Brisbane' AND CountryCode='AUS' may be too small. Increase the size of the log files (innodb_log_file_size and innodb_log_files_in_group) Set innodb_buffer_pool_size to 50-80% of memory. 01010 Set innodb_flush_log_at_trx_commit=2 if durability is not as important. Typically use innodb_flush_method=O_DIRECT if using a Hardware RAID controller.
Log Files
Buffer Pool
Tablespace
66
The End.
Questions?
67

MySQL 优化

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

MySQL 优化

Hochgeladen von

Copyright:

Verfügbare Formate

MySQL Optimization

(A couple of different ways.)

About this presentation

It all starts with EXPLAIN

Examples come from:

Find the Title Bambi

Additional filtering may be possible before passing to sort. 7

Aha! Now add an index:

Other ways of accessing

Number of rows to be examined has increased - we are not specific enough.

Whys that a range?

Monday, March 7, 2011

Could this be a range?

No, we cant traverse.

MySQL is (reasonably) smart.

19 * To refresh statistics run ANALYZE TABLE table_name;

Why avoid indexes?

B-Trees work like humans search a phone book;

Why avoid indexes (cont.)

We benchmarked this on a different schema:

What you should take away:

Data is absolutely critical.

Input values are absolutely critical.

See also: http://www.mysqlperformanceblog.com /2009/10/16/how-not-to-find-unused-indexes/

Monday, March 7, 2011

Improve this Query

This number of rows is a guess. It keeps changing between examples.

Were Spoiled for Choice.

Might work if...

Comparing the two:

INDEX py_t (production_year, title) INDEX t_py (title, production_year)

Monday, March 7, 2011

Index over multiple columns if it can improve ltering. i.e.

Don't know what order to specify the columns?

Monday, March 7, 2011

Monday, March 7, 2011

How InnoDB Works

Monday, March 7, 2011

Numbers everyone should know

See: http://www.linux-mag.com/cache/7589/1.html and Google http:// www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf

Monday, March 7, 2011

10,000,000 ns = 10ms = 100 operations/second.

The actual time has dramatic variation.

[default] Everything is buffered!

Block 1, 4, 5, 9, 10, 200

What happens to this buffer if we loose power?

The OS provides a way!

Monday, March 7, 2011

.. and InnoDB algorithms have to compensate.

Basic Operation (High Level)

SELECT * FROM City WHERE CountryCode=AUS

Basic Operation (High Level)

SELECT * FROM City WHERE CountryCode=AUS

Basic Operation (High Level)

SELECT * FROM City WHERE CountryCode=AUS

Basic Operation (High Level)

SELECT * FROM City WHERE CountryCode=AUS

Basic Operation (High Level)

SELECT * FROM City WHERE CountryCode=AUS

Basic Operation (High Level)

SELECT * FROM City WHERE CountryCode=AUS

Basic Operation (cont.)

Basic Operation (cont.)

Basic Operation (cont.)