Basic MySQL Troubleshooting For Oracle DBAs

Copyright 2014, Oracle and/or its affiliates. All rights reserved.
Basic MySQL Troubleshooting

for Oracle DBAs
Sveta Smirnova
Senior Principal Technical Support Engineer
MySQL Support
September 29, 2014
Copyright 2014, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracles products remains at the sole discretion of Oracle.
Program Agenda
1
Introduction
Basic troubleshooting techniques
High concurrency issues
High availability solutions
More information
Introduction
MySQL architecture
Click icon to add picture
Base
- Installation layout
- Log files
Connectors
- Clients
- APIs
Optimizer
Caches & buffers
Storage engines
Management
Typical installation layout

datadir
Schema directory
Table files: *frm, *ibd, *MYD, *MYI, *par, etc.

Trigger files
Schema
InnoDB shared tablespace

Log files
InnoDB log files
Binary, relay logs
Error log
Slow query log
General query log
Configurable
You can setup custom path for each
component
Including custom path for tables
Connectors
The way you connect to MySQL Server
Clients
MySQL CLI
MySQL Workbench
MySQL Enterprise Monitor (MEM)
Oracle Enterprise Manager with MEM plugin
APIs
Exist for most popular programming languages
C, C++, JDBC, PHP, Python, Net, ODBC
Plugins
MySQL Server is highly configurable via plugins
Storage engines
Full-text parsers
Daemon
INFORMATION_SCHEMA
Semisynchronous Replication
Audit
Authentication
Password-validation
Protocol Trace
5.7
Storage engines
From troubleshooting point of view
Own data
Own index format
Own locking model
Own diagnostic
Own log files
CHECK TABLE
Basic troubleshooting techniques
MySQL Access Privilege System

Typical issues
Privileged client cannot connect

Unprivileged client can connect
Privileged user cannot perform operation
Unprivileged user has undesired access

Overview
No roles by default, limited user limits

All records are in the mysql database (schema)
Pluggable authentication since version 5.5
Connections
TCP/IP with login-password
Socket (Unix)
Named pipe (Windows)

Where to find out why you have connection issues?
select user, host from mysql.user order by user

desc, host desc;
Most descriptive host first, then wildcard

Socket connection by default on Unix

Where to find out why you have connection issues?
mysql> select user, host from mysql.user order by user desc, host desc;
+------+------------+
| user | host
+------+------------+
| root | localhost
| root | delly
| root | ::1
| root | 127.0.0.1
| foo
| %
| localhost
+------+------------+
6 rows in set (0.00 sec)

Wrong access
SHOW GRANTS [FOR user@host]
Grant tables
mysql.db
mysql.tables_priv
mysql.columns_priv
mysql.procs_priv
mysql.proxies_priv
SELECT USER(), CURRENT_USER()

Wrong access
mysql> show grants;

+--------------------------------------------------------------------+
| Grants for root@localhost
+--------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION|
| GRANT PROXY ON ''@'' TO 'root'@'localhost' WITH GRANT OPTION
+--------------------------------------------------------------------+

Wrong access
mysql> show grants for foo@'%';

+-----------------------------------------------------+
| Grants for foo@%
+-----------------------------------------------------+
| GRANT USAGE ON *.* TO 'foo'@'%'
| GRANT ALL PRIVILEGES ON `test`.* TO 'foo'@'%'
+-----------------------------------------------------+
Error handling
Errors vs warnings
mysql> select max (f1) from t1;

ERROR 1630 (42000): FUNCTION test.max does not exist. Check the
'Function Name Parsing and Resolution' section in the Reference Manual
mysql> select * from t1 where "f1"=1;
Empty set, 1 warning (0.05 sec)
mysql> show warnings;
+-----------+--------+----------------------------------------------------+
| Level
| Code
| Message
+-----------+--------+----------------------------------------------------+
| Warning
| 1292
| Truncated incorrect DOUBLE value: 'f1'
+-----------+--------+----------------------------------------------------+
1 row in set (0.00 sec)

Application (C API)
Error information
mysql_errno
mysql_error
Warnings
mysql_info
mysql_sqlstate
mysql_warning_count
Error handling
perror
[sveta@delly ~]$ perror 1630

MySQL error code 1630 (ER_FUNC_INEXISTENT_NAME_COLLISION):
FUNCTION %s does not exist. Check the 'Function Name Parsing and
Resolution' section in the Reference Manual
MySQL error code 1292 (ER_TRUNCATED_WRONG_VALUE): Truncated
incorrect %.32s value: '%.128s'
OS error code
2:
No such file or directory

MySQL error code 150: Foreign key constraint is incorrectly formed
Error handling
In stored routines
GET DIAGNOSTICS
GET DIAGNOSTICS rows = ROW_COUNT, conditions = NUMBER;
GET DIAGNOSTICS CONDITION 1

msg = MESSAGE_TEXT;
code = RETURNED_SQLSTATE,
http://dev.mysql.com/doc/refman/5.6/en/diagnostics-area.html
What can affect query execution?

You run a query, it does not return an error, but still behaves not as expected
Startup options or system variables

How optimizer creates query plan
Storage engine used
Parallel execution
next section
System variables and startup options

Scope
Global
Control parameters, necessary for all server processes

Location of server files: datadir etc.
Shared buffers
More
Session
Control connection-specific parameters
http://dev.mysql.com/doc/refman/5.6/en/mysqld-option-tables.html

How to set
SET [GLOBAL] var_name = NEW_VAL

Command line option
Configuration file
In default location
http://dev.mysql.com/doc/refman/5.6/en/option-files.html
Specified by option --defaults-file

Who can change
Global options and few session options
A user with privilege SUPER
Session options
Anybody
There is no limits!

When allocated
Those which control behavior of whole server
Once at server startup
Can start with low values, then grow to specified
Connection options
For every connection when connection opens
Operation-specific
For every operation when needed

How to control
SHOW [GLOBAL] STATUS
GLOBAL
Since server start
SESSION
For operations in current session
Can be reset
FLUSH STATUS

How to control
mysql> show global status like 'Handler_read_rnd_next'\G
*************************** 1. row ***************************

Variable_name: Handler_read_rnd_next
Value: 27
mysql> show status like 'Handler_read_rnd_next'\G
*************************** 1. row ***************************
Variable_name: Handler_read_rnd_next
Value: 7

Troubleshooting best practices
Record currently used variables
SHOW [GLOBAL] VARIABLES
Make change dynamically if possible
SET [GLOBAL] var_name=NEW_VAL
Test in one session first

Then change global variable
Change configuration file after you are happy with results

When affecting option is not known
Record currently used variables
SHOW [GLOBAL] VARIABLES
Start mysqld with option --no-defaults
This option must be first one!
Check if problem is solved

Change variable values one-by-one until you find one which leads to the
problem
Diagnostic tool: INFORMATION_SCHEMA

When affecting option is not known
Contain metadata information
Tables
Indexes
Other
Allows to create plugins
InnoDB plugins
We will discuss them later
Oracle analog
Data Dictionary Views

MySQL Optimizer
Overview
EXPLAIN is less powerful if compare with Oracle
It is improved in version 5.7.3
Graphic EXPLAIN in MySQL Workbench 6.0
EXPLAIN EXTENDED
EXPLAIN PARTITIONS
EXPLAIN FORMAT=JSON
INFORMATION_SCHEMA.TRACE
Status variables 'Handler_%'
Default in 5.7
EXPLAIN in Oracle
http://docs.oracle.com/cd/B10500_01/server.920/a96533/ex_plan.htm
EXPLAIN PLAN SET statement_id = 'example_plan4' FOR
SELECT h.order_number, l.revenue_amount, l.ordered_quantity
FROM so_headers_all h, so_lines_all l
WHERE h.customer_id = :b1
AND h.date_ordered > SYSDATE30
AND l.header_id = h.header_id ;
Plan
-------------------------------------------------SELECT STATEMENT
NESTED LOOPS
TABLE ACCESS BY INDEX ROWID SO_HEADERS_ALL
INDEX RANGE SCAN SO_HEADERS_N1
TABLE ACCESS BY INDEX ROWID SO_LINES_ALL
INDEX RANGE SCAN SO_LINES_N1
EXPLAIN in MySQL
mysql> EXPLAIN SELECT user, host FROM mysql.user\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: user
type: index
possible_keys: NULL
key: PRIMARY
key_len: 228
ref: NULL
rows: 4
Extra: Using index
EXPLAIN in MySQL
Variant: EXTENDED
mysql> EXPLAIN EXTENDED SELECT user, host FROM mysql.user\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: user
type: index
possible_keys: NULL
key: PRIMARY
key_len: 228
ref: NULL
rows: 4
filtered: 100.00
mysql> SHOW WARNINGS\G

******* 1. row *******
Level: Note
Code: 1003
Message: /* select#1 */
select
`mysql`.ùser`.Ùser` AS
ùser`,`mysql`.ùser`.`H
ost` AS `host` from
`mysql`.ùser`
Extra: Using index

1 row in set, 1 warning (0.00 sec)
EXPLAIN in MySQL
Variant: FORMAT=JSON
...
mysql> EXPLAIN FORMAT=JSON SELECT

user, host FROM mysql.user\G
"used_key_parts": [
"Host",
************** 1. row **************
"User"
EXPLAIN: {
],
"query_block": {
"key_length": "228",
"select_id": 1,
"rows": 4,
"table": {
"filtered": 100,
"using_index": true
"table_name": "user",
"access_type": "index",
"key": "PRIMARY",
...
1 row in set, 1 warning (0.03 sec)
EXPLAIN in MySQL
More information
http://www.slideshare.net/SvetaSmirnova/troubleshooting-my-sqlperformanceaddonsen
http://dev.mysql.com/doc/refman/5.6/en/explain-output.html
MySQL EXPLAIN in Practice [HOL9232] (past)
MySQL 5.7: Whats New in the Parser and the Optimizer? [CON2830] (past)
How to Analyze and Tune MySQL Queries for Better Performance
[TUT3157] (past)
Using MySQL Workbench Performance Tools [HOL9237] (past)
MySQL Optimizer
Handler_% status variables
mysql> flush status;
Query OK, 0 rows affected (0.00 sec)
mysql> SHOW STATUS LIKE 'Handler_read_%';
+----------------------------+-------+
| Variable_name
| Value |
+----------------------------+-------+
| Handler_read_first
| 0
| Handler_read_key
| 0
| Handler_read_last
| 0
| Handler_read_next
| 0
| Handler_read_prev
| 0
| Handler_read_rnd
| 0
| Handler_read_rnd_next
| 0
+----------------------------+-------+
MySQL Optimizer
Handler_% status variables
mysql> select count(*) from employees join titles using(emp_no) where title='Senior Engineer'\G
*************************** 1. row ***************************
count(*): 97750
mysql> SHOW STATUS LIKE 'Handler_read_%';
+----------------------------+--------+
| Variable_name
| Value
+----------------------------+--------+
| Handler_read_first
| 1
| Handler_read_key
| 300027 |
| Handler_read_last
| 0
| Handler_read_next
| 397774 |
...
MySQL Optimizer
Other tools
INFORMATION_SCHEMA.TRACE
http://dev.mysql.com/doc/internals/en/optimizer-tracing.html
join_preparation, join_optimization,
join_execution
considered_execution_plans, refine_plan, more
Query Analyzer in MEM
http://dev.mysql.com/doc/mysql-monitor/3.0/en/mem-quan-using.html
Graphic EXPLAIN in MySQL Workbench 6.0
MySQL Storage Engines

Specifics
Own way to handle
Corruption
Index statistics
CHECK TABLE to check for errors
They care about physical data, so all data information are on their level
Options usually start from engine name
myisam_*, innodb_*, custom_*
MySQL Storage Engines

InnoDB
Transactional storage engine

Physical layout
*frm file table definition
Shared tablespace
*ibd file tablespace for individual table
Optional: --innodb_file_per_table
Recommended and default since 5.6
Redo log files
OPTIMIZE TABLE = ALTER + ANALYZE

Automatic startup check

Common problems
Query or transaction waits a lock, held by another one

Fight for system resources
Resource overload
Resource underload
Transactions and their relation to locks

Lock types
Levels
MDL
Table-level
Row-level
Transactions
Server-level
MDL locks
What do they lock

Read locks
Engine level
Table locks
Row locks
Block writes
Write locks
AUTOCOMMIT
Supported
Block both reads and writes
Locking issues
Diagnostic tools
SHOW [FULL] PROCESSLIST
Universal tool which lists all currently running connections
SHOW ENGINE INNODB STATUS

INFORMATION SCHEMA
PROCESSLIST
InnoDB tables
PERFORMANCE SCHEMA
MDL locks
Table locks
Server-level transactions
Storage engine specifics

Diagnostic tools
Transactions and engine-level locks are done at the engine level

Storage engines have own diagnostic tools
Use them when hit an issue with such a lock

InnoDB
SHOW ENGINE INNODB STATUS

InnoDB monitors
Pseudo-tables
Turn periodical logging into error log
CREATE TABLE innodb_monitor(f1 int) ENGINE=INNODB;

Lock monitor
Changes output format of InnoDB Monitor
CREATE TABLE innodb_lock_monitor(f1 int) ENGINE=INNODB;

Options
innodb_status_output
innodb_status_output_locks

InnoDB
Tables in INFORMATION_SCHEMA
INNODB_TRX
INNODB_LOCKS
INNODB_LOCK_WAITS
INNODB_METRICS
Options innodb_monitor_*
Option innodb_print_all_deadlocks
Diagnostic tool: performance_schema

For whole server
Monitors internal operations
Events
Waits
Mutexes
Statements
Stages
MDL, table-level locks, transactions
Memory
Replication
5.7+
Similar to Oracle wait interface

Diagnosting locks
Summary
Table-level locks
PROCESSLIST: Waiting for table lock
P_S.TABLE_HANDLES
5.7+
Row-level locks
InnoDB monitors, SHOW ENGINE INNODB STATUS
Tables in INFORMATION_SCHEMA
Option --innodb_print_all_deadlocks
MDL locks
PROCESSLIST: Waiting for metadata lock
P_S.METADATA_LOCKS
5.7+
High availability solutions
MySQL Cluster
Special storage engine: NDB
Stores data on two or more physical machines
Two or more copies
General troubleshooting techniques
Applicable
Specific NDB storage engine techniques
MySQL Replication
Overview
Always available, but you must setup it before use

Asynchronous master-slave
Master
Keeps all updates in separate file: binary log
Slave
IO thread read updates from master and stores in relay log file
SQL thread executes updates
Replication IO thread
Communication issues
Access error
Check slave error log
SHOW SLAVE STATUS
P_S.replication_connection_status
Try to connect using normal MySQL client using slave credentials

SHOW GRANTS
Fix privileges on master
Restart slave
Replication SQL thread

Typical issues
Simple master-slave
Data is different on master and slave

Replication event can not be applied
Different errors on master and slave
Slave lags far behind the master
Circular replication or other writes in addition to slave SQL thread

Was the table modified besides the SQL thread?
How?
Can it affect content of the table in the wrong way?
Are the table definitions same on master and slave?
MySQL Utilities
mysqlrplsync, mysqldbcompare, mysqldiff
Is it possible that master events was applied in wrong order?
Use mysqlbinlog to find queries which caused the issue
Check master's application to find what is wrong


Events from master were applied in wrong order
Lock issues
InnoDB
Triggers
SET GLOBAL slave_skip_counter
Synchronize tables!
Different options
Start slave with master's options, then check

Slave lags far behind the master
Threads
Master runs in multiple update threads
Slave uses single
Seconds_behind_master is growing
Tune slave performance
Buffers
Indexes (for statement-based replication)
More information
OpenWorld sessions
October, 01
Practical MySQL Optimization [CON5522]
Peter Zaitsev, 3:30 PM, Moscone South - 262
Whats New in MySQL 5.7 Security [CON1985]
Georgi Kodinov, 4:45 PM, Moscone South - 262
MySQL Cost Model [CON3163]
Olav Sandst, 5:30 PM, Moscone South - 250
OpenWorld sessions
October, 02
MySQL Troubleshooting with the Performance Schema [HOL9215]
Sveta Smirnova, Lig Isler-turmelle
8:30 AM, Hotel Nikko - Monterey
MySQL Enterprise Edition Features in Practice [HOL9216]
Matt Lord, 10:00 AM, Hotel Nikko - Monterey
MySQL Performance: Demystified Tuning and Best Practices [CON5097]
Dimitri Kravtchuk, 10:45 AM, Moscone South - 252
Further reading
MySQL Troubleshooting book
http://shop.oreilly.com/product/0636920021964.do
Marc Alff's Performance Schema blog
http://marcalff.blogspot.ru/
Planet MySQL
http://planet.mysql.com/
http://dev.mysql.com/support/blogs/
http://mysqlserverteam.com/
References
MySQL User Reference Manual
http://dev.mysql.com/doc/refman/5.6/en/index.html
Knowledge Management Database

Forums
http://forums.mysql.com
Bug trackers
http://bugs.mysql.com
Oracle Internal Bugs database
My Oracle Support
https://support.oracle.com
You will find me at

https://twitter.com/svetsmirnova
https://blogs.oracle.com/svetasmirnova/
http://www.slideshare.net/SvetaSmirnova/
Thank you!

Basic MySQL Troubleshooting For Oracle DBAs

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Basic MySQL Troubleshooting For Oracle DBAs

Hochgeladen von

Copyright:

Verfügbare Formate

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Basic MySQL Troubleshooting

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Basic troubleshooting techniques

High concurrency issues

High availability solutions

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Typical installation layout

Table files: *frm, *ibd, *MYD, *MYI, *par, etc.

InnoDB shared tablespace

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Basic troubleshooting techniques

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

MySQL Access Privilege System

Privileged client cannot connect

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

MySQL Access Privilege System

No roles by default, limited user limits

TCP/IP with login-password

Named pipe (Windows)

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

MySQL Access Privilege System

select user, host from mysql.user order by user

Most descriptive host first, then wildcard

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

MySQL Access Privilege System

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

MySQL Access Privilege System

SHOW GRANTS [FOR user@host]

SELECT USER(), CURRENT_USER()

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

MySQL Access Privilege System

mysql> show grants;

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

MySQL Access Privilege System

mysql> show grants for foo@'%';

| GRANT ALL PRIVILEGES ON `test`.* TO 'foo'@'%'

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

mysql> select max (f1) from t1;

| Truncated incorrect DOUBLE value: 'f1'

MySQL Access Privilege System

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

[sveta@delly ~]$ perror 1630

No such file or directory

[sveta@delly ~]$ perror 150

GET DIAGNOSTICS rows = ROW_COUNT, conditions = NUMBER;

GET DIAGNOSTICS CONDITION 1

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

What can affect query execution?

Startup options or system variables

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

System variables and startup options

Control parameters, necessary for all server processes

Control connection-specific parameters

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

System variables and startup options

SET [GLOBAL] var_name = NEW_VAL

Table files: frm, ibd, MYD, MYI, *par, etc.

* 1. row *

1. row

myisam_, innodb_, custom_*