A Study of the Top 15 Wait Events that Cause Database Delays

As part of Confio Softwares consulting and database tuning exercises, we have the chance to observe performance data from hundreds of customers with databases across thousands of applications. This statistical analysis looks at the source of database delays as experienced across a range of different scenarios. This research is aimed to help companies improve their database and application performance by helping them focus on the most serious causes of database related application wait time. Background This study is a comprehensive analysis by Confio Software across hundreds of production databases in multiple industries as measured by wait events. As the Oracle database processes SQL requests, there are hundreds of separate processes in the Oracle kernel through which the SQL statement passes. These are known as wait events because they impose a delay as the SQL waits for processing. Oracle and other database vendors have identified these and exposed them for analysis. In Oracle 10g, there are over 800 different wait events. Because these wait events represent shared resources, where the SQLs must wait for their turn to be processed, they deliver an excellent method of identifying where delays accumulate in executing SQL requests, and exposing what are the bottlenecks limiting database performance for application users. In recent years, leading analysts and experts have identified wait event analysis as the emerging best practice for database tuning, based on the ability to isolate root cause and to make improvements that directly impact end-users. For more detail, see Oracle Wait Interface, Oracle Press 2004 (1) as an excellent reference. 1 Confio Ignite for Oracle is a performance monitoring and tuning tool that focuses on measuring wait time as the most important determinant of database performance. Built specifically to monitor and analyze wait events on a continuous basis, Ignite utilizes an agentless architecture to enable capture of detailed wait time data without imposing measurable load on the monitored server. Study Results The databases in the study focus on Oracle implementations, the most common among the Oracle, Microsoft and IBM databases supported by Confio. Highlights of the research include: Two-thirds of all wait time accumulates in Input/Output (I/O) related delays Cache requests related to clustering are the next most common source of wait-time in Oracle RAC databases, and can be a significant bottleneck The top 15 wait events ranked by total wait time represent over 90% of all wait events collected

Oracle Wait Interface: A Practical Guide to Performance Diagnostics & Tuning. R. Shee, K. Deshpande, K. Gopalakrishnan. Oracle Press 2004. Confio Software www.confio.com

May 2007

Along with an index and ranking of the top 15 wait events ranked by total wait time, this study also provides several high level suggestions for reducing the wait time that will result in increased application performance.

Wait Event / Suggestions db file sequential read tune indexing tune SQL tune disks increase buffer cache db file scattered read add indexes tune SQL tune disks refresh statistics create materialized view direct path read / direct path read temp review P1 (file id), P2 (starting dba), P3 (blocks) if reading temporary data o increase pga_aggregate_target (workarea_size_policy=AUTO) o increase sort_area_size (workarea_size_policy<>AUTO) if reading application data o could be related to parallel query o ensure DISK_ASYNC_IO = TRUE cache temporary datafiles at O/S level global cache cr request RAC event similar to buffer busy waits tune SQL to request less data tune network latency between RAC nodes localize data access buffer busy waits / read by other session tune SQL tune indexing we often see this event along with full table scans review P1 (file id), P2 (block id) for hot blocks if the SQL is inserting data, consider increasing FREELISTS and/or INITRANS if the waits are on segment header blocks, consider increasing extent sizes SQL*Net more data from dblink may not be a problem reduce amount of data transferred across dblink tune network between databases log file sync tune applications to commit less often tune disks where redo logs exist try using nologging / unrecoverable options log buffer could be too large direct path write / direct path write temp review P1 (file id), P2 (starting dba), P3 (blocks) similar approaches as direct path read
May 2007 Confio Software

Wait Class User I/O

Pct 28%

Seconds 81,907,388

User I/O

27%

76,886,866

User I/O

10%

16,699,540

Cluster

5%

14,390,938

Concurrency

5%

11,378,165

Network

4%

10,497,202

Commit

3%

10,039,584

User I/O

3%

7,371,823

www.confio.com

could be related to direct path loads library cache lock need to find the session holding the lock look for DML manipulating an object being accessed if the session is trying to recompile PL/SQL, look for other sessions executing the code Review Metalink Note: 122793.1 for other ideas SQL*Net more data to client may not be a problem reduce amount of data being returned by query tune network access between client and database db file parallel read / db file parallel write tune SQL tune indexing tune disk I/O increase buffer cache library cache pin if many sessions are waiting, tune shared pool if few sessions are waiting, lock is session specific o find the blocking sessions o review Metalink Note: 115656.1 for ideas log buffer space increase LOG_BUFFER parameter move log files to faster disks tune application o use NOLOGGING o look for poor behavior that updates an entire row when only a few columns change enq: TX - row lock contention (10g) / enqueue (9i) multiple sessions are updating same row cache buffers chains typically caused by hot blocks distribute data activity

Concurrency

2%

6,086,052

Network

2%

5,490,140

User I/O

2%

5,180,632

Concurrency

1%

3,336,422

Configuration

1%

1,900,724

Application Concurrency

1% 1%

865,463 722,427

Conclusions As would be expected, when combining statistics across all customer database applications, I/O operations create the most wait time. The study captured nearly 6 aggregate years of wait time in I/O operations alone, demonstrating that this can have a very significant effect on end users. I/O can be tuned through an understanding of the specific SQLs performing the operations, the type of I/O, and the application code that generates the SQL statement. All of these are visible with a fine degree of granularity using wait event analysis. For individual databases, the less frequently encountered wait events will often be the most critical hidden problem. In a specific database situation, wait events like buffer busy, library cache lock, cache buffers, or enqueue can be the most significant source of delay. Because these issues are much less common than typical I/O, they are harder to pinpoint and harder to resolve. Wait event tools will show the exact source of the bottleneck and where to begin resolving it.

May 2007

Confio Software

www.confio.com

In general, if a skilled DBA knows the root cause of the problem, he/she can resolve it and have a significant positive impact on user performance. The challenge is finding definitive evidence of where the problem lies which database processes and for which SQL statement. That is the value of wait event analysis. Confio provides performance consulting and hands-on training with every free trial. The objective is to make the DBA skilled in using the powerful information accessible through wait event analysis to find and resolve problems on his/her own. Confio Igniter Suite Free Trial and Consultation Confio Igniter Suite is the leading wait-time based performance tool for Oracle, DB2, SQL Server and J2EE based applications. Database monitoring installs in 30 minutes, without any agent software on the monitored database server. Free trials are available via download from www.confio.com. Each trial includes free performance consultation with Confio expert DBAs, where the actual performance bottlenecks and tuning recommendations are presented.

May 2007

Confio Software

www.confio.com