0

Advanced Internal Oracle Tuning Techniques for SAP Systems

Michael Kennedy SMS Consulting
2009 Wellesley Information Services. All rights reserved.

What Well Cover

Understanding Oracle SGA and Components Understanding index design Proactive review of poorly performing SQL Index Rebuilds Oracle Wait Events SAP transactions for tuning Oracle best practices Wrap-up

Oracle SGA Basics

The 3 Most Important Memory Regions: 1. * Shared Pool Holds SQL Statements 2. * Buffer Cache Holds Oracle Blocks 3. * Redo Buffer Holds Transactional Changes

Oracle SGA Basics (cont.)

Tips Normally the Shared Pool Size (shared_pool_size parameter) is no larger than 2-3 GB due to bind variable usage. A very large Shared Pool may cause waits on Cursor Pin: S latches. See SAP Note 690241 for sizing recommendations.

The size of the Buffer Cache (db_cache_size parameter) can vary depending on the amount of free memory on the server. In general the larger the buffer cache the better. Try to keep the Oracle Buffer Hit Ratio above 94% (see SAP Note 789011). Some large customers are running 50-60 GB Buffer Cache. The size of the Redo Log (log_buffer parameter) is usually no larger than 1 MB as per SAP Note 830576 because of the frequent commit rate.

Oracle SGA and Background Processes

Sessions

Oracle Buffer Cache Used By Sessions

Queries to Use:
select a.object_name, count(b.objd) from dba_objects a, v$bh b where a.data_object_id = b.objd and a.object_name = upper('&my_object') group by a.object_name OBJECT_NAME COUNT(B.OBJD) -------------------- ---------------EDIDC 257,534 select a.object_name, b.num_buf from dba_objects a, X$KCBOQH b where a.data_object_id = b.obj# order by 2 desc; OBJECT_NAME NUM_BUF -------------------- ---------------CE4LBOC 125,822 EKPO 74,081 EDIDS~0 54,363 EDIDC 48,917 EDIDS~0 45,644

Background Processes (DBWR)

Database Writer Used to write modified (dirty) Oracle blocks from the Oracle buffer cache to the datafiles.

Background Processes (DBWR) (cont.)

The DBWR writes dirty blocks to disk when: 1. If a user process posts the DBWR because there are too many dirty blocks in memory 2. During an Oracle Checkpoint which should be a LOG SWITCH 3. Parameter Settings (i.e., fast_start_mttr_target) Indications of DBWR issues If the following WAIT events are constantly showing up Write Complete Waits Free Buffer Waits Solutions to DBWR issues 1. Check for Asynchronous I/O (SAP Note 999524) 2. Check the number of DBWRs configured (SAP Note 830576, 793113) 3. Confirm I/O queuing and/or device utilization
9

Log Writer (LGWR)

Log Writer Used to write transactional changes from the Log Buffer to the online redo log

10

Log Writer (LGWR) (cont.)

The LGWR writes changes when: 1. Every Commit (SAP R/3 commits frequently) 2. The Log Buffer is 1/3 Full 3. Every 3 seconds Indications of LGWR issues: If the following WAIT events are constantly showing up log file sync Sessions are waiting on commits log buffer space There is no space left in the log buffer Solutions to LGWR issues: For Log File Sync issues (SAP Note 619188) Check the size of the log buffer it may be too large Check the performance on the redo log filesystems Try to avoid RAID 5 for redo logs (SAP Note 793113) For Log Buffer Space issues increasing the size of the log buffer may reduce the wait event. However, only increase the size in small increments (i.e., 500 KB 1 MB) and continue to review filesystem performance. See SAP Note 619188.
11

What Well Cover

Understanding Oracle SGA and Components Understanding index design Proactive review of poorly performing SQL Index Rebuilds Oracle Wait Events SAP transactions for tuning Oracle best practices Wrap-up

12

B*Tree Design

13

B*Tree Indexes

B*Tree indexes are always balanced All leaf blocks, which hold the row ID, are on the same level Index entries are always ordered Index scans use a db file sequential read Single block fetch (not a multi-block fetch), except for Index Fast Full Scan Can be slow for large index RANGE scans

14

Optimal Oracle Block Access

What is the optimal number of blocks read for an index? The optimal number of blocks being accessed in Oracle is between 3 and 5 During a unique scan down the index, the root block, branch block, leaf block, then the table data is read into the database cache. Depending on the height of the index, a unique scan will only read 3 to 5 blocks. Use the 3 to 5 blocks as the optimal value to compare with other inefficient SQL statements

15

What Well Cover

Understanding Oracle SGA and Components Understanding index design Proactive review of poorly performing SQL Index Rebuilds Oracle Wait Events SAP transactions for tuning Oracle best practices Wrap-up

16

Proactive Review of Poor SQL ST04 Detailed Analysis SQL Requests

17

Proactive Review of Poor SQL (cont.)

18

Inefficient SQL

Example of an inefficient SQL statement

19

Efficient SQL

Example of an efficient SQL statement

20

Proactive Review of Poor SQL

ST04 Detailed Analysis SQL Requests Sort on Proc Rows Notice the HIGH number of PROC ROWS and the HIGH number of Rproc/Exec This may be an indication of poor code

NOTICE: There WHERE CLAUSE in the ABAP code passing VALUES SELECT * FROM zsvc_part_match UP TO 1 ROWS WHERE zzmatnr_app = com_vbrp_tab-matnr AND zz_delete = ' ' AND zzenroll = 'X'. ENDSELECT.

21

Proactive Review of Poor SQL

However, notice that there are NO WHERE clause values in the SQL sent to Oracle other than MANDT This is an example of where a CHECK FOR INITIAL in ABAP will fix this query

22

Reactive Troubleshooting: User Performance Complaint

23

Example: BSAS SQL Statement

ST04

Detailed Analysis

Oracle Session

24

BSAS Buffer Gets

25

BSAS Solution

There have been 13,905 executions while the database has been up for 20 days That means this query is executed 695 times per day or 28 times per hour Each execution needs to allocate 459 blocks of memory to find almost ZERO rows The BSAS statement has fetched almost 7 million blocks into memory with only 6 rows found out of the 13,905 executions Resolution: Add ZUONR to the custom index BSAS~Z1 Create a new custom index as ZUONR, BUKRS, HKONT, BUDAT

26

SQL Statement General Rules

In general, RANGE scans may be slower and require additional resources (CPU, memory, I/O) Try to access the least amount of data down an index Oracle joins may not be the fastest option depending on the driving table of the inner join If a join is slow, determine the inner-most index of the explain plan If the index is not efficient, a custom index may be created If the CBO is not picking the correct driving index, a HINT should be used (SAP Note 130480) Check SAP Notes for performance enhancements for suspected transaction some standard SAP ABAP code may pass the LITERALS instead of BIND variables Example: MB51 transaction is slow due to large number of rows in MSEG/MKPF Applying SAP Note 921164 may change that with LITERALS instead of BIND variables

27

What Well Cover

Understanding Oracle SGA and Components Understanding index design Proactive review of poorly performing SQL Index Rebuilds Oracle Wait Events SAP transactions for tuning Oracle best practices Wrap-up

28

When Should Oracle Indexes Be Rebuilt?

When the index storage quality drops After large data deletes When indexes are very large and you would like to reclaim space If an index has too many extents (greater than 1,000) When there are SQL statements that do large RANGE SCANS (<,>, like, between) on the index. Keeping the indexes small will improve performance.

29

Index Storage Quality

SAP Note 979054 (Program RSORAISQN)

30

Index Storage Quality (cont.)

SAP Note 771929 DB02 Detailed Analysis Enter Index Detailed Analysis Menu: Analyze Index Storage Quality Uses Oracles DBMS_SPACE package to calculate space usage

31

Rebuilding Indexes

SAP Note 771929 SE38 Program RSORATAD Ran for one hour on 20 GB index MSEG~0 with 304 million rows

32

Rebuilding Indexes (cont.)

SAP Note 771929 Analyze Index VALIDATE Structure Analyze index &index validate structure Select * from index_stats Analyze will lock all DML on the table so run on a copy of production
SQL> select name, height, lf_rows, del_lf_rows, pct_used from index_stats; NAME HEIGHT LF_ROWS DEL_LF_ROWS PCT_USED -------------------- ---------- ---------- ----------- ---------TST03~0 3 159608 115497 54

33

How Do You Monitor Index Creation or Rebuild Status?

The index in this example is created with a parallel degree of 5 and is being sorted in PSAPTEMP
create index "EKBE~Z1" on EKBE(MANDT,VGABE, BUDAT, WERKS) tablespace psaprtl online parallel (degree 5)
select a.username, a.tablespace, b.sid, a.contents,a.segtype, a.extents, a.blocks*8192 "Bytes" from v$sort_usage a, v$session b where a.session_addr = b.saddr

USERNAME TABLESPACE SID CONTENTS SEGTYPE EXTENTS Bytes ---------------------- ---------- --------- --------- ---------- ---------------SAPRTL PSAPTEMP 393 TEMPORARY SORT 64 67,108,864 SAPRTL PSAPTEMP 384 TEMPORARY SORT 40 41,943,040 SAPRTL PSAPTEMP 367 TEMPORARY SORT 80 83,886,080 SAPRTL PSAPTEMP 395 TEMPORARY SORT 61 63,963,136 SAPRTL PSAPTEMP 382 TEMPORARY SORT 67 70,254,592 ---------- ---------------sum 312 327,155,712

34

How Do You Monitor Index Creation or Rebuild Status? (cont.)

With this script, you can monitor the index being created in the target tablespace
select segment_name, tablespace_name, bytes, extents from dba_segments where segment_type = 'TEMPORARY' ORDER BY 2

SEGMENT_NAME --------------31.680100 63.290980 97.325500 208.153460 237.178036 sum

TABLESPACE_NAME BYTES EXTENTS --------------- ---------------- ---------PSAPRTL 98,631,680 84 PSAPRTL 79,626,240 81 PSAPRTL 85,262,336 82 PSAPRTL 81,002,496 81 PSAPRTL 95,420,416 86 ---------------- ---------439,943,168 414

35

Tips for Index Creations and Rebuild

The pga_aggreagate_target only uses 100 MB for sorts. Using _pga_max_size may help, which is a setting in SAP NetWeaver BW. Use the parallel query for faster creation times Run the create or rebuild index during a low peak time, because high Data Manipulation Language (DML) may cause a table lock (SAP Note 682926) Use ONLINE keyword so the table is not locked Use NOLOGGING to minimize redo generation The index will need to be dropped and recreated if using Oracle dataguard The index will need to be dropped and recreated if a recovery is needed if a backup was not taken after the create or rebuild command PSAPTEMP needs to be at least 1.5 times the size of the largest index being created in order to hold the ordered data 36

Tips for Index Creations and Rebuild (cont.)

Schedule Index Rebuilds SE38 Program RSANAORA List of Indexes in variant Online Index Rebuilds If locking occurs Kill the Oracle SQL*Net process, NOT the SAP session

37

What Well Cover

Understanding Oracle SGA and Components Understanding index design Proactive review of poorly performing SQL Index Rebuilds Oracle Wait Events SAP transactions for tuning Oracle best practices Wrap-up

38

Oracle Wait Events

Users who are waiting are not happy users! Every session that waits must wait for a specific event, and knowing where to look can answer why the session is waiting Areas to look for waits: V$SESSION_WAIT V$SYSTEM_EVENT V$SESSION V$LOCK SAP Transaction ST04 Detailed Analysis Oracle Sessions

39

Common Oracle Wait Events

DB FILE SEQUENTIAL READ Session is waiting for I/O from an explain plan using an index. A single block I/O is done. DB FILE SCATTERED READ Session is waiting for I/O when doing a FULL TABLESCAN or FAST FULL INDEX scan. Multi-block operations are done. DB FILE PARALLEL WRITE The DBWR is writing data to the datafiles. LOG FILE SYNC The session is waiting for the LGWR to write the data from the log buffer to the online redo logs because of the commit. LATCH FREE The session is waiting for an Oracle Latch. Use P2 value to determine latch name to resolve the issue in 9i. Oracle 10g will show the full latch name.

40

Common Oracle Wait Events (cont.)

ENQUEUE The session is waiting for an Oracle Enqueue, which is most likely either a(n): ST Enqueue Space Transaction Enqueue for extent management TX Enqueue Multiple sessions modifying the exact same row concurrently (usually a SELECT FOR UPDATE) WRITE COMPLETE WAIT The session is waiting for the DBWR to write the dirty block to disk FREE BUFFER WAIT The session is waiting for the DBWR to clean up dirty blocks in the Oracle buffer cache LIBRARY CACHE PIN or LIBRARY CACHE LOCK Waiting for exclusive lock to the library cache and may indicate a Shared Pool issue

41

Oracle Wait Events Example

In this example on Oracle 9i, there are many users waiting for a LATCH FREE for KNB1

42

Oracle Wait Events Example (cont.)

In this example on Oracle 10g, there are many users waiting for a LATCH on cache buffers chains

43

Oracle Wait Events Example (cont.)

In this example from v$session_wait, the event and the time waiting can be seen Shown below, a FULL TABLESCAN (db file scattered read) can be seen. Why? FULL TABLESCANS should not be common! Similar information in ST04 Detailed Analysis Oracle Session
Select sid, seq#, event, p1,p2,p3, seconds_in_wait from v$session_wait where event not like SQL% and event not like %ipc% and event not like %timer;

SID SEQ# EVENT P1 P2 P3 SECONDS_IN_WAIT ---------- ---------- ------------------------------ ---------- ---------- ---------- --------------624 18902 buffer busy waits 354 44041 130 0 2002 41785 db file scattered read 286 73460 8 0 157 30307 db file sequential read 53 320723 1 0 2245 1775 db file sequential read 354 44209 1 0 11 48565 log file parallel write 2 20 2 0

44

Oracle Enqueue Waits

Usually, there are two types of Oracle Enqueues (not the same as SAP Enqueues) TX Enqueue is a row level lock (i.e., DATA) and common in parallel jobs NRIV Locking Locking on similar tables due to SELECT FOR UPDATE Non-TX Enqueue (i.e., TM, ST, CI) is usually locking in Oracle kernel and is not DATA related ST Enqueue due to Dictionary Managed Tablespaces and extent management Oracle bug

45

Oracle Enqueue Waits (cont.)

Steps to diagnose Oracle Enqueues SM66 can show concurrent access to a table (e.g., CE3LBOC below)

46

Oracle Enqueue Waits (cont.)

Use ST04 Detailed Analysis Oracle Session Shows the SQL statement causing the enqueue

47

Oracle Enqueue Waits (cont.)

Use transaction DB01 to see the TX enqueues and the waiter/holder Use the Oracle SID (System ID) shown below in DB01 to map to the session in ST04

48

Oracle Enqueue Waits (cont.)

For Oracle 9i, run the following scripts in SQL*Plus to determine TYPE of enqueue. The TYPE determines in what area the problem resides. The example below shows two sessions with TX enqueue locking

Prompt This will show the TYPE of enqueue SELECT sid, chr(bitand(p1,-16777216)/16777215)|| chr(bitand(p1, 16711680)/65535) "Lock", to_char( bitand(p1, 65535) ) "Mode" FROM v$session_wait WHERE event = 'enqueue; SID Lock Mode ----- ------ ---------------------------------------383 TX 4 385 TX 4

49

Oracle Enqueue Waits (cont.)

For 9i, use the following script to see the object and row of the enqueue prompt. This will show the object and row of enqueue. In this example, the Object 122961 is where the TX enqueue occurs

col event format a25 select a.sid, b.event, a.row_wait_obj#, a.row_wait_file#, a.row_wait_block#, a.row_wait_row#, b.seconds_in_wait from v$session a, v$session_wait b where a.sid = b.sid and b.event = 'enqueue' order by 2, 3, 4, 5;
SID EVENT ROW_WAIT_OBJ# ROW_WAIT_FILE# ROW_WAIT_BLOCK# ROW_WAIT_ROW# SECONDS_IN_WAIT ---------- --------- ------------- -------------- --------------- ------------- --------------381 enqueue 122961 25 25020 0 46 383 enqueue 122961 25 25020 0 43 385 enqueue 122961 25 25020 0 80

50

Oracle Enqueue Waits (cont.)

Oracle 10g will show the TYPE of enqueue in the event

51

General Rules for Oracle Waits

Determine the WAIT event for the sessions, then a course of action Full tablescans are not common and probably should be fixed Large index RANGE scans may be slow due to single block fetch from db file sequential read Transactional enqueues (TX) cause row-level ENQUEUE waits that are usually waiting for the commit due to high concurrency. Try to commit more frequently. Determine the TYPE of enqueue If it is TX, then it is most likely application concurrency If it is NOT TX, then it may be in the Oracle kernel and more research is needed Concurrent Select for UPDATE statements are a common cause of TX enqueues
52

What Well Cover

Understanding Oracle SGA and Components Understanding index design Proactive review of poorly performing SQL Index Rebuilds Oracle Wait Events SAP transactions for tuning Oracle best practices Wrap-up

53

SAP Transactions for Tuning: SM66

Use SM66 as an indicator of normal SAP workload

54

SAP Transactions for Tuning: ST04

Use Transaction ST04 Detailed Analysis an indicator of a normal workload

Oracle Session as

55

SAP Transactions for Tuning: ST05

Use ST05 to trace any specific transactions that may be causing issues

56

SAP Transactions for Tuning: SE30

Use SE30 to quickly identify if the problem is in SAP or the database layer

57

SAP Transactions for Tuning: ST12

Combination ST05 and SE30 (see SAP Note 755977) Holds the history of each trace

58

SAP Transactions for Tuning: ST12 (cont.)

Here we can see SE30 trace information from the ABAP Trace in ST12

59

SAP Transactions for Tuning: ST12 (cont.)

Here we can see ST05 trace information from the ST12 trace

60

SAP Programs: RSORAVDV

Use to see DBA and V$ view information Run in SE38

61

SAP Programs: RSORADJV

Use to access DBA and V$ view information Run in SE38

62

SAP Programs: RSORADJV (cont.)

Output from RSORADJV

63

SAP Programs: RSBDCOS0

Output from RSBDCOS0 Can be used to execute OS-level commands from SAP

64

OS Debugging Methods

truss Trace Unix system calls Example: truss p <unix pid> strace Print streams information strace is available for Windows sar Displays system activity (CPU, disk, network) Example: sar 5 10 vmstat Displays virtual memory statistics Example: vmstat 5 10
65

OS Debugging Methods (cont.)

ping Used to send network packets to host Examples: ping <hostname> ping t <hostname> Use these OS tracing methods in conjunction with the 10046 tracing to confirm elapse times

66

What Well Cover

Understanding Oracle SGA and Components Understanding index design Proactive review of poorly performing SQL Index Rebuilds Oracle Wait Events SAP transactions for tuning Oracle best practices Wrap-up

67

Oracle Best Practices: Tablespaces

Locally Managed Tablespaces (LMTS) Benefits: Does not use the single ST Enqueue, which can cause performance issues Reduces DBAs workload by configuring auto allocate or uniform extent sizes Disadvantages: Still uses pct_used and pct_free, which are not efficient Concurrent inserts are still sent to the same free block on the freelist

68

Oracle Best Practices: Tablespaces (cont.)

ASSM tablespaces Benefits: Same benefits as listed on previous slide for LMTS No longer uses the pct_used and pct_free; instead, manages the block filling based on the percentages Concurrent inserts are automatically sent to different blocks

69

Oracle Best Practices

AUM Automatically configures the number of undo segments needed Minimize ORA-1555 by keeping the changed data in the undo segments based on a retention time PGA tuning Allows the PGA to allocate memory up to a certain value that is greater than the parameters sort_area_size and hash_area_size Minimizes sorts operations (I/O) to PSAPTEMP

70

Oracle Best Practices (cont.)

Auto extend Define only one datafile for each tablespace to auto extend in the event that the DBA cannot add space in time. This will reduce ORA-1653 and ORA-1654 space errors. Using parallel query Use parallel query servers to create indexes faster Use parallel query servers to create tables faster when using create table as select commands

71

Oracle Best Practices (cont.)

SPFILE Using the SPFILE aids in the managing of the database and the parameters that are changed Analyze table/index <name> validate structure online Can be used to look for corruption without putting a lock on the table, which happened in previous Oracle versions Online table reorganization Tables that do not have LONG or LONG RAW columns can now be reorganized online using DBMS_REDEFINITION package. This method is based on materialized views technology.

72

What Well Cover

Understanding Oracle SGA and Components Understanding index design Proactive review of poorly performing SQL Index Rebuilds Oracle Wait Events SAP transactions for tuning Oracle best practices Wrap-up

73

Resources

Oracle/SAP Newsletter www.oracle.com/newsletters/sap/current.html Wait Events service.sap.com/notes * SAP Note 619188

* Requires login credentials to the SAP Service Marketplace

74

7 Key Points to Take Home

Understanding the Oracle fundamentals helps identify problems and provide solutions The most efficient I/O done, besides blocks already residing in cache, will be 3-4 blocks per row return Proactively monitor the system for inefficient SQL statements using ST04 and ST05 transactions Rebuilding indexes can improve runtimes Always determine what event is causing users to wait

75

7 Key Points to Take Home (cont.)

Use SAP transactions ST12, SE30, ST05, SM66, and ST04 to quickly identify the area of waits Set up the Oracle/SAP database per SAPs Best Practices Standards (ASSM, AUM, SPFILE, etc.)

76

Your Turn!

How to contact me: Mike Kennedy michael.kennedy@smsconsultinginc.com

77

Disclaimer
SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, PartnerEdge, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Wellesley Information Services is neither owned nor controlled by SAP.

78