EBS Tuning Kanda Session Perf

Safe Harbor Statement
The following is intended to outline our general product direction.

It is intended for information purposes only, and may not be
incorporated into any contract. It is not a commitment to deliver any
material, code, or functionality, and should not be relied upon in making
purchasing decisions. The development, release, and timing of any
features or functionality described for Oracles products remains at the
sole discretion of Oracle.
Copyright 2014 Oracle and/or its affiliates. All rights reserved. |
Best Practices In E-Business Suite

Performance Tuning
Kandasamy Nainamalai
Senior Principal Software Engineer - Applications Technology Group
Program Agenda
1
Applications Architecture
Defining & Isolating issue
How to approach a performance issue
SQL Trace and TKPROF output
AWR/Statspack Report
Best Practices Maximizing the performance

<Insert Picture Here>
Oracle Applications
Architecture
Oracle Applications Architecture

3-tier architecture
Database Tier
Application/Middle Tier
Desktop Tier
3-Tier Architecture
Defining and Isolating

the Issue
Defining and Isolating the Issue

What is slow?
1. The entire system is slow.
2. A subsystem is slow:
- Forms-Based Applications
- Web-Based Applications
- Concurrent Manager
1. A particular process is slow.

When is it slow?
Month end/year end |

Peak hours
| Concurrency, due to Load (or) H/W is not sized.
All the time
|
Was it working fine earlier?
All of a sudden
List out recent changes
Patch, Gather Schema, New RAC-node/modules/users, Data load, OS, Network/DNS
Deteriorated over a period
Who all are affected?
All users
Specific to some module

By this time, we should be in a position to isolate which component is the
source of the performance issue.
Once we identify the component, we need to investigate what could be
the problem.
How to go about doing that?
How to Approach a
Performance Issue
Approaching Performance Issue

Gathering Required Information

Desktop
Although its a thin client; browser uses Java for the presentation Layer.
It requires some resource at the Desktop.
Check if theres enough resource CPU/RAM
Ensure that no heavy applications are running when accessing
applications.

Concurrent Manager
Concurrent jobs are CPU intensive.

Hence ensure enough resource CPU/RAM is available on the DB tier.
Check RAC & PCP configuration.
Check concurrent manager definition for Sleep time & Cache are set correctly
Check resource-intensive batch requests are scheduled during peak hours. ( if yes, Separate
manager for long running process with less number of managers)
Check there are short running requests waiting for a longer time to process (Resolution: create
Specialized Managers)
Check Fnd_concurrent_requests & Processes are purged & defragmented regularly.
Avoid enabling an excessive number of standard or specialized managers.
Note 1057802.1 Best Practices for Performance for Concurrent Managers

Forms Server
Memory Intensive. Ensure sufficient RAM,Swap & CPU.
Check Sar & top (look for spinning f60webmx)
Check FORMS60_CATCHTERM FORMS_RECORD_GROUP_MAX
Check if Debug/FRD enabled.
Check load balancing enabled and works effectively.
Latest Forms Patch set level with IO patches.
Servlet Vs Socket Mode

Applications
How many concurrent users are there?
What are all the modules used?
Any resource-intensive module is used.
When was gather schema statistics request run last?

Diagnostics profiles are enabled ; this will consume lots of resource.
Sign on Audit is disabled if not needed.
Audit on other tables have been enabled (additional overhead)
Purging is done periodically.

Database
DB processes are basically CPU Intensive.
Check adequate resource (CPU/RAM) is available.
Monitor Sar/vmstat output (first thing to do)
If Wio is high then need to check I/O setup & SQL queries
If CPU is high then check for CPU-intensive queries.
Monitor top OS process at OS level

Top (HP) / prstat (Sun) / ps aux (AIX)
init.ora setup (Refer note 174605.1 )

Version of the Database & performance patch applied.
AWR / Statspack

Web Tier (OC4J, Apache & Jserv)
Check if JVM is hanging (due to Out of Memory issue).

Check Heap size and how many JVM threads are defined.
Disable statement/debug level logs when not needed.
AOL/J Database Connection Pool Status (Doc ID 278868.1)
Review DBC file configuration
Review Apache/Jserv configuration
Network
It plays vital role if Applications accessed through VPN/WAN

Check for sufficient Bandwidth (ping host l 1024 t)
Latency should be as less as possible. Network Test ((Doc ID 556738.1)
Socket Vs Servlet
SQL Trace and

TKPROF Output
When specific process takes time
Different Trace Levels

Four possible levels:
Level 1 : Regular Trace / SQL Trace
Level 4 : Trace with Binds
Level 8 : Trace with Waits
Level 12 : Trace with Binds and Waits
Level 8 is the preferred for analyzing performance issues.
Create Trace File Form

Trace menu availability: 11.5.7 or higher
Trace file name contains application User Name
Trace in Pre 11.5.7
Identify the Session/Process Id then enable trace as below.
ORADEBUG SETOSPID &&pid
ORADEBUG EVENT 10046 TRACE NAME CONTEXT FOREVER, LEVEL 8
rem ORADEBUG EVENT 10046 TRACE NAME CONTEXT off
Concurrent Report Tracing

Report Level Vs Program level
1. Set Enable Trace flag in the Concurrent Program Definition Form.
(Program level)
2. Enable trace for a specific request (SRS form > Debug Options)
(Request level)
The profile Concurrent: Allow Debugging should be set to Yes.
Recommended way to enable trace.
Enabling Trace at the Program Level
Enabling Trace at the Request Level
Create a Trace File Self Service

To generate DB trace if web pages are slow. There are two options.
1. Set Profile Option FND: Diagnostics to Yes
2. Alternative: Profile Option called Initialization SQL Statement Custom
What to choose from the above two options?
Choose option 2 , if the web page hangs and prevents you from enabling trace.
Create a Trace File Anywhere

Profile Option :
Initialization SQL Statement - Custom
Can always be used, e.g.:
Concurrent Programs
Background processes like Workflow
...
Set on User level only

SQL code will execute for each new DB session which it initialized for an application
user
SQL Statement for a level 12 trace:
begin fnd_ctl.fnd_sess_ctl ('','','TRUE','TRUE','LOG','ALTER

SESSION SET tracefile_identifier=''MYTRACE'' EVENTS=''10046 TRACE
NAME CONTEXT FOREVER, LEVEL 12'''); end;
What Is TKPROF?
Formatted report of SQL Trace file
Includes all SQL statements with statistical info
Allows to identify the performance problem by using different sort
methods
Invoke tkprof from Database Tier (10g/11g)
Getting TKPROF Output

tkprof tracefile.trc output.txt sort=fchela,exeela,prsela
Invoke tkprof from Database Tier
Upload raw trace and tkprof output
i.e. output.txt and tracefile.trc
total
time
# rows
Identify Culprit SQL Statement

Identify Culprit SQL statement is within TKPROF files
Most relevant is TKPROF Sorted by elapsed time
In general the first SQL statement is the culprit
Tip: Always look for large numbers
Gather Enhanced Explain Plan

Isolate the culprit SQL statement in a text file
Gather Enhanced Explain Plan using SQLTXPLAIN.SQL: Note.215187.1
Isolate the expensive DML.
Collect statistics for the tables involved in the query.
exec fnd_stats.gather_table_stats ('APPLSYS','FND_CONCURRENT_REQUESTS',PERCENT=>99);
Any full table scan is used. See any index can be used
Create new custom indexes on custom tables. Its suggested not to create indexes on seeded tables unless
recommended by us.
Re-write the custom query to speed up the query

Ensure Temp, interface tables are purged and defragmented.
Unselective index is being used.
Rebuild the indexes associated only if necessary.
Patch (for Higher version of code ) which resolves it.
Log an SR with Oracle Support
AWR/Statspack Report
DEMO
Demo AWR Report
If you see any DML is consuming more elapsed time, get the explain plan
using sql_id as below. You can run the following commands to obtain the
run-time statistics, as well as past plans recorded in AWR:
SQL> SELECT * FROM
table(dbms_xplan.display_cursor('&sql_id',null,'ALLSTATS')) ;
SQL> SELECT * FROM
table(DBMS_XPLAN.DISPLAY_AWR('&sql_id',null,null,'ALL'));
AWR/ Statspack Report

Top 5 Events section
This section shows the Top 5 timed events that must be considered to focus the tuning efforts.
Top 5 Wait Events
~~~~~~~~~~~~~~~~~
Event
Wait
Waits
Time (cs)
Wt Time
------------------------------------ ------------ ---------db file sequential read
% Total
------------
1,797,963
11,207,291
42.67
db file scattered read
792,958
9,651,032
36.75
buffer busy waits
158,725
1,554,054
5.92
31,317
994,507
3.79
15,489,369
497,062
1.89
log file sync

SQL*Net break/reset to client
-------------------------------------------------------------
AWR / Statspack Report

Handling db file scattered read
This wait happens when a session is waiting for a multi block IO to complete. This typically occurs
during full table scans or index fast full scans.
Check if there are any full table scans. In this case query tuning should be used to optimize the SQL.
See if partitioning can be used to reduce the amount of data you need to scan.
Review SGA Database Buffer Cache
Best Practices for Maximizing the Performance

Techstack upgrade (for Performance & High-Availability)
Upgrade Database to 11.2.0.4 (Note 1058763.1 & Doc ID 881505.1)
OracleAS 10g Release 3 Patch Set 5 (10.1.3.5.0) (Note 454811.1)
Forms bundle patch on10.1.2.3.0 (Doc ID 437878.1)
Upgrade EBS to Latest version 12.1.3 (or) 12.2.4 (Note 269.1 , 1592197.1 )
Apply Recommended Performance Patches (Doc ID 244040.1)

Regular Health check to be performed
Database
Sufficient resource CPU/RAM available (20 - 30% free RAM/CPU is healthy)
Sar & Top process has to be examined (Store it ; can be used later to compare)
Ensure init.ora parameters are as per standards.
AWR/Statspack report examined often.
Forms Server
Sufficient resource RAM & SWAP available
Spinning of forms process
set FORMS_RECORD_GROUP_MAX=10000 (Doc ID 745711.1)
Cancel Query FND: Enable Cancel Query to No
Tweak FORMS60_CATCHTERM & FORMS_CATCHTERM
Servlet Vs Socket Mode https://blogs.oracle.com/stevenChan/entry/which_is_better_forms_servlet_or_socket_mode
Note 384241.1 Using Forms Socket Mode with Oracle E-Business Suite Release 12

Concurrent Manager
Tune using Best Practice Concurrent Manager (Note 1057802.1)
Sufficient resource CPU/RAM available
Sleep time (30 secs) & Cache size (equal to number of process)
Specialized Manger are used if needed.
Purge / de-fragment Concurrent tables
Applications
Gather schema statistics collected at regular interval with 10 40 %
11g Database , Apply Apps Patch 16410424 (12.1.x) 16410424 (12.0.x) 14707975 (11.5.10) to speed to Gather schema Stats.
Leave % as null (Doc ID 1586374.1)
Diagnostics profiles/FRD/Traces are disabled if not needed.
Hardware Sizing.
Sign-on Audit & Audit tables are disabled if not needed.
Purge & De-fragment interface/transaction tables periodically. (Note 752322.1)
Purge portal is available in OAM that can be used to purge Apps.

Network
Audit Network usage.
Tool to help identify the source of the problem
Oracle E-Business Suite Network Utilities: Best Practices [ID 556738.1]
Note 967992.1 Symmetrical Network Acceleration with Oracle E-Business Suite Release 12
Note 786582.1 Performance Degradation in Forms in R12 When not Using Swan Color Scheme
Web Tier (OC4J, Apache & Jserv)

Ensure heapsize is set properly (Doc ID 362851.1 )
Tune s_oacore_jvm_start_option (R12) (Doc ID 362851.1)
$ grep s_oacore_jvm_start_option $CONTEXT_FILE (OS command to check it out)
Tune oacore_nprocs( R12)

Tune ApJServGroup OACoreGroup in jserv.properties (R11i )
Best Practices for maximizing performance (12.2 )

Oacore JVM Configuration
Configure 2 GB JVM Heap Space for oacore.

This can roughly support 150 to 200 users depending on usage.
Default is 512 MB (roughly supports 50 users)
Add more oacore instance to support more users.

use multiple managed instances instead increasing heap size.
Note: You should always size your systems based on tests using above
representative data and workloads for your own environment.
44
Best Practices for maximizing performance

Oacore JVM Configuration
45
Summary
Performance Tuning
Monitor Resource (CPU/ I/O /RAM) Utilization.
Ensure Init.ora standards are met (Note 174605.1)
AWR Report for Database wide performance issue
Best Practice Concurrent Manager (Note 1057802.1)
Purge & Defragment interface/transaction tables. (Note 752322.1)
Gather Schema Statistics at regular Interval.
Apply FND_STATS patch 16410424 (12.1.x) 16410424 (12.0.x) 14707975
(11.5.10) to speed to Gather schema Stats (relevant for 11g Database).
Upgrade Applications & Techstack.
References
A Holistic Approach to Performance Tuning Oracle Applications Systems (Doc ID 69565.1)

Best Practices for Performance for Concurrent Managers in EBS (Doc ID 1057802.1)
Reducing Your EBS Data Footprint using Archiving, Purging, and ILM (Doc ID 752322.1)
bde_chk_cbo.sql - EBS initialization parameters - Healthcheck (Doc ID 174605.1)
Collecting Diagnostic Data for Performance Issues in Oracle E-Business Suite (Doc ID
1121043.1)
Oracle E-Business Suite Recommended Performance Patches (Doc ID 244040.1)
https://blogs.oracle.com/stevenChan/entry/which_is_better_forms_servlet_or_socket_mode
References
JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite
11i and R12 (Doc ID 362851.1).
Forms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications
R12 (Doc ID 745711.1)
Best Practices for Gathering Statistics with Oracle E-Business Suite (Doc ID
1586374.1)
References
Upgrading E-Business Suite.
Which Oracle E-Business Suite Release Should You Target: 12.1 or 12.2? (Doc ID
1592197.1)
Upgrade Advisor: E-Business Suite (EBS) Upgrade from 11.5.10.2 to 12.1.3 (Doc ID
269.1)
Interoperability Notes EBS R12 with Database 11gR2 (Doc ID 1058763.1)
Interoperability Notes Oracle EBS 11i with Oracle Database 11gR2 (11.2.0) (Doc ID
881505.1)
Upgrading to the Latest OracleAS 10g 10.1.3.x Patch Set in Oracle E-Business Suite
Release 12 (Doc ID 454811.1)
Upgrading OracleAS 10g Forms and Reports in Oracle E-Business Suite Release 12
(Doc ID 437878.1)
References
Upgrading E-Business Suite.
These documents provide a starting point for customers moving to Oracle E-Business Suite Release 12.2.
Which Oracle E-Business Suite Release Should You Target: 12.1 or 12.2? (Doc ID 1592197.1)
Scenarios for Getting Started With Oracle E-Business Suite Release 12.2 (Doc ID 1587195.1)
Oracle E-Business Suite Technical Planning Guide, First Edition, Release 12.2 (Doc ID 1585857.1)
Useful 12.2 Documents For Customers And ATG Support Engineers (Doc ID 1585889.1)
Database Preparation Guidelines for an E-Business Suite Release 12.2 Upgrade (Doc ID 1349240.1)
12.2: Consolidated List of Patches and Technology Bug Fixes (Doc ID 1594274.1)
R12.1 and 12.2 Oracle E-Business Suite Preinstall Patches Report [Video] (Doc ID 1448102.2)
Oracle E-Business Suite Recommended Performance Patches (Doc ID 244040.1)
Best Practices for Minimizing Oracle E-Business Suite Release 12 Upgrade Downtime (Doc ID 1581549.1)
Express Diagnosis of Oracle E-Business Suite Release 12 Upgrade Performance Issues (Doc ID 1583752.1)
Oracle E-Business Suite Performance Guide (Doc ID 1672174.1)
Oracle E-Business Suite Release 12.2: Upgrade Sizing and Best Practices(Doc ID 1597531.1)
http://docs.oracle.com/cd/E26401_01/index.htm
You can visit us at
https://communities.oracle.com
Q&A
52
53

EBS Tuning Kanda Session Perf

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

EBS Tuning Kanda Session Perf

Hochgeladen von

Copyright:

Verfügbare Formate

Safe Harbor Statement

The following is intended to outline our general product direction.

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Best Practices In E-Business Suite

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Defining & Isolating issue

How to approach a performance issue

SQL Trace and TKPROF output

Best Practices Maximizing the performance

<Insert Picture Here>

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Oracle Applications Architecture

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

<Insert Picture Here>

Defining and Isolating

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Defining and Isolating the Issue

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Defining and Isolating the Issue

Month end/year end |

Was it working fine earlier?

Who all are affected?

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Defining and Isolating the Issue

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

<Insert Picture Here>

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

<Insert Picture Here>

Approaching Performance Issue

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Approaching Performance Issue

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Approaching Performance Issue

Concurrent jobs are CPU intensive.

Approaching Performance Issue

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Approaching Performance Issue

When was gather schema statistics request run last?

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Approaching Performance Issue

Monitor top OS process at OS level

init.ora setup (Refer note 174605.1 )

Approaching Performance Issue

Check if JVM is hanging (due to Out of Memory issue).

It plays vital role if Applications accessed through VPN/WAN

<Insert Picture Here>

SQL Trace and

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Different Trace Levels

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Create Trace File Form

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Concurrent Report Tracing

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Enabling Trace at the Program Level

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Enabling Trace at the Request Level

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Create a Trace File Self Service

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |

Copyright 2014 Oracle and/or its affiliates. All rights reserved. |