<Insert Picture Here>

Remote Diagnostic Agent & DBA Diagnostics

Tom Wright
Proactive Support Education
tom.wright@oracle.com
This document is for informational purposes. It is not a
commitment to deliver any material, code, or functionality, and
should not be relied upon in making purchasing decisions. The
development, release, and timing of any features or
functionality described in this document remains at the sole
discretion of Oracle. This document in any form, software or
printed matter, contains proprietary information that is the
exclusive property of Oracle. This document and information
contained herein may not be disclosed, copied, reproduced or
distributed to anyone outside Oracle without prior written
consent of Oracle. This document is not part of your license
agreement nor can it be incorporated into any contractual
agreement with Oracle or its subsidiaries or affiliates.
This presentation is for informational purposes only and may not be incorporated into a contract or agreement.


Support Tools
DBA / Developer Resources & Tools
My Oracle Support Configuration Manager
Dozens of Healthchecks and Patch advisors
Oracle Support Lifecycle Advisors
Database Upgrade Troubleshooting Tool
10g &11gR1 & 11gR2 Upgrade Companion
Remote Diagnostic Agent (RDA)
21 pre-built tests and enhanced functions
300+ DBA script library
Performance Tuning Tools (Unix/Linux only)
LTOM Lightweight Transaction OnBoard Monitor
OSW Operting System Watcher
HangFG Hang File Generator
Stackx Stack Trace Extraction
PL/SQL Tuning Scripts
PL/SQL Profiler
SQLXPLAIN Plan
TRACANALYZER
Ora-600/7445 Tool
Remote Diagnostic Agent
RDA is a set of command line, diagnostic and data collection scripts

RDA is used to gather detailed diagnostic information about an Oracle
environment

Oracle Support encourages the use of RDA because it provides a
comprehensive picture of the customer's environment. It collects useful
data for Oracle Support only and a security filter is provided if required.
Why use RDA?
Proactive Problem Avoidance
Reduce cycle time to Service Request resolution.
RDA is essential for the following types of Service Requests:
Acquired company product issues
Developer issues
Installation/configuration issues
ORA-600, ORA-7445, ORA-3113, and ORA-4031 errors
Oracle Database issues
Oracle Application Server/Fusion Middleware issues
Oracle Collaboration products (Oracle Collaboration Suite and Oracle Beehive)
Oracle Application issues
Other corrective issues
Performance issues
Upgrade, migration, and linking issues
Other corrective issues
Why use RDA? (contd)
Characteristics
Compact File Size - < 5MB*
Portable Multi-platform / OSs
Secure coding standards (password encryption)
Security Filter filters out sensitive data (hostnames, IPs etc)
Execution Modes
Profile Options
Modular Options
Full
Products Supported
OLAP Products (Express Server, Financial Analyzer, and Demand Planning Server)
Oracle Application Server (iAS 1.0.2.x/9.0.x/10.1.2.x/10.1.3.x,10.1.4.x,WebLogic Server (WLS) Release 9.x and 10.x,11g
(WLS), HTTP Server,WebCache,J2EE/OC4J)
Oracle Billing and Revenue Management products
Oracle BPEL Process Manager
Oracle Collaboration Suite (Email Server,Calendar,Discussions,Content Services,Workspaces,WebClient,and Wireless)
Oracle Data Integrator
Oracle Developer (Forms and Reports)
Oracle Ebusiness Suite 11i and 12
Oracle Enterprise Content
Oracle Enterprise Single Sign-on
Oracle Enterprise performance management (Hyperion) products
Oracle Guardian
Oracle Identity Management products
Oracle JDBC/PRO *C/ODBC and other development client related information
Oracle Management Server and Intelligent Agent (Grid Server, Agent Server, DB Control)
Oracle Networking products
Oracle RAC Cluster (Single/Multiple Nodes, Automatic Storage Management, Oracle Cluster File System, Data Guard)
Oracle RDBMS Server (Standard and Enterprise Editions)
Oracle Retail (Retek)
Oracle SQL*Plus/iSQL*Plus
Oracle Universal Archive (11g)
PeopleSoft
Siebel
Platforms Supported
Apple Mac OS X/Darwin
HP OpenVMS Alpha 7.3-2 and above
HP OpenVMS Itanium
HP Tru64 UNIX
HP-UX Itanium
HP-UX PA-RISC (32 and 64 bit)
IBM AIX on POWER Systems(32 and 64 bit)
IBM Dynix/Ptx
IBM Linux on POWER
IBM zSeries Based Linux
Linux Itanium
Linux x86 (32 and 64 bit)
Microsoft Windows 2000 Workstation and Server
Microsoft Windows 2003 Server
Microsoft Windows 2008
Microsoft Windows Vista
Microsoft Windows XP Professional
Sun Solaris Intel
Sun Solaris SPARC (32 and 64 bit
Following Platforms are NOT Supported:
IBM Z/OS (OS/390)
RAC Cluster / Multi Node
Starting from release 4.2, RDA has the capability to perform remote
data collections.
Only UNIX and Windows nodes are supported.
RDA can be also installed on shared disk. In a shared disk
context, all nodes must be able to write in that directory.
The collecting node could be outside the cluster if that node has a
database connection to get the cluster topology
RDA against Oracle RAC cluster has been successfully tested to
run on the following Supported platforms:
IBM AIX
Intel Linux (Oracle Enterprise, RedHat, and SuSE)
HP-UX (10.* and 11.*)
HP Tru64
Mac OS X/Darwin
Microsoft Windows (requires Cygwin or MKS)
Sequent Dynix/Ptx
Sun Solaris (2.6 - 2.10)

Security Filters
RDA provides you with a default filter, which currently
filters out the information in the list below.

Domain names
Group names
Host names
IPv4 and IPv6 addresses
LDAP domain components
Network masks
User names

Oracle Configuration Manager
HealthCheck recommendations based on Support best practices
when using configuration auto-collection
Improves Oracle's understanding of your business needs

Personalized access to best practices and the Oracle Knowledge
base
Pro-active configuration-specific notification of Security and
General Alerts
Project cataloging of key milestones and contacts associated with
your configurations
Secure, automated configuration collection
Simplified Service Request logging, tracking and reporting
For troubleshooting information or to learn more about Oracle
Configuration Manager see the following MOS Article 369619.1
Oracle Diagnostic Assistant

The Diagnostic Assistant (DA) tool provides a
common, light-weight interface to multiple diagnostic
collection tools (ADR, RDA, OCM, Explorer).
To learn more about Diagnostic Assistant see the
following Oracle Knowledge 201804.1.
For your benefit, Oracle created an RDA/OCM
bundle. This bundle contains RDA, OCM, and DA
software and it allows customers to install OCM and
DA as part of the RDA data collection process.


RDA 4.27
Internal and External Release
Note: 414970.1
What is new with RDA 4.27?
New - CS module:
Collects Oracle Communications Calendar Server Information
New - MES module:
Collects Oracle Messaging Exchange Server Information
New - MYCL module:
Collects Oracle MySQL Cluster Information
New - MYSQ module:
Collects Oracle MySQL Server Information
New - NCC module:
Collects Network Charging and Control Information
New - OIA module:
Collects Oracle Identity Analytics Information
New - OSSO module:
Collects Oracle OpenSSO Information
New - OW module:
Collects Oracle Waveset Information
Profile-related Changes
New - Com_NCC profile:Oracle Communications Network Charging and
Control problems
Collects OS, PROF, PERF, NET, ONET, DB, DBA, NCC
information
New - MySQLCluster profile:Oracle MySQL Cluster problems
Collects OS, PROF, PERF, NET, MYSQ, MYCL information
New - MySQLDatabase profile:Oracle MySQL Server problems
Collects OS, PROF, PERF, NET, MYSQ information
RDA Engine Changes
Separated the report abbreviation and report prefix concepts
Extended the Heath Check Validation Engine (HCVE)
Introduced an opt-out mechanism
Added support for single-entry menu choice
Resolved variable in text at rule evaluation time instead of at reporting
time
Added a HKCU abbreviation for accessing Windows registry information
Improved cross buffer boundary operations in HTML and XML parsers
Extended conversion to native paths
100+ Improvements to existing modules

Using RDA Proactively
Most customers use RDA reactively after logging a Service Request.

Proactive Benefits
Eliminate Service Requests Problem Avoidance
Automate or schedule on a daily, weekly, or monthly basis
Track changes in the environment
Keep a fresh RDA available in case you need to log a Service Request
Shorten the time to resolution
Software Configuration Manager, configuration profiles
Research performance issues
Viewing RDA Report Output Files
RDA output is not encrypted and can be viewed by anyone. You can view
the RDA report files using almost any web browser.
We recommend using Mozilla 1.x ( or Higher), Netscape 4.x (or higher), or
Internet Explorer 4.x (or higher).

Bundled RDA tools
Run the following modules with the -T mode option

diff Compares Systems
em Runs the Enterprise Manager Tool
jrockt Collects Oracle JRockit Process Information
hcve Executes HCVE Tests
merge Merges Alert Log and Trace Files
na Runs the Network Advisor
ora600 Diagnoses ORA-600 Oracle Internal Errors
oraddc Runs the Oracle Database Diagnostic Collector
root Collects System Log and Executes Commands as root User
secure Identifies Potential Security Risks
siebel Collects Siebel Crash Information
sos Collects siga/sosreport/sysreport
Bundled RDA tests
Run the following modules with the -T mode option

alert Analyzes alert.log
cell Tests Connection to Exadata Cells
core Tests Stack Trace Extraction
db Tests Local Database Access
dbi Tests Oracle Database Access Using the Database Interface
dst Daylight Saving Time Tool Box
env Tests the Environment
event Extracts Event Log Information
inv Tests Oracle Home Inventory Content
ocm Tests Configuration Manager Discovery Information
odbc Tests Oracle Database Access Using ODBC
odi Displays the Current Oracle Data Integrator Module Setup
sql Tests SQL Settings
ssd Analyzes System State Dumps
ssh Tests Remote Connectivity and Operations
vms Verifies Current User Environment for VMS
My Oracle Support
&
RDA Report Sample
RDA 4.x - References
Main RDA 4.10 Overview - Note: 392719.1
Remote Diagnostic Agent (RDA) 4 - RAC cluster Guide
Note: 359395.1
Remote Diagnostic Agent (RDA) 4 - HP OpenVMS Guide
Note: 171748.1
Remote Diagnostic Agent (RDA) 4 - Training
Note:330344.1
Remote Diagnostic Agent (RDA) 4 - Troubleshooting Guide
Note:330362.1
Remote Diagnostic Agent (RDA) 4 - FAQ
Note:330363.1
Remote Diagnostic Agent (RDA) 4 - Main Man Page
Note:330364.1

Reporting Problems or Feedback Diagnostics Community in My Oracle Support
Important Note:If you have issues with the Remote Diagnostic Agent (RDA) , you can file a SR by selecting
OSS Support Tools from the product list of values on the Create a SR screen. Click on "Expand the Product
List" button to see the full set of products. Select the "Remote Diagnostic Agent (RDA) Issue" type and
complete the SR.
Performance Tuning Tools
Hang File Generator Gathers Systemstate and HangAnalyze Trace files
Note:362094.1
OS Watcher Gathers common operating system and network level metrics
Note:301137.1
Lightweight Transaction On-Board Monitor - provides an integrated solution
for detecting and collecting trace files for system performance issues
Note:352363.1
Oracle Performance Diagnostic Guide (OPDG)
Intended to help you resolve query tuning, hang/locking, and slow database issues.
The guide is not an automated tool but rather seeks to show methodologies,
techniques, common causes, and solutions to performance problems
Note:390374.1

Characteristics:
Different tools that you can use to monitor,
diagnose and solve performance issues before and
after issue appears
Performance tools
Value :
Analyze pre-existent Problems
Resolution time Reduction
Very useful on the SRs follow up
Systems Monitor
Adventages:
No aditional Cost.
Available to monitor:
Data Base
Operating System
Network
Oracle Performance Diagnostic Guide



Lets start on MOS over the Knowledge
Browser and Diagnostic Tools
Lets start here
Characteristics:
This guide is intended to help you resolve query
tuning, hang/locking, and slow database issues
OPDG (Oracle Performance Diagnostic Guide)
Value :
Provide you with an step by step
methodology to isolate the problem
and guide you to use our different
tools set.
Advantages :
Provide you with:
Issue Classification Section
Examples
Links to use other oracle tools
Categories:

Classification .- This category will assist you with
determining which type of performance issue you are
encountering.
Query Tuning.- involves issues where a particular query is
running too slow.
Hang/Locking.- involves issues where the entire database
or many sessions are not responding
Slow Database.- involves issues where many sessions are
running slowly
OPDG (Oracle Performance Diagnostic Guide)
Oracle Performance Diagnostic Guide



Classification
Oracle Performance Diagnostic Guide



Query Tuning
Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Hang/Locking
Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Slow Database
Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Oracle Performance Diagnostic Guide



Use
Performance Tools
OPDG Benefits :

Help to Identify and Isolate Performance
problems.

Help to improve your performance
knowledge skills

Help to guide you step by step our oracle
performance tools and documents to solve a
problem
LTOM (The On-Board Monitor)



Embedded Real-Time Data Collection and Diagnostics
Platform

Characteristics:

Real time monitoring and diagnostic tool
Proactive rather than reactive
LTOM runs on the customer's UNIX
server
Provide with a graphical system profiler
Support for RAC
LTOM (The On-Board Monitor)

Value :
LTOM does automatic problem detection and collects the necessary
diagnostic traces in real-time while the database/system performance
problem is occurring. LTOM provides services for:
System Profiler
Automatic Hang Detection
Automatic Session Tracing
LTOM (The On-Board Monitor)

Advantages :
Provides an integrated view of operating system and database
Collect performance metrics and produces a html report
Proactive
LTOM (The On-Board Monitor)

Supported Platforms:
Solaris
Linux
HP-UX
AIX
Tru64
Complete Reference Guide:

Go to MOS Note:352363.1- LTOM - The On-Board
Monitor User Guide
Will find
Documentation
Installation Process
Samples
Features
LTOM (The On-Board Monitor)

LTOM (The On-Board Monitor)
Directory Structure

Use
Performance Tools
System Profiler:
When to use it?
The system profiler is useful to frame performance
issues where a bottleneck may be outside Oracle.
Benefits:
Collect data up to just seconds prior to hang or crash
Collect Operating System data in additional to oracle
performance data.
Collect statistical data down to 1 second increments
Displays SQL currently executing.
RCA timeline.
Use
Performance Tools
System Profiler:
Use
Performance Tools
Automatic Hang Detection:
When to use it?
Any problem involving hangs/slowdowns when the
necessary information collected at the initial outage is
insufficient to diagnose the problem.
Benefits:
Collect systemstates and hanganalyze files during the
actual hang without operator intervention
Hang data collection 24x7
Hangs automatically detected
Email notification of hang
Use
Performance Tools
Automatic Session Tracing:
When to use it?
should be considered for situations where specific
sessions experience performance problems.
Benefits:
Collect 10046 trace only when a performance
problem occurs
Collect SQL associated with a session's performance
problem
Tie data collection to a specific Oracle wait event or
CPU utilization
Session tracing for only problematic sessions
OSWatcher


Characteristics:
OSW consists of a series of shell scripts.
Data collection intervals are configurable.
OSW invokes distinct operating system utilities
ps
top
mpstat
iostat
netstat
traceroute
vmstat
OSWatcher
Simple Installation:
OSWatcher
uncompress osw.tar.Z
tar xvf osw.tar

More information. Note:301137.1
Use
OSWatcher
OSW Benefits:

Simple tool that use pre existent Operating
System commands

Easy to use

Trace Analyzer TRCANLZR


Interpreting Raw SQL Traces with Binds and/or Waits
generated by EVENT 10046

Characteristics:
Given a Raw SQL Trace with Binds and/or
Waits, generated by EVENT 10046 Levels 1, 4,
8 or 12
Generates a comprehensive report that can be
used to troubleshoot common SQL performance
issues, or functional issues requiring to find the
values of bind variables






Trace Analyzer TRCANLZR
Installation:
Trace Analyzer TRCANLZR
You should unload and uncompress the
TheTRCANLZR file and then execute some
configuration steps.

The tools can be executed on a different
database where the raw file was generated

More information available
Note:224270.1
Use
Trace Analyzer TRCANLZR
TRCANLZR Beneffits:

Simple tool that generate a comprehensive
output to analyze raw files including bind
variables, wait events

The Trace Analyzer provides an accurate
SUM of CALLs
SQL Tuning Scripts
PL / SQL Profiler
Helps to identify the lines of PL/SQL code which are taking longer to process
Note:243755.1
SQLTXPLAIN.SQL
Generates comprehensive diagnostic files normally used in the area of SQL
analysis and tuning.
Note:301137.1
Trace Analyzer -TRCANLZR
Reads and parses one or several SQL traces and produces a comprehensive report
with performance related details.
Output report includes all the details found on TKPROF, plus additional information
normally used for a process performance analysis
Note:224270.1
SQLTXPLAIN


Enhanced Explain Plan
and related diagnostic
info for one SQL
statement
Complete Reference Guide
Note:215187.1
Characteristics:

Designed to help during the diagnostic phase of
SQL performance issue.

Provide a comprehensive report

Facilitates the creation of test cases on
separate environments



SQLTXPLAIN
SQLTXPLAIN.SQL
Improved Explain Plan
Install
Sqlplus sys as sysdba
SQL> @sqcreate.sql
Execute some modules, Sql
Statements
Create a report with the tool
Check the output via Internet Browser
3 different ways:

1. Xplain - Standard - Generates Explain Plan,
SQL is not executed
2. Xtract - Advanced - Generates Explain Plan,
SQL is in memory already
3. Xecute - Special - SQL is executed, then
Execution Plan is obtained

Using the SQLTXPLAIN:
Improved Explain report
Use
SQLTXPLAIN
SQLTXPLAIN Benefits:

Improved explain plan
Very useful to diagnose sql performance
problems
Solve some known issues with the standard
explain plan.
You can upload the output information to
your SRs.
Performance Tools Quick Reference Guide
Now Available Through MOS!

A quick guide to available tools
Can be used for resolving performance issues.
Can be used for follow up Service Requests
Include documentation about different performance
tools
Go to MOS Note:438452.1- Performance Tools Quick
Reference Guide