Best Practice of Oracle Monitoring by ITCAM Extended Agent

Jan 2015

Best Practices of Oracle

Database Monitoring by ITCAM
Extended Agent

Document version [0.5]

Tivoli China Development Lab

 Copyright International Business Machines Corporation 2010, 2015.
US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.

Page 2 of 52
 CONTENTS

Contents.............................................................................................................................iii

List of Figures....................................................................................................................vi

List of Tables.....................................................................................................................vii

Revision History...............................................................................................................viii

1Overview..........................................................................................................................1

1.1Planning your enterprise monitoring infrastructure............................................1

1.2Reference sample Oracle database environment.............................................1

2Strategy of Monitoring.......................................................................................................3

2.1Local Monitoring or Remote Monitoring.............................................................3

2.2Oracle Instant Client or Oracle Home................................................................3

2.3Database Connection Number per Agent Instance...........................................4

3Agent Installation and Configuration.................................................................................5

3.1Remote installation............................................................................................5

3.2Apply the latest Fix Pack and Interim Fix..........................................................5

3.3Oracle Instant Client..........................................................................................5

3.4Mount Oracle alert log directory from remote DB server...................................6

3.5Configure LISTENER.ORA and TNSNAMES.ORA files in agent side...............6

iii
 3.6Monitor IPC/TCPS listener................................................................................8

3.7Agent configuration from command line............................................................8

3.8ASM database monitoring.................................................................................9

3.9Dataguard database monitoring......................................................................10

3.10[Optional] Use non-root user to startup agent to monitor Oracle database....11

3.11Check the status for connections..................................................................12

4Customization.................................................................................................................14

4.1Extend agent timeout......................................................................................14

4.2Listener Monitoring..........................................................................................14

4.2.1Disable the Listener/Net Service Monitoring..................................................14

4.2.2Selected Listener/Net Services to be monitored............................................14

4.3Alert Log Monitoring........................................................................................15

4.4Customized SQL.............................................................................................16

4.5Product Predefined SQL Customization..........................................................16

4.6Cluster Support...............................................................................................16

4.7Usage of KRZ_SQL_PREFILTER=TRUE/FALSE...........................................16

5FAQ................................................................................................................................18

5.1The SQL query the agent uses.......................................................................18

5.2How the agent checks net service status........................................................18

iv
 5.3About RDB_Instance_Info/ASM_Instance attribute groups.............................19

6Oracle Monitoring Concept and Situations......................................................................20

6.1Monitor Oracle Availability...............................................................................20

6.1.1Monitor Oracle Instance Status......................................................................20

6.1.2Monitor Oracle Tablespace............................................................................22
6.1.3Monitor Oracle Automatic Storage Management (ASM)................................23
6.1.4Monitor Oracle Real Application Cluster (RAC).............................................24
6.1.5Monitor Oracle Database...............................................................................25
6.1.6Monitor Oracle Alert Log................................................................................26

6.2Monitor Oracle Performance...........................................................................27

6.2.1Buffer Cache Hit Ratio...................................................................................27

6.2.2System Global Area (SGA)............................................................................29
6.2.3Top SQL........................................................................................................31
6.2.4Lock Contention and Dead Locks..................................................................32

7Problem Determination...................................................................................................35

Appendix............................................................................................................................vi

8Appendix A. How to use Oracle Instant Client for Tivoli Extended Oracle Agent..............vi

References.........................................................................................................................8

v
 LIST OF FIGURES

vi
 LIST OF TABLES

Oracle Node, Instance Name, Server Name and the Net Service name in the Listener...................1
Compare of Local monitoring and Remote monitoring.....................................................................2
Environment variable for agent timeout.............................................................................12
Environment variables for alert log monitoring..................................................................13
Agent Affinities...................................................................................................................14

vii
 REVISION HISTORY

Date Version Revised By Comments

2011/4/27 0.1 liujingq@cn.ibm.com initials
2011/10/17 0.2 chengliu@cn.ibm.com Add more best practices items
2012-4-30 0.3 chengliu@cn.ibm.com Add ASM/Dataguard
10/15/12 0.4 chengliu@cn.ibm.com Add sample monitoring situations
1/26/15 0.5 liyane@cn.ibm.com Revise and add new items

viii
1 Overview
The IBM Tivoli Composite Application Manager Extended Agent for Oracle Database
provides you with the capability to monitor Oracle Database. You can also use the agent to
take basic actions with the Oracle Database.

IBM Tivoli Monitoring is the base software for the Oracle Database Extended agent. The
Oracle Database Extended agent monitors general Oracle database performance, Oracle
RAC performance, Oracle ASM performance, and Data Guard performance.

You can get detail about agents features from web link below:

http://www-
01.ibm.com/support/knowledgecenter/SS3JRN_7.2.1/com.ibm.itcama.doc_7.2.1/oracleext
ended/fac_landing_install.html?cp=SS3JRN_7.2.1.1%2F10-0&lang=en

Oracle Real Application Cluster Database (RAC) support users to access the same data
from multiple instances on different servers. In RAC environment, the instance level
information includes multiple instances information; the database level information has only
one copy.

For example, you can get following information for each instance: Instance Name, SGA,
Session, and Process. The database level information includes Tablespace, Table, Index
and Datafile, such kind of database level information is consistent and shared by all
instances in a RAC environment.

This document will introduce the best practices in using the Oracle Extended Agent to
monitor general Oracle database, RAC database, ASM database or Dataguard database.

1.1 Planning your enterprise monitoring infrastructure

The Oracle Extended Agent v6.3.1 requires IBM Tivoli Monitoring (ITM) Infrastructure
version 622FP2 or higher.

1.2 Reference sample Oracle database environment

The following Oracle Database environment will be used as sample environment in this
document.

The RAC database is installed into two Solaris Sparc servers as two nodes (RAC1 and
RAC2). The listener is running on port 1521 in both two nodes.

The Oracle cluster ware (including ASM) is installed at /opt/oracle/grid, the Oracle
database software is installed at /opt/oracle/db. The Oracle database servers alert
log location is /opt/oracle/base/diag/rdbms/rac1/trace/alert_rac1.log. The Oracle software
version is 11.2.0.1.

The Oracle RAC database myrac is created with two RAC instances, one instance name
is rac1 in the RAC1 node, and the other instance name is rac2 in the RAC2 node.

1
The Oracle ASM database +ASM is created and there are also two instance, one instance
is +asm1 in the RAC1 node, the other is +asm2 in the RAC2 node.

The Oracle Dataguard database is installed into two Solaris Sparc Sun OS 5.10 servers as
Primary Node in the machine sunpri and standby node in the machine sunstd.

Table 1: Oracle Node, Instance Name, Server Name and the Net Service name in the Listener

Oracle Node Type Oracle Instance Name Server Name Net Service Name in
Listener
RAC1 rac1 sol_rac1 myrac
RAC2 rac2 sol_rac2 myrac
ASM1 +ASM1 sol_rac1 +ASM
ASM2 +ASM2 sol_rac2 +ASM
Dataguard Primary oradb sol_dg1 dbpri
Dataguard Standby oradb sol_dg2 dbstd

2
2 Strategy of Monitoring
This chapter introduces the monitoring strategy with the Oracle Database Extended Agent
about the agent capacity, limitation and scalability.

2.1 Local Monitoring or Remote Monitoring

Oracle extended agent support local monitoring and remote monitoring. Following table
compares these two different monitoring ways.

Table 2: Compare of local monitoring and remote monitoring

Category Local Monitoring Remote Monitoring

Installation in Oracle Yes No
Server machine
Oracle physical disk Yes Not support
usage monitoring
Oracle alert log Yes Yes, need to mount the remote
monitoring machines log directory to local,
and need to specify mount path
at agent configuration
IPC listener monitoring Yes Not support
Other monitoring Yes Yes
features
Maintenance effort Big, need to apply agent patch Little, only need to apply patch
in each Oracle server at agent installation machine
machines. once.

Note: For a RAC environment, we recommend installing the agent into one node of the
RAC environment, and then the agent performs the remote monitoring. The disk
usage for local node can still be monitored; the alert log for remote nodes can be
monitored after mount the alert log directory at the remote node.

2.2 Oracle Instant Client or Oracle Home

The Oracle Database Extended agent is built with Oracle Call Interface (OCI) technology,
the agent can use the dynamic library provided as Oracle Instant Client or Oracle server
installation (as Oracle home).

We recommend using the Oracle instant client, because the user can always use the latest
version of Oracle Instant Client package which contains the latest Oracle client dynamic
library code fixes. By using the latest Oracle Instant Client, the user does not need to
upgrade the Oracle server in case there are some known bugs in Oracle Client dynamic
library.

Oracle Instant Client can be downloaded from the Oracle web site:
http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html

3
2.3 Database Connection Number per Agent Instance
We suggest each agent instance monitor no more than 10 database connections in case
there are no more than 5 situations or historical collections in one agent.

This will ensure the total request number in an agent instance is less than 50(10
connection * 5 situations/historical collections). If there are more situations or historical
collections, then we suggest reducing the db connection number for each agent instance
and adding more agent instances.

The number of agent instances is affected by ITM limitation. Including other agents, a
maximum of 15 agent instances can be started on a single system by default. This is a
common ITM limitation, but the limitation can be extended by changing agent instances
configuration, pls refer to Resolving the problem in the below ITM technote:

No more that 15 IP.PIPE processes can be active on a single system image.

http://www-01.ibm.com/support/docview.wss?uid=swg21263550

4
3 Agent Installation and Configuration
3.1 Remote installation
The Oracle Database Extended agent supports remote installation and you can install the
agent into a separated remote machine with Oracle servers.

Example The agent installation directory is /opt/IBM/Tivoli.

3.2 Apply the latest Fix Pack and Interim Fix

By Jan.2015, the latest version of the agent is 06.31.02.02.

We strongly recommend you apply the latest interim fix 6.3.1.2-TIV-ITM_KRZ-IF0002 if the
agent version is in 06.31.02.00 in your environment, since 6.3.1.2-TIV-ITM_KRZ-IF0002
includes important fixes.

6.3.1-TIV-ITM_EXT-FP0002:
http://www-01.ibm.com/support/docview.wss?uid=swg24035571

6.3.1.2-TIV-ITM_KRZ-IF0002:
http://www-01.ibm.com/support/docview.wss?uid=swg24038442

For the most current information about the latest fix pack or interim fix, see the agents wiki
page:
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli
%20Composite%20Application%20Manager/page/Oracle_Extended_Agent

3.3 Oracle Instant Client

If there is no Oracle Home or Oracle Client installed in the agent machine by the Oracle
OUI installer, then you need to download the Oracle Instant Client package and extract it to
one readable directory in the agent machine, for example,
/opt/IBM/Tivoli/instantclient_11_2.

Note: The LITE version of Oracle Instant Client is NOT supported.

You need to create a soft link from libclntsh.so.11.1 to file libclntsh.so in the Oracle Instant
Client directory.

Example #ln -s /opt/IBM/Tivoli/instantclient_11_2/libclntsh.so.11.1

/opt/IBM/Tivoli/instantclient_11_2/libclntsh.so

Note: There is a full step-by-step guide for the Oracle Instant Client downloading in the
Appendix A. How to use Oracle Instant Client for Oracle Database Extended
Agent.

5
3.4 Mount Oracle alert log directory from remote DB server
The agent monitors Oracle alert log by reading alert log file configured in agent instance
configuration item - Oracle Alert Log File Paths (KRZ_LOG_PATHS).

If the agent is installed in the remote server, Oracle alert log directory (for example
/opt/oracle/base/diag) in the Oracle server need to be mounted to the agent
machine.

Following are sample steps:

Step 1) Check if the NFS server is started in the Oracle server, or run the following
command as user root.
# /etc/init.d/nfs.server start

Step 2) Check alert log directory permission in the Oracle server and add the rx
permission for directory recursively to ensure the mount client can access the
mounted alert log files, run the following command as user root.
# chmod -R +rx /opt/oracle/base/diag

Step 3) Share the alert log directory with following command in the Oracle server rac1.
# share -F nfs -o ro=rac1 /opt/oracle/base/diag

Step 4) Create a local directory and mount the remote alert log directory into local
directory.
# mkdir /mnt/rac1
# mount rac1:/opt/oracle/base/diag /mnt/rac1

3.5 Configure LISTENER.ORA and TNSNAMES.ORA files in

agent side
The agent gets listeners and Net Service names information from listener definition file
listener.ora and Net Service name definition file tnsnames.ora, and can monitor the
status of listeners and Net Service names.

There are 3 places where the agent tries to read listener.ora and tnsnames.ora:

If you configure the agent instance with Oracle Home (KRZ_ORACLE_HOME), the agent
tries to read listener.ora and tnsnames.ora from KRZ_ORACLE_HOME/network/admin on
Unix or KRZ_ORACLE_HOME\network\admin on Windows.

If the environment variable TNS_ADMIN is defined in agent running environment or agent

configuration files, the agent also tries to read listener.ora and tnsnames.ora from
TNS_ADMIN/network/admin on Unix or TNS_ADMIN\network\admin on Windows.

You can also copy listener.ora and tnsnames.ora files from remote DB server systems or
create self-defined listener.ora and tnsnames.ora files, put them into a directory on the
system where the agent is installed, and configure their directory path into agent instance
configuration item - Net Configuration Files Directories (KRZ_TNS_PATHS). Multi-
directories are allowed to be configured. The agent tries to read listener.ora and
tnsnames.ora from here as well.

6
Following is sample step to create self-defined tnsnames.ora:

Use editor like Vi to create a new file tnsnames.ora file in the server tivsun21 at
/opt/IBM/Tivoli/config directory with following sample lines. The file permission
should be set as rx-rx-rx.

Note: Connection name of TIVRAC must start from the far left of the line; other lines should
NOT start from left of the line.

Example # cat /opt/IBM/Tivoli/config/tnsnames.ora

#RAC database connection string
TIVRAC =
(DESCRIPTION=
(LOAD_BALANCE=ON) (FAILOVER=on)
(ADDRESS = (PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521
))
(ADDRESS = (PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521
))
(CONNECT_DATA=(SERVICE_NAME=myrac)
)
)
#ASM database connection string
TIVASM =
(DESCRIPTION=
(ADDRESS = (PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521
))
(ADDRESS = (PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521
))
(CONNECT_DATA=(SERVICE_NAME=+asm)
)
)
#Dataguard database primary node connection string
TIVDGPRI =
(DESCRIPTION=
(ADDRESS = (PROTOCOL=tcp)(HOST=sol_dg1)(PORT=1521)
)
(CONNECT_DATA=(SERVICE_NAME=dgpri)
)
)
#Dataguard database standby node connection string
TIVDGSTD =
(DESCRIPTION=

7
 (ADDRESS = (PROTOCOL=tcp)(HOST=sol_dg2)(PORT=1521)
)
(CONNECT_DATA=(SERVICE_NAME=dgstd)
)
)

3.6 Monitor IPC/TCPS listener

For Inter-process communication (IPC) protocol listener monitoring. Only local agent (local
agent means the agent is installed on the same system where the monitoring Oracle server
resides) is able to monitor IPC listener; remote agent is unable to monitor IPC protocol
listener, because the IPC listener only works on the local Oracle server.

For TCPS listener monitoring. You need to set TNS_ADMIN environment variable into
agent running environment or agent configuration files. The TNS_ADMIN path need to
include net service configuration file sqlnet.ora, and the sqlnet.ora file need to include
necessary information for TCPS listener monitoring, for example WALLET_LOCATION. If
sqlnet.ora does not include necessary information, the TCPS listener status is shown as
Inactive, and you will see the following error message in agent log file:

"Oracle error at ServerAttach during Net Service checking" error code is:28759

3.7 Agent configuration from command line

Launch agent configuration from command line and input the specific value as following.

Launch configuration tool from command line.

Example #/opt/IBM/Tivoli/bin/itmcmd config -A rz

When ask for the input filed of Oracle Home Directory, input a
blank space to make sure there is no valid value for Oracle Home Directory

Example Oracle home directory: (default is: ): [Input

space and press Enter]

When ask for the input field of Oracle Instant Client Installation
Directory, input the path where the Oracle Instant Client is extracted.

Example Oracle instant client installation directory:

(default is: ):
/opt/IBM/Tivoli/instantclient_11_2

When ask for selection to edit the database connection

settings, input 1

Example Edit 'Database connection' settings, [1=Add,

2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 5): 1

When ask for the input filed of Database Connection Name,

input the database name rac.

Example Database connection name: (default is: ): rac

8
 When ask for the input field of Oracle Connection String, input
the tnsnames.ora connection identifier which is defined in the file
/opt/IBM/Tivoli/config/tnsnames.ora, current value is TIVRAC. Or you can input
the full connection string directly.

Example Oracle connection string: (default is: ): TIVRAC

Example Oracle connection string: (default is: ):

(DESCRIPTION=(LOAD_BALANCE=ON)(FAILOVER=ON)
(ADDRESS=(PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521)
)(ADDRESS=(PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521
))(CONNECT_DATA=(SERVICE_NAME=myrac)))

When ask for the input filed of Oracle alert log file paths, input
the local alert log file full paths which is mounted from remote Oracle servers.

Example The absolute file path of mapped alert log files

of remote database instances in this database
connection. Multiple files are separated by ";"
on Windows systems, or ":" on UNIX systems. Each
file is matched to a database instance, by the
file name pattern alert_<instance>.log, or
ignored if unmatched. Local database instance
alert log files can be discovered automatically.
If this item was not configured, the alter logs
of remote database instances would be unable to
be collected. Oracle alert log file paths:
(default is: ):
/mnt/sol_rac1/rdbms/rac/rac1/trace/alert_rac1.log
:
/mnt/sol_rac2/rdbms/rac/rac2/trace/alert_rac2.log

3.8 ASM database monitoring

You can use existing agent instance to add one new connection to monitor one ASM
database or create a new agent instance with a new connection.

When ask for selection to edit the database connection

settings, input 1

Example Edit 'Database connection' settings, [1=Add,

2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 5): 1

When ask for the input field of Database Connection Name,

input the connection name like asm.

Example Database connection name: (default is: ): asm

Note: Only letters, Arabic number, underline and minus characters can be used in the
connection name. Other connection name like +asm is invalid.

When ask for the input field of Oracle Connection String, input
the tnsnames.ora connection identifier which is defined in the file
/opt/IBM/Tivoli/config/tnsnames.ora, current value is TIVASM. Or you can input
the full connection string directly.

9
 Example Oracle connection string: (default is: ): TIVASM

Example Oracle connection string: (default is: ):

(DESCRIPTION=(LOAD_BALANCE=ON)(FAILOVER=ON)
(ADDRESS=(PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521)
)(ADDRESS=(PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521
))(CONNECT_DATA=(SERVICE_NAME=+ASM)))

When ask for the input field of Username, Password and

Role, input the user information with SYSDBA role.

Example The set of privileges to be associated with the connection. For a

user that was granted the SYSDBA system privilege, you can
specify a connection that includes the privilege. If this item was not
defined, 'DEFAULT' would be used as the role of the user.
Role: [ 1=SYSDBA, 2=SYSOPER, 3=SYSASM, 4=DEFAULT ]
(default is: 4): 1

Note: In Oracle 10g ASM database, a new Oracle user cannot be created in the sqlplus
directly, you need to create the new Oracle user and grant as SYSDBA role in the
RDBMS sqlplus and copy the RDB password file to the ASM password file,
reference the below technote:

Create a database user for ASM with Oracle 10g

http://www-01.ibm.com/support/docview.wss?uid=swg21589173

3.9 Dataguard database monitoring

You can use existing agent instance to add two new connections for Dataguard Primary
database and Standby database separately, or you can create a new agent instance with
two new connections.

When ask for selection to edit the database connection

settings, input 1 for Primary Node connection.

Example Edit 'Database connection' settings, [1=Add,

2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 5): 1

When ask for the input filed of Database Connection Name,

input the connection name dgpri for the primary node connection.

Example Database connection name: (default is: ): dgpri

When ask for the input field of Oracle Connection String, input
the tnsnames.ora connection identifier which is defined in the file
/opt/IBM/Tivoli/config/tnsnames.ora, current value is TIVDGPRI. Or you can input
the full connection string directly.

Example Oracle connection string: (default is: ):

TIVDGPRI

Example Oracle connection string: (default is: ):

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=sol_dg1
)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=dgpri)))

10
 When ask for the input field of Username, Password and
Role, input the user with SYSDBA role. The user with SYSDBA role is not
mandatory for Dataguard Primary Node,

When ask for the selection to edit the database connection

settings, input 1 for the Standby Node connection.

Example Edit 'Database connection' settings, [1=Add, 2=Edit, 3=Del,

4=Next, 5=Exit] (default is: 5): 1

When ask for the input filed of Database Connection Name,

input the connection name dgpri for the primary node connection.

Example Database connection name: (default is: ): dgstd

When ask for the input field of Oracle Connection String, input
the tnsnames.ora connection identifier which is defined in the file
/opt/IBM/Tivoli/config/tnsnames.ora, current value is TIVDGPRI.

Example Oracle connection string: (default is: ):

TIVDGSTD

Example Oracle connection string: (default is: ):

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=sol_dg1
)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=dgstd)))

When ask for the input field of Username, Password and

Role, input the user with SYSDBA role.

Example The set of privileges to be associated with the connection. For a

user that was granted the SYSDBA system privilege, you can
specify a connection that includes the privilege. If this item was not
defined, 'DEFAULT' would be used as the role of the user.
Role: [ 1=SYSDBA, 2=SYSOPER, 3=SYSASM, 4=DEFAULT ]
(default is: 4): 1

3.10 [Optional] Use non-root user to startup agent to monitor

Oracle database
If you want to use ITM utility $ITMHOME/bin/itmcmd to startup agent appending with agent
instance name in the -o option by non-root user, eg. tivmon, you need to grant the r+w
permission of files installdir/config/.Config/RunInfo* and installdir/logs/*rz* to the non-
root user before starting the agent.

Example #su - tivmon -c /opt/IBM/Tivoli/bin/itmcmd agent

-o inst1 start rz

3.11 Check the status for connections

Go to the Oracle Database Extended agent root node and check the Database Status
column in the view of Database Connection.

11
If the Database Status is not Active or there is not data in the view of Database
Connection, then check the Error Message column and Suggestion column in the
Agent Event workspace.

Figure 1: Blank Database Connection and Agent Event

Figure 2: Dataguard Database Connections

12
Figure 3: ASM Database Connection

13
4 Customization
4.1 Extend agent timeout
You can set the environment variable below to extend the agent itmeout from default 60
seconds to more.

Table 3: Environment variable for agent timeout

Variable Description

CDP_COLLECTION_TIMEOUT The maximum time of data collection. If the

execution time exceeded, the agent stops
the execution, and reports timeout. The
default value is 60 (in seconds). The
minimum value is 5 (in seconds).

4.2 Listener Monitoring

4.2.1 Disable the Listener/Net Service Monitoring

The Oracle Database Extended agent version 06.31.00.01 and above versions provide
configuration parameters to disable Listener/Net Service monitoring:

Enhancement: INTERNAL

Abstract: Providing the option to control whether monitoring the listener and Oracle net-
service.

Additional Information: The listener monitoring and net-service monitoring are started by
default. We provide the option that if you do not want this monitoring; you can turn off the
related monitoring functions.

The environment variables KRZ_LISTENER_PING_INTERVAL and

KRZ_TNS_PING_INTERVAL control the interval. If the value is less than 1, then it exits the
thread and stops checking.

Example KRZ_LISTENER_PING_INTERVAL=0

Example KRZ_TNS_PING_INTERVAL=0

4.2.2 Selected Listener/Net Services to be monitored

If the Oracle home directory is selected during the agent configuration, then agent will load
Listener definition from file $ORACLE_HOME/network/admin/listener.ora and load Net
Service definition from file $ORACLE_HOME/network/admin/tnsnames.ora automatically.

14
 If user do not want to monitor all Listener or Net Service from ORA files in the Oracle home
directory and do not want to revise those files too, customer can specify the TNS_PATH
during the agent configuration and create new listener.ora file and tnsnames.ora files in the
new path and only the targeted Listener or Net Service is specified in the new files.

Example This is the directory that contains Oracle database net

configuration file. This directory is defined by the TNS_ADMIN
environment variable for each Oracle database instance. The
default directory is $ORACLE_HOME/network/admin on UNIX or
Linux systems, and %ORACLE_HOME%\NETWORK\ADMIN on
Windows systems. If there are multiple net configuration file
directories, use ";" on Windows systems, or ":" on UNIX systems,
to separate the directories. If this item was not configured, the
default directory would be used. Net configuration files directories:
(default is: ): /opt/IBM/Tivoli/config/myrac

4.3 Alert Log Monitoring

To customize the severity of Oracle alert log messages, reference Info Center:

http://www-
01.ibm.com/support/knowledgecenter/SS3JRN_7.2.1/com.ibm.itcama.doc_7.2.1/oracleext
ended/fac_config_agentspecific_customizingoraclealertlog.html?
cp=SS3JRN_7.2.1.1%2F10-0-2-2-6&lang=en

You can set environment variable in the table below for the alert log monitoring.

Table 4: Environment variables for alert log monitoring

Variable Description

KRZ_LOG_MAXREAD Only the latest part of alert logs is scanned

by the Oracle Database Extended agent
while the agent starts. The size of the
scanned part is specified by the
KRZ_LOG_MAXREAD attribute. The default
value is 5M. The value must be positive.
The valid data suffixes are M/m presenting
megabyte, and K/k presenting kilobyte.

KRZ_LOG_INTERVAL This attribute defines the search interval (in

seconds) for alert log files. The default value
is 300 (seconds). Setting the value to 0
disables search. The valid range is from 60
to 3600.

KRZ_LOG_NOREPEAT If the value is true, the Oracle Database

Extended agent does not re-send alert log
message entries that were found in the last
running of the Oracle Database Extended
agent. The default value is FALSE. The
valid values are TRUE and FALSE.

15
4.4 Customized SQL
To define a customized SQL statement to monitor in the Customized SQL workspace,
reference Info Center:

http://www-
01.ibm.com/support/knowledgecenter/SS3JRN_7.2.1/com.ibm.itcama.doc_7.2.1/oracleext
ended/fac_config_agentspecific_definingsql.html?cp=SS3JRN_7.2.1.1%2F10-0-2-2-
8&lang=en

4.5 Product Predefined SQL Customization

You can use the environment variable KRZ_CUSTOM_SQLXML to specify a customized
krzsql_custom.xml with the revised SQL query for the existing agent attribute groups like
RDB_Session_Detail, reference below technote:

http://www.ibm.com/support/docview.wss?uid=swg21588392

4.6 Cluster Support

Reference to below technote:

Agent clustering and positioning in the TEP Navigator

http://www-01.ibm.com/support/docview.wss?uid=swg21326517

Table 5: Agent Affinities

Affnities Node in the TEP

%IBM.OracleAgents Oracle Database Extended
%IBM.OracleAgentRDB Oracle RDBMS
%IBM.OracleAgentASM Oracle ASM
%IBM.OracleAgentDG Oracle Dataguard

4.7 Usage of KRZ_SQL_PREFILTER=TRUE/FALSE

It is default behavior that the Oracle Database Extended agent adds prefilter into SQL
query that is sent to the Oracle database for data processing, and then returns data to the
Tivoli Enterprise Monitoring Server with records matched with the filter condition. The
purpose of this behavior is to improve the agent performance by reducing the data to be
queried and sent.

The prefilter information is usually gotten from situation's formula. When the agent receives
a request from the situation, this condition will be combined into the SQL query.

For example (Example 1):

A situation with formula tablespace_used_percentage >= 80%

The formula '>=80%' will be used in the where clause as below:

16
SELECT tablespace_name, tablespace_used_percentage, ... FROM gv$tablespace_table
WHERE tablespace_used_percentage >= 80;

But the agent has problem to handle the situation which has override formula/includes the
original situation's formula condition info. The root cause is that the override formula info
cannot be passed to agent in current ITM agent factory framework.

For example (Example 2):

A situation with formula tablespace_used_percentage >= 80% and the override formula is
tablespace_name = 'TEST' and tablespace_used_percentage >= 50.

Only the formula '>=80%' will be used in the where clause as below, because override
formula info cannot be passed to agent in current ITM agent factory framework:

SELECT tablespace_name, tablespace_used_percentage, ... FROM gv$tablespace_table

WHERE tablespace_used_percentage >= 80;

In above example, agent will only query data from Oracle which matches with original
formula of >=80%, so the other tablespace which matches with override formula will not be
queried and it will not be fired in the TEMS anyway. This is a potential drawback, as it can
cause alerts to be missed.

To resolve this issue, This parameter KRZ_SQL_PREFILTER has been introduced by

Oracle Extended Agent 6.3.1 Fix Pack 2. The valid values are TRUE and FALSE. To be
consistent with previous releases, the default value is TRUE, and the agent has same
behavior as previous releases. If you have override situation defined which is similar with
example 2, you need to set KRZ_SQL_PREFILTER to FALSE in agent instance
configuration file, and restart the agent instance. When KRZ_SQL_PREFILTER is set to
FALSE, the agent will not use any prefilter, all data for the attribute group will be queried
from database and be parsed to TEMS.

17
5 FAQ
5.1 The SQL query the agent uses
You can get query the agent uses for most attribute groups from the below file on the
system where the agent is installed.
UNIX:
<CANDLEHOME>/<Arch>/rz/bin/krzsql.xml
WINDOWS:
<CANDLEHOME>\TMAITM6(_x64)\krzsql.xml

Take "ASM Disk Group Capacity" workspace view as example, the corresponding attribute
group is ASM_DiskGroup. Search ID="ASM_DiskGroup" from krzsql.xml, you can get the
query for it:

For Oracle 10.1 and above version:

<SQL DBVersion="101">
...

For Oracle 10.2 and above version:

<SQL DBVersion="102">
...

For Oracle 11.1 and above version:

<SQL DBVersion="111">
...

5.2 How the agent checks net service status

For agent v6.31.00.02 and above. If you configure the agent instance with Oracle Home
(KRZ_ORACLE_HOME) and does not set KRZ_USE_ORACLE_TNSPING=FALSE, the
agent will try to use KRZ_ORACLE_HOME/bin/tnsping on Unix or
KRZ_ORACLE_HOME\bin\tnsping.exe on Windows to test the status of listeners by
analyzing the output of tnsping utility. If you configure the agent instance with Oracle
Instant Client, the agent will use Oracle OCI library API to get the status of listeners.

The agent always use Oracle OCI library API to get the status of net service names.

If the environment variable TNS_ADMIN is NOT set or there is not sqlnet.ora file in
TNS_ADMIN, the agent generates net service configuration file
<Installation_Dir>/logs/<Host_Name>_rz_<Agent_Instance_Name>/sqlnet.ora, for
example,

-bash-3.2# cat sqlnet.ora

# SQLNET.ORA Logging and Tracing parameters, generated by Oracle Extended Agent
DIAG_ADR_ENABLED=off
LOG_DIRECTORY_CLIENT=<Installation_Dir>/logs/<Host_Name>_rz_<Agent_Instance_
Name>
LOG_FILE_CLIENT=krzsqlnet.log

18
 The agent auto-generated sqlnet.ora only contains basic configuration information of net
service.

If the environment variable TNS_ADMIN is set, the agent copies

listener.ora/tnsnames.ora/sqlnet.ora to
<Installation_Dir>/logs/<Host_Name>_rz_<Agent_Instance_Name>. By default,
LOG_DIRECTORY_CLIENT and LOG_FILE_CLIENT in sqlnet.ora will be modified to
<Installation_Dir>/logs/<Host_Name>_rz_<Agent_Instance_Name>/krzsqlnet.log in order
to ensure net service log information is generated in a file which the agent is able to control
when the agent checks net service status. If you do not want the agent do this change,
change KRZ_REDIRECT_TNS configuration item to FALSE.

Example of net service log information in net service log file:

Fatal NI connect error 12514, connecting to:

(DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=orcl)
(CID=(PROGRAM=krzstart)(HOST=tivagent)(USER=root)))
(ADDRESS=(PROTOCOL=TCP)(HOST=rac1.cn.ibm.com)(PORT=1521)))

5.3 About RDB_Instance_Info/ASM_Instance attribute groups

The RDB_Instance_Info and ASM_Instance attribute groups were originally designed to
only show information of active instances.

For agent version 6.31.00.01 and above versions, The RDB_Instance_Info and
ASM_Instance attribute groups were enhanced to show both active and inactive instances.
For inactive instance, the status is shown as OFFLINE. But there is a prerequisite to show
inactive instance, the instance need to be active when the agent instance is started,
otherwise, the agent is unable to save its information into cache and cannot show its
information.

Enhancement: INTERNAL 145418

Abstract: support instance OFFLINE status for RDB_Instance_Info and
ASM_Instance_Info
Additional Information: With this enhancement, RDB_Instance_Info and
ASM_Instance_Info attribute groups can show instance information with 'OFFLINE' Status.

19
6 Oracle Monitoring Concept and Situations

6.1 Monitor Oracle Availability

6.1.1 Monitor Oracle Instance Status
The Oracle database is made of a set of operating system files containing data entered by
users or applications and structural information about the database itself called metadata.
Information is stored persistently in these files.

To enable users and applications to view or update data in the database, Oracle must start
a set of processes, called background processes, and must allocate some memory to be
used during database operation. The background processes and memory allocated by
Oracle together make up an instance. An instance must be started to read and write
information to the database. However, having a database is not necessary to run an
instance.

Example Instance Overview workspace about Oracle Instance

Status

Figure 4: Navigate to Instance Overview workspace

20
Figure 5: Instance Status table in the Instance Overview workspace:

Figure 6: Navigate to product predefined situations from Instance Node

21
 Figure 7: Product predefined situations for Oracle Inactive Instance

6.1.2 Monitor Oracle Tablespace

Tablespace is the logical storage unit that groups related logical structure together.

Example Create a sample tablespace in the sqlplus:

SQL> CREATE TABLESPACE tbs_3 DATAFILE 'E:\tbs_3.dbf'
size 10M AUTOEXTEND ON MAXSIZE 80M;

Tablespace created.

You can navigate to the Tablespace Usage workspace from the Tablespace node.

22
 Figure 8: Navigate to Tablespace Usage workspace

Then you can get the usage percentage for all tablespaces like TBS_3 which is created in
the example above.

Figure 9: Non-Temporary Tablespace Usage table in the Tablespace Usage workspace

If the option of AUTOEXTEND for the Tablespace is OFF, Maximum File Size is equal
to the Allocation Size, then %Free_To_Maximum is equal to the %Free_To_Allocated
also.

% Free To Maximum = (Maximum file size-Used size)/Maximum file size*100%

% Free To Allocated = (Allocation size-Used size)/Allocation size*100%

In the sample Tablespace TBS_3, the Maximum file size is 80M, the Allocation size is10M
and the Used size is 1M. So its %Free_To_Maximum is (80-1)/80*100%=98.75% which is
greater than the %Free_To_Allocated of (10-1)/10*100%=90%.

6.1.3 Monitor Oracle Automatic Storage Management (ASM)

23
 ASM is a vertical integration of both the file system and the volume manager built
specifically for Oracle database files. It extends the concept of stripe and mirrors
everything to optimize performance, while removing the need for manual I/O tuning.

Oracle storage is added and removed from ASM disk groups in units of ASM disks.

ASM disks can be monitored in the ASM Disk Capacity workspace linked from the
Automatic Storage Management node.

Figure 10: navigate to the ASM Disk Capacity workspace

Figure 11: ASM Disk Capacity table in the ASM Disk Capacity workspace

6.1.4 Monitor Oracle Real Application Cluster (RAC)

RAC is an option that allows multiple concurrent instances to share a single physical
database.

There are several workspaces for the RAC monitoring in the Node of Real Application
Cluster.

24
 Figure 12: Navigate to RAC related workspaces from Real Application Cluster node

Figure 13: GCS CR Latency Report for RAC monitoring

6.1.5 Monitor Oracle Database

Oracle is a relational database. In a relational database, all data is stored in two-

dimensional tables that are composed of rows and columns. The Oracle Database enables
you to store data, update it, and efficiently retrieve it.

Oracle provides software to create and manage the Oracle database. The database
consists of physical and logical structures in which system, user, and control information is
stored. The software that manages the database is called the Oracle database server.
Collectively, the software that runs Oracle and the physical database are called the Oracle
database system.

25
 Figure 14: Navigate from Database node

Figure 15: Global Resources Consumption table in the Resource Limitation workspace

6.1.6 Monitor Oracle Alert Log

26
 Each Oracle instance you run produces an alert log, which is a sequential log of text
messages pertaining to overall database operations including Oracle startup and
shutdown, major database events such as log archiving and tablespace definition, and
certain categories of errors. Submitting a copy of your alert log data is required in many
problem determination situations. Refer to the Oracle Database Administrator's Guide for
general information on the alert log.

Figure 16: Alert Log workspace and its tables

6.2 Monitor Oracle Performance

6.2.1 Buffer Cache Hit Ratio
The buffer cache hit ratio calculates how often a requested block has been found in the
buffer cache without requiring disk access. This ratio is computed using data selected from
the dynamic performance view V$SYSSTAT. The buffer cache hit ratio can be used to
verify the physical I/O as predicted by V$DB_CACHE_ADVICE.

Example below has been simplified by using values selected directly from the V$SYSSTAT
table, rather than over an interval. It is best to calculate the delta of these statistics over an
interval while your application is running, then use them to determine the hit ratio.

27
SQL> SELECT NAME,VALUE FROM V$SYSSTAT

2 WHERE NAME IN ('db block gets from cache', 'consistent gets from cache', 'physical
reads cache');

NAME VALUE

--------------------------------------------- ----------

db block gets from cache 81182

consistent gets from cache 419831

physical reads cache 13286

Using the values in the output of the query, calculate the hit ratio for the buffer cache with
the following formula:

1 - (('physical reads cache') / ('consistent gets from cache' + 'db block gets from cache')

28
 Figure 17: Buffer Cache performance table from Cache node

6.2.2 System Global Area (SGA)

SGA is a group of shared memory structures that contain data and control information for
one Oracle database instance. If multiple users are concurrently connected to the same
instance, then the data in the instance's SGA is shared among the users. Consequently,
the SGA is sometimes referred to as the shared global area.

29
Figure 18: Navigate to SGA workspace from Memory node

Figure 19: SGA Usage bar chart in the SGA Detail workspace

There are two different concepts about SGA free memory with the Oracle Database
Extended agent. Here is a clarification of the two attributes, this is normal behavior.

In Figure 20, the "Free Total SGA (MB)" attribute in the "SGA Allocation" attribute group
uses the v$sgastat Oracle performance view. It means the free memory of allocated
memory in the buffer cache, shared pool, etc.

30
Figure 20: Free Total SGA (MB) in the SGA Allocation workspace

In Figure 21, the Free SGA Memory Available (MB)" attribute in the "SGA Usage" attribute
group uses the v$sgainfo Oracle performance view. It means the free SGA memory which
is still not allocated to buffer cache, shared pool, etc.

31
 Figure 21: Free SGA Memory Available (MB) in the SGA Usage workspace

6.2.3 Top SQL

The Top SQL section describes the top SQL statements of the sampled session activity.
Use this information to identify high-load SQL statements that may be the cause of the
transient performance problem.

32
 Figure 22: Top SQL workspace

Figure 23: Top SQL By Elapsed Time workspace

6.2.4 Lock Contention and Dead Locks

When two Oracle users are attempting to change the same row, there is a lock contention
with the lock type of TX. A deadlock can occur when two or more Oracle users are waiting
for data locked by each other. Deadlocks prevent some transactions from continuing to
work. Figure below is a hypothetical illustration of two transactions in a deadlock.

33
Figure 24: Two Transaction in a deadlock

In Figure 24, no problem exists at time point A, as each transaction has a row lock on the
row it attempts to update. Each transaction proceeds without being terminated. However,
each tries next to update the row currently held by the other transaction. Therefore, a
deadlock results at time point B, because neither transaction can obtain the resource it
needs to proceed or terminate. It is a deadlock because no matter how long each
transaction waits, the conflicting locks are held.

Figure 25: Navigate to Lock Contention from Contention node

34
Figure 26: Lock Conflict table with the deadlock record in Lock Contention workspace

35
7 Problem Determination
Troubleshooting: Slow Database Monitoring

http://www-01.ibm.com/support/docview.wss?uid=swg21508178

Question:

How do you extended the timeout period of the oracle agent?

Cause:

The oracle database is slow on the system and the agent often times out during
monitoring.

Answer:

Update the CDP_COLLECTION_TIMEOUT environment variable. The default is 60

seconds and the agent needs more time to verify database connection.

Troubleshooting: TOP SQL historical collection

http://www-01.ibm.com/support/docview.wss?uid=swg21511760

Question:

How to configure the TOP SQL attribute group historical collection for the RZ agent

Answer:

To configure the historical collection for the TOP SQL attribute group, the user needs to
click on the third tab ("Filters") and fill the requested filters: "begin hour" ,"end hour", "order
by" and "row order". This setting is required to get the historical collection enabled for this
attribute group.

Historical collection settings:

36
Figure 27: History Collection Configuration for TOPSQL

Troubleshooting: DB connection fails during Oracle Extended Agent configuration

Problem (Abstract):

Attempts to configure result in error "ORA-01722: invalid number" despite IF0001 is

already applied for the RZ agent v6.3.1.

Symptom:

Error message "ORA-01722: invalid number"

Resolving the problem:

If the IF0001 has been applied, you can safely ignore the error message for test
connection, as follows.

1. Set "NLS_NUMERIC_CHARACTERS" in <AGENT_INSTALL_HOME>/config/rz.ini

NLS_NUMERIC_CHARACTERS=.

Reconfigure agent and ignore the above error message.

Try to start this agent, if agent is not started, try next step (2).

2. Set "NLS_NUMERIC_CHARACTERS" in <AGENT_INSTALL_HOME>/config/rz.ini

NLS_NUMERIC_CHARACTERS=.,

Reconfigure agent and ignore the above error message.

37
Try to start this agent, if agent is not started, please collect the debugging data for the IBM
support investigation as explained at next step (3).

3. Set "KBB_RAS1" in <AGENT_INSTALL_HOME>/config/rz.ini

KBB_RAS1=ERROR (UNIT:krz all)

Start agent and send to IBM Support the output of "digup -a" command.

Also, please use "sqlplus" to connect to database directly: issue following SQL and send
the output to the IBM Support

select * from nls_database_parameters;

select * from nls_instance_parameters;

select * from nls_session_parameters;

select userenv('language') from dual;

select * from v$nls_valid_values;

Then quit "sqlplus" and setup the environment as follows:

NLS_LANG=AMERICAN_AMERICA.AL32UTF8 && export NLS_LANG

NLS_NUMERIC_CHARACTERS=. && export NLS_NUMERIC_CHARACTERS

Use again "sqlplus" to connect to oracle db and issue:

select userenv('language') from dual

select * from nls_session_parameters

Then quit "sqlplus" and setup system environment again as follows:

NLS_NUMERIC_CHARACTERS='.,' && export NLS_NUMERIC_CHARACTERS

Finally, issue the last SQL command:

Select * from nls_session_parameters

Please send the outputs from all the above SQLs statements to the IBM Support for more
investigation.

Troubleshooting: Monitoring Oracle alert logs with remote RZ agent

Question:

How to remotely monitor the Oracle alert log on AIX with an RZ agent installed on
Windows?

38
Answer:

For monitoring remote alert logs on AIX, UNIX or Linux systems, the file system containing
the logs needs to be locally mounted on the RZ agent machine.

To mount AIX file systems in Windows, you need to install Samba on AIX.

Troubleshooting: Oracle Extended Agent (rz) fails loading Oracle OCI library

Problem (Abstract):

The Oracle Extended Agent is not running properly and produces the error: Loading Oracle
OCI library failed!

Symptom:

The Oracle Extended Agent starts and connects to TEMS but the Agent does not display
Oracle data.

Cause:

This error occurs because the Oracle Extended Agent(rz) can not use the Oracle Client
Interface(OCI) to connect to the Oracle instance.

Diagnosing the problem:

The Oracle Extended Agent on unix/linux starts the krzagent and then launches the
krzclient process. The krzclient process fails when it tries to connect to the Oracle
database.

Example instance_rz_server_krzclient_timestamp.log reflects the following

errors:

Fail to dynamically load library libclntsh.so.10.1, return code 8

Try to load another possible Oracle OCI library

Fail to dynamically load library libclntsh.so.11.1, return code 2

Loading Oracle OCI library failed!

Resolving the problem:

Reconfigure the agent and correct Oracle home or Oracle Instant Client Library.
Reconfigure the agent and restart.

39
 APPENDIX

8 Appendix A. How to use Oracle Instant

Client for Tivoli Extended Oracle Agent
Step 1) open Oracle Instant Client download page from Oracle Site below:

http://www.oracle.com/technetwork/database/features/instant-client/index-
097480.html

Step 2) click one suitable platform from list, especially for the 32-bit and 64-bit
platforms.

Figure 28: Selection of Oracle Instant Client Downloads

Step 3) select Accept License Agreement. For example:

vi
Instant Client Downloads for Solaris Operating System(SPARC 64-bit)

Step 4) download Instant Client Package - Basic: All files requires to run OCI, OCCI,
and JDBC-OCI applications.

Example basic-11.2.0.2.0-solaris-sparc64.zip (64,102,971 bytes) (cksum -

4103480643)

vii
 REFERENCES

[Reference List] (Use bibliography format. Style automatically creates a hanging indent when text
runs over to the next line.)
Best Practices for Oracle Monitoring with Oracle Extended Agent.

Copyright IBM Corporation 2009
IBM United States of America
Produced in the United States of America
US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM Corp.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM
representative for information on the products and services currently available in your area. Any reference to an IBM
product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used.
Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be
used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program,
or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of
this document does not grant you any license to these patents. You can send license inquiries, in writing, to:

IBM Director of Licensing

IBM Corporation
North Castle Drive
Armonk, NY 10504-1785
U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are
inconsistent with local law:
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PAPER AS IS WITHOUT WARRANTY OF
ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow
disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes may be made periodically to the
information herein; these changes may be incorporated in subsequent versions of the paper. IBM may make improvements
and/or changes in the product(s) and/or the program(s) described in this paper at any time without notice.

Any references in this document to non-IBM Web sites are provided for convenience only and do not in any manner serve
as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product
and use of those Web sites is at your own risk.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of
this document does not give you any license to these patents. You can send license inquiries, in writing, to:

IBM Director of Licensing

IBM Corporation
4205 South Miami Boulevard
Research Triangle Park, NC 27709 U.S.A.

All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent
goals and objectives only.

This information is for planning purposes only. The information herein is subject to change before the products described
become available.

If you are viewing this information softcopy, the photographs and color illustrations may not appear.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp.,
registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other
companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at
www.ibm.com/legal/copytrade.shtml.

==> The listed trademarks of the following companies require marking and attribution only if you mention any of
them in your paper:

9
Best Practices for Oracle Monitoring with Oracle Extended Agent.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe
Systems Incorporated in the United States, and/or other countries.

IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now
part of the Office of Government Commerce

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep,
Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States
and other countries.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States,
other countries, or both.

ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is
registered in the U.S. Patent and Trademark Office

UNIX is a registered trademark of The Open Group in the United States and other countries.

Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both
and is used under license therefrom.

Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other
countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

10