APM - 9.5 - Java Agent Implementation Guide

Java Agent Implementation Guide
Release 9.5
CA Application Performance
Management

This Documentation, which includes embedded help systems and electronically distributed materials, (hereinafter referred to
as the Documentation) is for your informational purposes only and is subject to change or withdrawal by CA at any time.
This Documentation may not be copied, transferred, reproduced, disclosed, modified or duplicated, in whole or in part, without
the prior written consent of CA. This Documentation is confidential and proprietary information of CA and may not be disclosed
by you or used for any purpose other than as may be permitted in (i) a separate agreement between you and CA governing
your use of the CA software to which the Documentation relates; or (ii) a separate confidentiality agreement between you and
CA.
Notwithstanding the foregoing, if you are a licensed user of the software product(s) addressed in the Documentation, you may
print or otherwise make available a reasonable number of copies of the Documentation for internal use by you and your
employees in connection with that software, provided that all CA copyright notices and legends are affixed to each reproduced
copy.
The right to print or otherwise make available copies of the Documentation is limited to the period during which the applicable
license for such software remains in full force and effect. Should the license terminate for any reason, it is your responsibility to
certify in writing to CA that all copies and partial copies of the Documentation have been returned to CA or destroyed.
TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENTATION AS IS WITHOUT WARRANTY OF ANY
KIND, INCLUDING WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE, OR NONINFRINGEMENT. IN NO EVENT WILL CA BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY LOSS OR DAMAGE,
DIRECT OR INDIRECT, FROM THE USE OF THIS DOCUMENTATION, INCLUDING WITHOUT LIMITATION, LOST PROFITS, LOST
INVESTMENT, BUSINESS INTERRUPTION, GOODWILL, OR LOST DATA, EVEN IF CA IS EXPRESSLY ADVISED IN ADVANCE OF THE
POSSIBILITY OF SUCH LOSS OR DAMAGE.
The use of any software product referenced in the Documentation is governed by the applicable license agreement and such
license agreement is not modified in any way by the terms of this notice.
The manufacturer of this Documentation is CA.
Provided with Restricted Rights. Use, duplication or disclosure by the United States Government is subject to the restrictions
set forth in FAR Sections 12.212, 52.227-14, and 52.227-19(c)(1) - (2) and DFARS Section 252.227-7014(b)(3), as applicable, or
their successors.
Copyright 2013 CA. All rights reserved. All trademarks, trade names, service marks, and logos referenced herein belong to
their respective companies.

CA Technologies Product References
This document references the following CA Technologies products and features:
CA Application Performance Management (CA APM)
CA Application Performance Management ChangeDetector (CA APM
ChangeDetector)
CA Application Performance Management ErrorDetector (CA APM ErrorDetector)
CA Application Performance Management for CA Database Performance (CA APM
for CA Database Performance)
CA Application Performance Management for CA SiteMinder (CA APM for CA
SiteMinder)
CA Application Performance Management for CA SiteMinder Application Server
Agents (CA APM for CA SiteMinder ASA)
CA Application Performance Management for IBM CICS Transaction Gateway (CA
APM for IBM CICS Transaction Gateway)
CA Application Performance Management for IBM WebSphere Application Server
(CA APM for IBM WebSphere Application Server)
CA Application Performance Management for IBM WebSphere Distributed
Environments (CA APM for IBM WebSphere Distributed Environments)
CA Application Performance Management for IBM WebSphere MQ (CA APM for
IBM WebSphere MQ)
CA Application Performance Management for IBM WebSphere Portal (CA APM for
IBM WebSphere Portal)
CA Application Performance Management for IBM WebSphere Process Server (CA
APM for IBM WebSphere Process Server)
CA Application Performance Management for IBM z/OS (CA APM for IBM z/OS)
CA Application Performance Management for Microsoft SharePoint (CA APM for
Microsoft SharePoint)
CA Application Performance Management for Oracle Databases (CA APM for Oracle
Databases)
CA Application Performance Management for Oracle Service Bus (CA APM for
Oracle Service Bus)
CA Application Performance Management for Oracle WebLogic Portal (CA APM for
Oracle WebLogic Portal)
CA Application Performance Management for Oracle WebLogic Server (CA APM for
Oracle WebLogic Server)
CA Application Performance Management for SOA (CA APM for SOA)

CA Application Performance Management for TIBCO BusinessWorks (CA APM for
TIBCO BusinessWorks)
CA Application Performance Management for TIBCO Enterprise Message Service
(CA APM for TIBCO Enterprise Message Service)
CA Application Performance Management for Web Servers (CA APM for Web
Servers)
CA Application Performance Management for webMethods Broker (CA APM for
webMethods Broker)
CA Application Performance Management for webMethods Integration Server (CA
APM for webMethods Integration Server)
CA Application Performance Management Integration for CA CMDB (CA APM
Integration for CA CMDB)
CA Application Performance Management Integration for CA NSM (CA APM
Integration for CA NSM)
CA Application Performance Management LeakHunter (CA APM LeakHunter)
CA Application Performance Management Transaction Generator (CA APM TG)
CA Cross-Enterprise Application Performance Management
CA Customer Experience Manager (CA CEM)
CA Embedded Entitlements Manager (CA EEM)
CA eHealth Performance Manager (CA eHealth)
CA Insight Database Performance Monitor for DB2 for z/OS
CA Introscope
CA SiteMinder
CA Spectrum Infrastructure Manager (CA Spectrum)
CA SYSVIEW Performance Management (CA SYSVIEW)

Contact CA Technologies
Contact CA Support
For your convenience, CA Technologies provides one site where you can access the
information that you need for your Home Office, Small Business, and Enterprise CA
Technologies products. At http://ca.com/support, you can access the following
resources:
Online and telephone contact information for technical assistance and customer
services
Information about user communities and forums
Product and documentation downloads
CA Support policies and guidelines
Other helpful resources appropriate for your product
Providing Feedback About Product Documentation
If you have comments or questions about CA Technologies product documentation, you
can send a message to techpubs@ca.com.
To provide feedback about CA Technologies product documentation, complete our
short customer survey which is available on the CA Support website at
http://ca.com/docs.

Contents 7

Contents

Chapter 1: Introduction to the Java Agent 19
About Introscope and Java Agents ............................................................................................................................. 19
Planning a Java Agent Deployment ............................................................................................................................ 20
Install and Evaluate the Default Functionality .................................................................................................... 20
Determine Configuration Requirements ............................................................................................................. 21
Define a Baseline Agent Profile with Appropriate Configuration Properties ...................................................... 21
Evaluate Agent Performance Overhead .............................................................................................................. 22
Validate and Deploy the Agent Configuration .................................................................................................... 22
Deploying the Java Agent ........................................................................................................................................... 22
Chapter 2: Installing and Configuring the Java Agent 23
Before You Install the Agent ...................................................................................................................................... 23
Select the method for installing the Java agent ......................................................................................................... 24
Install the Java Agent Interactively ..................................................................................................................... 24
Install the Java Agent Silently .............................................................................................................................. 26
Install Manually Using Installation Archives........................................................................................................ 30
About the Java Agent Directory Structure ................................................................................................................. 30
Configuring Detection of Synthetic Transactions ....................................................................................................... 32
Using the TagScript Utility ................................................................................................................................... 34
Java 7 Autoprobe ....................................................................................................................................................... 35
How to instrument applications ................................................................................................................................. 35
Configuring the Application Server to Start the Java Agent ....................................................................................... 36
Configure Apache Tomcat to use the Java agent ................................................................................................ 36
Configure JBoss to Use the Java Agent ............................................................................................................... 37
Configure Oracle WebLogic to Use the Java Agent ............................................................................................. 40
Configure WebLogic with JRockit JVM to Use the Java Agent ............................................................................ 45
Configure WebLogic with JRockit JVM to View Socket Metrics .......................................................................... 45
Configure IBM WebSphere to Use the Java Agent .............................................................................................. 46
Configure Oracle Application Server to use the Java agent ................................................................................ 57
Configure GlassFish to Use the Java Agent ......................................................................................................... 57
Configure SAP Netweaver to Use the Java Agent ............................................................................................... 58
Configuring the connection to the Enterprise Manager ............................................................................................ 59
Connect to the Enterprise Manager Using a Direct Socket Connection ............................................................. 59
Connect to the Enterprise Manager with HTTP tunneling .................................................................................. 60
Configure a proxy server for HTTP tunneling ...................................................................................................... 61
Connect to the Enterprise Manager with HTTPS tunneling ................................................................................ 61

8 Java Agent Implementation Guide

Connect to the Enterprise Manager over SSL ..................................................................................................... 62
Configure Agent Load Balancing ......................................................................................................................... 63
Upgrade Multiple Agent Types................................................................................................................................... 63
Uninstalling the Java agent ........................................................................................................................................ 64
Uninstalling the Java agent from z/OS ................................................................................................................ 65
Chapter 3: Configuring Agent Properties 67
How to modify communication with Enterprise Manager ......................................................................................... 67
How to configure backup Enterprise Managers and failover properties ................................................................... 67
How to enable and use additional GC metrics ........................................................................................................... 69
How to enable and configure thread dumps ............................................................................................................. 69
How to Configure an Agent to Collect Distribution Statistic Metrics ......................................................................... 72
Examples of Distribution Statistics Metrics ......................................................................................................... 73
Chapter 4: AutoProbe and ProbeBuilding Options 75
AutoProbe and ProbeBuilding Overview.................................................................................................................... 75
Unsupported Instrumentation Method .............................................................................................................. 76
Configuring ProbeBuilding ......................................................................................................................................... 76
Full or typical tracing options .............................................................................................................................. 76
Dynamic ProbeBuilding ....................................................................................................................................... 77
Configure Dynamic ProbeBuilding ...................................................................................................................... 79
Dynamic Instrumentation Impacts Performance for IBM JDK ............................................................................ 80
Dynamic ProbeBuilding Versus Dynamic Instrumentation ................................................................................. 80
ProbeBuilding Class Hierarchies .......................................................................................................................... 81
Removing line numbers in bytecode ................................................................................................................... 84
Chapter 5: ProbeBuilder Directives 85
ProbeBuilder Directives Overview ............................................................................................................................. 85
Components traced by the default PBDs ............................................................................................................ 86
Default PBD Files ................................................................................................................................................. 87
Default PBD Files From Previous Releases .......................................................................................................... 89
Default PBL files .................................................................................................................................................. 90
Default Tracer Groups and Toggles Files ............................................................................................................. 90
Turning tracer groups on or off ........................................................................................................................... 99
Adding classes to a tracer group ....................................................................................................................... 100
EJB Naming ........................................................................................................................................................ 102
Using the IntroscopeAgent.profile, PBLs, and PBDs together .................................................................................. 103
Applying ProbeBuilder Directives ............................................................................................................................. 103
Using JVM AutoProbe ....................................................................................................................................... 103
Using the ProbeBuilder Wizard or command-line ProbeBuilder ...................................................................... 104

Contents 9

Instrumenting with new and changed PBDs ..................................................................................................... 104
Creating custom tracers ........................................................................................................................................... 106
Using a custom BlamePointTracer tracer for common metrics ........................................................................ 106
Directive names and arguments used in tracer syntax ..................................................................................... 107
Commonly used tracer names and examples ................................................................................................... 109
Advanced single-metric tracers......................................................................................................................... 111
Skip directives ................................................................................................................................................... 114
Counting object instances ................................................................................................................................. 114
Turning on InstrumentPoint directives ............................................................................................................. 115
Combining custom tracers ................................................................................................................................ 115
Instrumenting and Inheritance ......................................................................................................................... 115
Java Annotations ............................................................................................................................................... 116
Using Blame Tracers to mark blame points.............................................................................................................. 117
Blame Tracers .................................................................................................................................................... 117
High agent CPU overhead from deep nested frontend transactions ................................................................ 118
Custom FrontendMarker Directive ................................................................................................................... 118
Blame Tracers in standard PBDs ....................................................................................................................... 119
Boundary Blame and Oracle Backends ............................................................................................................. 119
Chapter 6: Java Agent Naming 121
Understanding the Java Agent name ....................................................................................................................... 121
How the Agent Determines its Name ............................................................................................................... 122
How Introscope resolves agent naming conflicts ............................................................................................. 123
Agent naming considerations for clustered applications ......................................................................................... 124
Specifying an agent name using a Java system property ......................................................................................... 124
Specifying an agent name using a system property key .......................................................................................... 125
Obtaining an agent name from the application server ............................................................................................ 125
Application servers that support agent naming ................................................................................................ 125
Automatic agent naming .......................................................................................................................................... 126
Automatic agent naming and renamed agents ................................................................................................. 127
Advanced automatic agent naming options ..................................................................................................... 127
Enabling cloned agent naming in clustered environments ...................................................................................... 129
Cloned agent naming scenario .......................................................................................................................... 129
Configuring unique names for application instances ........................................................................................ 129
Application triage map and the agent name ............................................................................................................ 130
Chapter 7: Java Agent Monitoring and Logging 131
Configuring agent connection metrics ..................................................................................................................... 131
Socket metrics .......................................................................................................................................................... 132
Restricting socket and SSL metric collection ..................................................................................................... 132
Fine-tuning socket and SSL metric collection .................................................................................................... 133


SSL, NIO, and Socket Tracing in the Application Triage Map ............................................................................ 133
Change the Component Name in the Application Triage Map ......................................................................... 134
Turning off socket and SSL metric collection .................................................................................................... 135
Backwards Compatibility ................................................................................................................................... 135
Configuring logging options ..................................................................................................................................... 136
Running the agent in verbose mode ................................................................................................................. 136
Redirecting agent output to a file ..................................................................................................................... 137
Changing the name or location of the agent logfile .......................................................................................... 138
Agent log files and automatic agent naming .................................................................................................... 138
Rolling up logs by date or size ........................................................................................................................... 139
Managing ProbeBuilder Logs ................................................................................................................................... 140
Command-line ProbeBuilder and ProbeBuilder Wizard log name and location ............................................... 140
AutoProbe log name and location .................................................................................................................... 140
Chapter 8: Configuring LeakHunter and ErrorDetector 141
LeakHunter ............................................................................................................................................................... 141
How LeakHunter works ..................................................................................................................................... 142
What LeakHunter tracks in Java ........................................................................................................................ 142
What LeakHunter does not track ...................................................................................................................... 143
System and version requirements .................................................................................................................... 143
Enabling and disabling LeakHunter .......................................................................................................................... 144
Configuring LeakHunter properties .......................................................................................................................... 144
Ignoring collections that cause poor performance ........................................................................................... 146
Running LeakHunter ................................................................................................................................................. 147
Identifying potential leaks with collection IDs ......................................................................................................... 147
LeakHunter log file ................................................................................................................................................... 148
Potential leak first identified log entry ............................................................................................................. 148
Identified potential leak stops leaking log entry ............................................................................................... 149
Identified potential leak is leaking again log entry ........................................................................................... 150
LeakHunter timeout log entry ........................................................................................................................... 150
Using LeakHunter ..................................................................................................................................................... 150
ErrorDetector ........................................................................................................................................................... 151
Types of errors .................................................................................................................................................. 151
How ErrorDetector works ................................................................................................................................. 152
Enabling ErrorDetector in the Java Agent ................................................................................................................ 152
Configuring ErrorDetector options ........................................................................................................................... 153
Advanced error data capture ................................................................................................................................... 154
Defining new error types.......................................................................................................................................... 154
ExceptionErrorReporter .................................................................................................................................... 154
MethodCalledErrorReporter ............................................................................................................................. 155
ThisErrorReporter ............................................................................................................................................. 155

Contents 11

HTTPErrorCodeReporter ................................................................................................................................... 156
Using error tracer directives with caution ........................................................................................................ 156
Using ErrorDetector ................................................................................................................................................. 156
Chapter 9: Configuring Boundary Blame 157
Understanding Boundary Blame .............................................................................................................................. 157
Using URL groups ..................................................................................................................................................... 157
Defining keys for URL groups ............................................................................................................................ 158
Defining membership of each URL group ......................................................................................................... 158
Define the name for a URL group ..................................................................................................................... 159
Advanced naming techniques for URL groups (optional) ................................................................................. 159
Running the URLGrouper .................................................................................................................................. 163
Using Blame tracers ................................................................................................................................................. 163
Chapter 10: Configuring Transaction Trace Options 165
New Mode of Transaction Tracing ........................................................................................................................... 165
Configuring the Agent to Use Legacy Mode Transaction Tracing ..................................................................... 166
Controlling automatic Transaction Tracing behavior ............................................................................................... 168
Transaction Trace component clamp ................................................................................................................ 168
Transaction trace sampling ............................................................................................................................... 169
Agent Heap Sizing ............................................................................................................................................. 169
Cross-Process Transaction Tracing ........................................................................................................................... 170
Extending transaction trace data collection ............................................................................................................. 170
About User ID data ............................................................................................................................................ 170
About servlet request data ............................................................................................................................... 171
Configuring Agent to collect additional transaction trace data ........................................................................ 171
Configuring Component Stall Reporting .................................................................................................................. 173
Downstream Subscriber Component Stalls ....................................................................................................... 173
Upstream Inherited Component Stalls ............................................................................................................. 174
Disabling the capture of stalls as Events ........................................................................................................... 174
Chapter 11: Configuring the Introscope SQL Agent 175
The SQL Agent overview .......................................................................................................................................... 175
The SQL Agent files ................................................................................................................................................... 176
SQL statement normalization ................................................................................................................................... 176
How poorly written SQL statements create metric explosions......................................................................... 176
SQL statement normalization options .............................................................................................................. 178
Turn Off Statement Metrics ..................................................................................................................................... 184
SQL metrics............................................................................................................................................................... 184


Chapter 12: Enabling JMX Reporting 187
Java Agent JMX Support ........................................................................................................................................... 187
Introscope Support for WebLogic JMX Metrics ................................................................................................ 188
Default JMX Metric Conversion Process .................................................................................................................. 188
Using Primary Key Conversion to Streamline JMX Metrics ...................................................................................... 189
Managing Metric Volume with JMX Filters .............................................................................................................. 190
JMX filters for WebLogic ................................................................................................................................... 191
Configure WebSphere and WebLogic to Use JMX Reporting ................................................................................... 191
Enable JSR-77 Data and View JMX Metrics on WAS ................................................................................................. 192
Chapter 13: Configuring Platform Monitoring 195
Platform Monitors .................................................................................................................................................... 195
Enabling platform monitors on Windows Server 2003 ............................................................................................ 196
Enable Platform Monitors on AIX ............................................................................................................................. 196
Disable Platform Monitors ....................................................................................................................................... 197
Configure Permissions to Access Platform Monitors on HP-UX ............................................................................... 197
Troubleshoot Platform Monitoring .......................................................................................................................... 197
Troubleshooting platform monitoring on Windows ......................................................................................... 198
Chapter 14: Integrating CA APM with CA LISA 199
How to Integrate CA APM with CA LISA ................................................................................................................... 199
Install CA LISA .................................................................................................................................................... 200
Configure CA LISA Tracers ................................................................................................................................. 204
Verify the CA APM and CA LISA Integration ...................................................................................................... 204
Chapter 15: Integrating CA APM Cloud Monitor with CA APM 207
How to Integrate CA APM Cloud Monitor In Your CA APM Deployment ................................................................. 208
Download and Install the CA APM Cloud Monitor Agent ................................................................................. 208
Validate the CA APM Cloud Monitor Agent Connection ................................................................................... 209
How to Limit Data .................................................................................................................................................... 210
Appendix A: Java Agent Properties 213
Configuring the IntroscopeAgent.profile location ................................................................................................... 213
Command-line property overrides ........................................................................................................................... 214
Agent failover ........................................................................................................................................................... 215
introscope.agent.enterprisemanager.connectionorder ................................................................................... 215
introscope.agent.enterprisemanager.failbackRetryIntervalInSeconds ............................................................ 216
Agent HTTP tunneling .............................................................................................................................................. 217
Agent HTTP tunneling for proxy servers ........................................................................................................... 217

Contents 13

Agent HTTPS tunneling...................................................................................................................................... 219
Agent memory overhead ......................................................................................................................................... 220
introscope.agent.reduceAgentMemoryOverhead ............................................................................................ 221
Agent metric aging ................................................................................................................................................... 221
Configuring agent metric aging ......................................................................................................................... 222
Agent metric clamp .................................................................................................................................................. 225
introscope.agent.metricClamp ......................................................................................................................... 225
Agent naming ........................................................................................................................................................... 226
introscope.agent.agentAutoNamingEnabled .................................................................................................... 227
introscope.agent.agentAutoNamingMaximumConnectionDelayInSeconds .................................................... 227
introscope.agent.agentAutoRenamingIntervalInMinutes ................................................................................ 228
introscope.agent.agentName ........................................................................................................................... 228
introscope.agent.agentNameSystemPropertyKey ............................................................................................ 229
introscope.agent.disableLogFileAutoNaming ................................................................................................... 229
introscope.agent.clonedAgent .......................................................................................................................... 230
introscope.agent.customProcessName ............................................................................................................ 230
introscope.agent.defaultProcessName ............................................................................................................. 231
introscope.agent.display.hostName.as.fqdn .................................................................................................... 231
Agent recording (business recording) ...................................................................................................................... 231
introscope.agent.bizRecording.enabled ........................................................................................................... 232
Agent thread priority ................................................................................................................................................ 232
introscope.agent.thread.all.priority .................................................................................................................. 233
Agent to Enterprise Manager connection ................................................................................................................ 233
introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT ................................................................. 233
introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT ................................................................. 234
introscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT .................................................. 234
introscope.agent.enterprisemanager.transport.tcp.local.ipaddress.DEFAULT ................................................ 235
introscope.agent.enterprisemanager.transport.tcp.local.port.DEFAULT ......................................................... 235
Application triage map ............................................................................................................................................. 235
introscope.agent.appmap.enabled ................................................................................................................... 236
introscope.agent.appmap.metrics.enabled ...................................................................................................... 236
introscope.agent.appmap.queue.size ............................................................................................................... 237
introscope.agent.appmap.queue.period .......................................................................................................... 237
introscope.agent.appmap.intermediateNodes.enabled................................................................................... 238
Application Triage Map and Catalyst Integration ..................................................................................................... 238
Configure the Ability to Send Information ........................................................................................................ 238
Configure a List of Available Networks ............................................................................................................. 239
Application triage map business transaction POST parameters .............................................................................. 240
introscope.agent.bizdef.matchPost .................................................................................................................. 240
Known limitations ............................................................................................................................................. 241
Application triage map managed socket configuration ........................................................................................... 242
introscope.agent.sockets.managed.reportToAppmap ..................................................................................... 243


introscope.agent.sockets.managed.reportClassAppEdge................................................................................. 243
introscope.agent.sockets.managed.reportMethodAppEdge ............................................................................ 244
introscope.agent.sockets.managed.reportClassBTEdge ................................................................................... 244
introscope.agent.sockets.managed.reportMethodBTEdge .............................................................................. 245
AutoProbe ................................................................................................................................................................ 245
introscope.autoprobe.directivesFile ................................................................................................................. 245
introscope.autoprobe.enable ........................................................................................................................... 246
introscope.autoprobe.logfile ............................................................................................................................ 246
Bootstrap Classes Instrumentation Manager ........................................................................................................... 247
introscope.bootstrapClassesManager.enabled ................................................................................................ 247
introscope.bootstrapClassesManager.waitAtStartup ....................................................................................... 247
CA CEM Agent Profile Properties ............................................................................................................................. 248
introscope.autoprobe.directivesFile ................................................................................................................. 248
introscope.agent.remoteagentconfiguration.allowedFiles .............................................................................. 249
introscope.agent.remoteagentconfiguration.enabled ..................................................................................... 250
introscope.agent.decorator.enabled ................................................................................................................ 250
introscope.agent.decorator.security ................................................................................................................ 251
introscope.agent.cemtracer.domainconfigfile.................................................................................................. 251
introscope.agent.cemtracer.domainconfigfile.reloadfrequencyinminutes ...................................................... 252
introscope.agent.distribution.statistics.components.pattern .......................................................................... 253
Configure the Session ID Collection .................................................................................................................. 253
ChangeDetector configuration ................................................................................................................................. 254
introscope.changeDetector.enable................................................................................................................... 254
introscope.changeDetector.agentID ................................................................................................................. 254
introscope.changeDetector.rootDir .................................................................................................................. 255
introscope.changeDetector.isengardStartupWaitTimeInSec............................................................................ 255
introscope.changeDetector.waitTimeBetweenReconnectInSec....................................................................... 255
introscope.changeDetector.profile ................................................................................................................... 256
introscope.changeDetector.profileDir .............................................................................................................. 256
introscope.changeDetector.compressEntries.enable ....................................................................................... 257
introscope.changeDetector.compressEntries.batchSize .................................................................................. 257
Cross-process tracing in WebLogic Server ............................................................................................................... 257
introscope.agent.weblogic.crossjvm ................................................................................................................. 258
Cross-process transaction trace ............................................................................................................................... 258
introscope.agent.transactiontracer.tailfilterPropagate.enable ........................................................................ 258
Dynamic instrumentation ........................................................................................................................................ 259
introscope.autoprobe.dynamicinstrument.enabled ......................................................................................... 259
autoprobe.dynamicinstrument.pollIntervalMinutes ........................................................................................ 260
introscope.autoprobe.dynamicinstrument.classFileSizeLimitInMegs .............................................................. 260
introscope.autoprobe.dynamic.limitRedefinedClassesPerBatchTo .................................................................. 260
introscope.agent.remoteagentdynamicinstrumentation.enabled ................................................................... 261
introscope.autoprobe.dynamicinstrument.pollIntervalMinutes ...................................................................... 261

Contents 15

ErrorDetector ........................................................................................................................................................... 261
introscope.agent.errorsnapshots.enable .......................................................................................................... 262
introscope.agent.errorsnapshots.throttle ........................................................................................................ 262
introscope.agent.errorsnapshots.ignore.<index> ............................................................................................. 262
Extensions ................................................................................................................................................................ 263
introscope.agent.extensions.directory ............................................................................................................. 263
introscope.agent.common.directory ................................................................................................................ 263
GC Monitor ............................................................................................................................................................... 264
introscope.agent.gcmonitor.enable.................................................................................................................. 264
Java NIO .................................................................................................................................................................... 264
Channels ............................................................................................................................................................ 265
NIODatagramTracing metrics ............................................................................................................................ 265
Restricting Java NIO metrics ............................................................................................................................. 266
JMX ........................................................................................................................................................................... 270
introscope.agent.jmx.enable ............................................................................................................................ 270
introscope.agent.jmx.ignore.attributes ............................................................................................................ 271
introscope.agent.jmx.name.filter ..................................................................................................................... 271
introscope.agent.jmx.name.jsr77.disable ......................................................................................................... 272
introscope.agent.jmx.name.primarykeys ......................................................................................................... 273
introscope.agent.jmx.excludeStringMetrics ..................................................................................................... 274
LeakHunter ............................................................................................................................................................... 274
introscope.agent.leakhunter.collectAllocationStackTraces .............................................................................. 275
introscope.agent.leakhunter.enable ................................................................................................................. 276
introscope.agent.leakhunter.leakSensitivity..................................................................................................... 276
introscope.agent.leakhunter.logfile.append .................................................................................................... 277
introscope.agent.leakhunter.logfile.location .................................................................................................... 277
introscope.agent.leakhunter.timeoutInMinutes .............................................................................................. 278
introscope.agent.leakhunter.ignore.<number> ............................................................................................... 278
Logging ..................................................................................................................................................................... 279
log4j.logger.IntroscopeAgent ............................................................................................................................ 280
log4j.appender.logfile.File................................................................................................................................. 281
log4j.logger.IntroscopeAgent.inheritance ........................................................................................................ 281
log4j.appender.pbdlog.File ............................................................................................................................... 282
log4j.appender.pbdlog ...................................................................................................................................... 282
log4j.appender.pbdlog.layout ........................................................................................................................... 283
log4j.appender.pbdlog.layout.ConversionPattern ............................................................................................ 283
log4j.additivity.IntroscopeAgent.inheritance ................................................................................................... 284
Metric count ............................................................................................................................................................. 284
introscope.ext.agent.metric.count ................................................................................................................... 285
Multiple inheritance ................................................................................................................................................. 285
introscope.autoprobe.hierarchysupport.enabled ............................................................................................ 286
introscope.autoprobe.hierarchysupport.runOnceOnly .................................................................................... 286


introscope.autoprobe.hierarchysupport.pollIntervalMinutes .......................................................................... 287
introscope.autoprobe.hierarchysupport.executionCount ................................................................................ 287
introscope.autoprobe.hierarchysupport.disableLogging .................................................................................. 288
introscope.autoprobe.hierarchysupport.disableDirectivesChange .................................................................. 288
Platform monitoring ................................................................................................................................................. 288
introscope.agent.platform.monitor.system ...................................................................................................... 289
Remote configuration .............................................................................................................................................. 289
introscope.agent.remoteagentconfiguration.enabled ..................................................................................... 289
introscope.agent.remoteagentconfiguration.allowedFiles .............................................................................. 290
Security ..................................................................................................................................................................... 290
introscope.agent.decorator.security ................................................................................................................ 290
Servlet header decorator ......................................................................................................................................... 291
introscope.agent.decorator.enabled ................................................................................................................ 291
Socket metrics .......................................................................................................................................................... 291
introscope.agent.sockets.reportRateMetrics ................................................................................................... 292
introscope.agent.io.socket.client.hosts ............................................................................................................ 292
introscope.agent.io.socket.client.ports ............................................................................................................ 293
introscope.agent.io.socket.server.ports ........................................................................................................... 293
SQL Agent ................................................................................................................................................................. 293
introscope.agent.sqlagent.normalizer.extension ............................................................................................. 294
introscope.agent.sqlagent.normalizer.regex.matchFallThrough ...................................................................... 295
introscope.agent.sqlagent.normalizer.regex.keys ............................................................................................ 296
introscope.agent.sqlagent.normalizer.regex.key1.pattern .............................................................................. 296
introscope.agent.sqlagent.normalizer.regex.key1.replaceAll .......................................................................... 297
introscope.agent.sqlagent.normalizer.regex.key1.replaceFormat ................................................................... 297
introscope.agent.sqlagent.normalizer.regex.key1.caseSensitive ..................................................................... 298
introscope.agent.sqlagent.sql.artonly .............................................................................................................. 298
introscope.agent.sqlagent.sql.rawsql ............................................................................................................... 299
introscope.agent.sqlagent.sql.turnoffmetrics .................................................................................................. 299
introscope.agent.sqlagent.sql.turnofftrace ...................................................................................................... 299
SSL communication .................................................................................................................................................. 300
introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT ................................................................. 300
introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT ................................................................. 301
introscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT .................................................. 301
introscope.agent.enterprisemanager.transport.tcp.truststore.DEFAULT ........................................................ 302
introscope.agent.enterprisemanager.transport.tcp.trustpassword.DEFAULT ................................................. 302
introscope.agent.enterprisemanager.transport.tcp.keystore.DEFAULT .......................................................... 302
introscope.agent.enterprisemanager.transport.tcp.keypassword.DEFAULT ................................................... 303
introscope.agent.enterprisemanager.transport.tcp.ciphersuites.DEFAULT ..................................................... 303
Stall metrics .............................................................................................................................................................. 303
introscope.agent.stalls.thresholdseconds ........................................................................................................ 303
introscope.agent.stalls.resolutionseconds ....................................................................................................... 304

Contents 17

Thread dumps .......................................................................................................................................................... 304
introscope.agent.threaddump.enable .............................................................................................................. 305
introscope.agent.threaddump.deadlockpoller.enable ..................................................................................... 305
introscope.agent.threaddump.deadlockpollerinterval ..................................................................................... 306
introscope.agent.threaddump.MaxStackElements .......................................................................................... 306
Transaction tracing ................................................................................................................................................... 307
introscope.agent.bizdef.turnOff.nonIdentifying.txn ......................................................................................... 307
introscope.agent.transactiontracer.parameter.httprequest.headers .............................................................. 308
introscope.agent.transactiontracer.parameter.httprequest.parameters ........................................................ 308
introscope.agent.transactiontracer.parameter.httpsession.attributes ............................................................ 309
introscope.agent.transactiontracer.userid.key ................................................................................................. 309
introscope.agent.transactiontracer.userid.method ......................................................................................... 310
introscope.agent.transactiontrace.componentCountClamp ............................................................................ 311
introscope.agent.crossprocess.compression .................................................................................................... 312
introscope.agent.crossprocess.compression.minlimit...................................................................................... 313
introscope.agent.crossprocess.correlationid.maxlimit ..................................................................................... 314
introscope.agent.transactiontracer.sampling.enabled ..................................................................................... 314
introscope.agent.transactiontracer.sampling.perinterval.count ...................................................................... 315
introscope.agent.transactiontracer.sampling.interval.seconds ....................................................................... 315
introscope.agent.transactiontrace.headFilterClamp ........................................................................................ 316
introscope.agent.ttClamp ................................................................................................................................. 317
URL grouping ............................................................................................................................................................ 317
introscope.agent.urlgroup.keys ........................................................................................................................ 318
introscope.agent.urlgroup.group.default.pathprefix........................................................................................ 318
introscope.agent.urlgroup.group.default.format ............................................................................................. 318
WebSphere PMI ....................................................................................................................................................... 319
introscope.agent.pmi.enable ............................................................................................................................ 320
introscope.agent.pmi.enable.alarmManagerModule ....................................................................................... 320
introscope.agent.pmi.enable.beanModule ...................................................................................................... 321
introscope.agent.pmi.enable.cacheModule ..................................................................................................... 321
introscope.agent.pmi.enable.connectionPoolModule ..................................................................................... 322
introscope.agent.pmi.enable.hamanagerModule ............................................................................................ 322
introscope.agent.pmi.enable.j2cModule .......................................................................................................... 323
introscope.agent.pmi.enable.jvmpiModule...................................................................................................... 323
introscope.agent.pmi.enable.jvmRuntimeModule ........................................................................................... 324
introscope.agent.pmi.enable.objectPoolModule ............................................................................................. 324
introscope.agent.pmi.enable.orbPerfModule .................................................................................................. 325
introscope.agent.pmi.enable.schedulerModule ............................................................................................... 325
introscope.agent.pmi.enable.servletSessionsModule ...................................................................................... 326
introscope.agent.pmi.enable.systemModule ................................................................................................... 326
introscope.agent.pmi.enable.threadPoolModule ............................................................................................. 327
introscope.agent.pmi.enable.transactionModule ............................................................................................ 327


introscope.agent.pmi.enable.webAppModule ................................................................................................. 328
introscope.agent.pmi.enable.webServicesModule .......................................................................................... 328
introscope.agent.pmi.enable.wlmModule........................................................................................................ 329
introscope.agent.pmi.enable.wsgwModule ..................................................................................................... 329
introscope.agent.pmi.filter.objrefModule ........................................................................................................ 330
WLDF metrics ........................................................................................................................................................... 330
introscope.agent.wldf.enable ........................................................................................................................... 330
Appendix B: Alternative Methods for Instrumentation 331
Java Agent Deployment on Other Application Servers ............................................................................................ 331
Configure Sun ONE to Use AutoProbe ..................................................................................................................... 332
Configure Oracle to Use AutoProbe ......................................................................................................................... 333
Configure WebLogic Server ...................................................................................................................................... 334
Configuring HTTP servlet tracing .............................................................................................................................. 335
Create an AutoProbe Connector File ....................................................................................................................... 335
Running the AutoProbe Connector for a JVM ................................................................................................... 336
Example: Use Xbootclasspath to Instrument WAS ........................................................................................... 338
About running ProbeBuilder manually ..................................................................................................................... 339
Configure AutoProbe for WebSphere for z/OS ........................................................................................................ 339
Appendix C: Using the PBD Generator 343
About the CA PBD Generator ................................................................................................................................... 343
Configuring the PBD Generator ................................................................................................................................ 344
Required PBD Generator parameters ............................................................................................................... 344
Using the PBD Generator ......................................................................................................................................... 344
Appendix D: Using the Network Interface Utility 347
Determine Network Interface Names ...................................................................................................................... 347
Index 349


Chapter 1: Introduction to the Java Agent

This section contains the following topics:
About Introscope and Java Agents (see page 19)
Planning a Java Agent Deployment (see page 20)
Deploying the Java Agent (see page 22)
About Introscope and Java Agents
CA Introscope is an enterprise application performance management solution that
enables you to monitor complex web applications in production environments 24x7,
detect problems before they affect your customers, and resolve these issues quickly and
collaboratively. A key part of the architecture for this solution is the low overhead
agent.
The agent is a data gathering component of Introscope that collects detailed
performance information about applications and the computing environment as
transactions are executed. The Java agent collects this information from applications
and resources running on Java Virtual Machines (JVMs) and sends the information to
the Enterprise Manager for further processing.
The Java agent inserts probes into the bytecode of the components that make up the
JVM that the application uses. Inserting the probes into the bytecode is part of the
instrumentation process that enables the monitoring of your applications.
Instrumenting an application also requires tracers that are defined in ProbeBuilder
Directive (PBD) files. The instructions or directives in PBD files identify the application
components to monitor. The tracers identify the metrics the agent should collect from
which probes as the application runs in the JVM. You can control what is monitored by
changing the PBD files to suit your environment.
When you install the Java agent, several default PBD files are deployed to enable default
monitoring of your environment. You can modify the default monitoring to achieve the
balance of visibility and performance that you require. After the application is
instrumented, the Java agent collects the data you are interested in and reports the
data to the Enterprise Manager. The Enterprise Manager then processes and stores the
data for real-time and historical reporting. You can then view and work with the
collected data using the Introscope Workstation to create alerts or take responsive
action.
Planning a Java Agent Deployment


For application management, the key activities are:
Deploy agents to monitor the performance and availability of application servers.
Test, adjust, and optimize monitoring of application components.
Customize the agent profile to control agent operations as needed.
Create application- or agent-specific metric groupings, dashboards, alerts, and
actions
Investigate, triage, and diagnose application issues.
When planning a deployment, your goal is to develop the right balance between agent
overhead and the visibility of application performance. Low overhead agents allow for
real-time monitoring of all transactions in production environments. Keeping overhead
low helps performance and availability of critical applications and server resources.
However, keeping overhead low does not provide enough information to diagnose
problems when they occur. Therefore, it is common to deploy and adjust the agent
configuration throughout the lifecycle of an application. In addition, monitoring more
components when an application is being developed or tested, then reducing the
components after an application is released to production.
Install and Evaluate the Default Functionality
The first step in deploying agents involves installing and evaluating the default agent
configuration. The default agent configuration demonstrates data collection for many
common components of the application and the computing environment. The default
agent configuration also includes several features enabled and other, less frequently
used features, disabled.
Your goal is to evaluate the depth and breadth of data collection provided by default
and become familiar with Introscope and how to monitor applications. After you are
familiar with the performance metrics the agent provides by default, you can customize
the agent to collect data, as needed.
As you evaluate the default performance, keep in mind that when the agent collects
more metrics, it consumes more system resources. If the agent collects fewer metrics,
you have less visibility into potential problems. As you refine the agent configuration,
try to strike a balance between the depth of data collection and an acceptable level of
performance. The appropriate level of instrumentation typically depends on where the
agent is deployed. For example, an agent monitoring within a test environment is
typically configured to collect large numbers of metrics. An agent on a production
server, however, is typically configured to deliver essential information.


Determine Configuration Requirements
Before you deploy the agent, determine your data collection requirements. This
information lets you tailor the data collection behaviors of the agent, and evaluate the
impact on overhead through alternative configurations of the agent.
Introscope can be used throughout the lifecycle of an application. For example, from
development through testing, load verification, staging, and into production. At each
stage in the lifecycle, the monitoring goals, environmental constraints, and service level
expectations are likely to be different. To address these differences, you configure the
agent to behave differently for the type of environment that is monitored.
Your goal is to determine the appropriate trade-off between the visibility of
performance details and resource overhead. Also consider the optimum level of visibility
at a reasonable cost for the environment being monitored.
In preproduction application environments, such as development, you typically
configure a higher level of data collection to provide deeper visibility into the
application performance. In production or high-volume transaction environments, you
typically reduce the metrics reported to control agent overhead. Depending on your
requirements, you can also configure agent properties to control specific agent
behavior. For example, track the maximum number of metrics collected and the
removal of old metrics.
For your environment, determine the appropriate level of visibility and acceptable
performance overhead, so you can configure the agent to match your requirements.
Define a Baseline Agent Profile with Appropriate Configuration Properties
After you identify the type of application environment to monitor, you can create a
"candidate" agent configuration. Most agent operations are controlled using properties
in the agent profile (IntroscopeAgent.profile). For example, the IntroscopeAgent.profile
file defines ProbeBuilder Directive and ProbeBuilder List files the agent uses. The files
listed in the agent profile then control the specific metrics that the agent gathers. The
IntroscopeAgent.profile file also provides properties that enable or disable specific
features or tune operations such as polling intervals.
Depending on your configuration and environment, you can adjust the properties in the
agent profile to evaluate the impact of each change. For example, you can start with a
default agent profile that does not have ChangeDetector enabled. Later, you can enable
ChangeDetector properties in the profile and evaluate agent performance after the
change before adding any other features.
Deploying the Java Agent


Evaluate Agent Performance Overhead
When evaluating an agent configuration, verify that the metrics collected provide
sufficient visibility into application performance and availability. In addition, verify that
the volume of metrics cannot impose an unacceptable load on the operating
environment. The agent cannot report more metrics than are necessary to identify and
localize performance and availability problems.
You can effectively evaluate agent performance by understanding the performance
characteristics of the application. For example, you can load test your application before
and after implementing default monitoring to verify impact.
To extend data collection in a controlled way, verify agent operation and overhead
before and after implementing changes. For example, only add monitoring support for
one application at a time, so you can evaluate the performance of each add-on
individually.
Validate and Deploy the Agent Configuration
After verifying that the candidate agent configuration provides the visibility required,
deploy the configuration across that environment.
You can deploy a validated configuration by installing the following configuration items
to the target environment:
IntroscopeAgent.profile file
modified or custom PBD files
Deploying the Java Agent
Deploying the agent involves the following high level steps:
1. Install the agent on the target computer.
2. Configure the application server (see page 35) to instrument your applications.
3. Configure the startup script (see page 36) or Java command for the application
server to include the agent and the location of the agent profile.
4. Configure the agent connection (see page 59) to the Enterprise Manager.
5. Restart the application server to instrument the application and start data
collection.
6. Configure the IntroscopeAgent.profile (see page 67) if you want to change any
properties set during installation or want to modify data collection or other agent
operations.


Chapter 2: Installing and Configuring the
Java Agent

Before You Install the Agent (see page 23)
Select the method for installing the Java agent (see page 24)
About the Java Agent Directory Structure (see page 30)
Configuring Detection of Synthetic Transactions (see page 32)
Java 7 Autoprobe (see page 35)
How to instrument applications (see page 35)
Configuring the Application Server to Start the Java Agent (see page 36)
Configuring the connection to the Enterprise Manager (see page 59)
Upgrade Multiple Agent Types (see page 63)
Uninstalling the Java agent (see page 64)
Before You Install the Agent
Before you install the agent, do the following steps:
1. Review the Java agent deployment process.
2. Verify that you have a supported version of the application server.
Note: The server where you plan to install the agent must have a supported version
of the JVM available locally. For application server and JVM requirements, see the
Compatibility Guide.
3. Verify that you have a supported version of the JVM. If you are unable to use a
supported version, use a previous version:
Use the 8.x Java agent for Java 1.4.x.
Use the 7.2 Java agent for Java 1.3.x.
4. Verify that the Introscope Enterprise Manager and Workstation components are
installed. Note the host name and port number for the Enterprise Manager
connection with the agent.
You can use ping or telnet to verify connectivity between the agent and the
Enterprise Manager.
Note: For information about installing the Enterprise Manager, Workstation, and
WebView components, see the CA APM Installation and Upgrade Guide.
Select the method for installing the Java agent


More information:
Planning a Java Agent Deployment (see page 20)
Deploying the Java Agent (see page 22)

You can install the Java agent in one of the following ways:
interactively using a graphical user interface (GUI) or text-based console installer
silently without interactive prompts using an edited response file
manually by extracting and configuring files for individual application servers
In most cases, you use the interactive installer when you want to install files locally on a
computer. If you install interactively, the prompts displayed are the same for the
graphical or text-based interface. However, the option to choose the GUI or text-based
installer depends on the operating system of the computer where you run the installer.
For example, most UNIX environments support the GUI or text-based console installer,
but start the console installer by default.
If you want to install files remotely on a computer or deploy a preconfigured set of
installation instructions, you can run the installer silently using an edited response file.
If you do not want to run the installer interactively or silently, you can manually install
and configure the agent using an installation archive. Manual installation enables you to
extract specific sets of agent files from application server- and operating system-specific
archives, and then manually configure deployment options. CA Technologies provides
these archives to expedite deployment of the Java agent onto multiple systems that
share the same type of application server and operating system.
For more information about installing interactively, silently, or manually see the
appropriate section.
Install the Java Agent Interactively
You can install the Java agent interactively by selecting the Java agent installer
appropriate for your operating system, starting the installer, and responding to the
prompts. If you use the GUI installer, you can make selections using drop-down menus
and selecting check boxes. In the text-based console installer, you make selections by
entering text.


Follow these steps:
1. Select the installation archive appropriate for your operating system. For example:
IntroscopeAgentInstaller<version>windows.zip to install on Microsoft
Windows.
IntroscopeAgentInstaller<version>unix.tar to install on supported UNIX or Linux
operating systems.
IntroscopeAgentInstaller<version>zOS.tar to install on IBM z/OS.
IntroscopeAgentInstaller<version>os400.zip to install on IBM OS/400 (IBM i
operating system).
2. Extract the installation archive files using a command appropriate to your operating
system. For example, on UNIX or Linux:
tar xvf IntroscopeAgentInstaller<version>unix.tar
3. Follow the prompts to start the installation.
4. Specify the location of the application server root directory.
If you do not want to specify an application server root directory, then accept the
default response (C:\ on Windows, / on UNIX).
5. Select the application server type from the list of valid application servers.
The application server that you select determines the additional monitoring options
available.
6. Specify the root installation directory for the Java agent. In most cases, use the
application server root directory.
Within the root installation directory, the installer creates the wily directory that
is the <Agent_Home> directory.
7. Specify whether you want to create an agent profile or use an existing agent profile.
If you create an agent profile, you are prompted to select:
Typical or Full instrumentation
Agent name and process name
Enterprise Manager host and port number
If you select an existing profile, you are prompted for the location of the file. Specify
the fully qualified path to the file.
The settings you specify can be changed after installation if needed, by editing the
IntroscopeAgent.profile file.
8. Specify the path to the Java executable that is used to launch your application
server.
9. Specify the ProbeBuilding method for instrumenting your application. In most
cases, use JVM AutoProbe.


10. Specify whether to enable the ChangeDetector agent extension.
If you enable ChangeDetector, you are prompted to name the ChangeDetector
agent.
If you do not enable ChangeDetector, the ChangeDetector files are installed in the
<Agent_Home>/examples directory. You can enable ChangeDetector later by
copying the files to the <Agent_Home>/core/ext directory and modifying the
Note: For more information about ChangeDetector, see the CA APM
ChangeDetector Guide.
11. Select additional monitoring options to install. For example, select CA APM for SOA
if you want to monitor web services and other SOA environment components.
If you do not enable other monitoring options during installation, the files are
installed in the <Agent_Home>/examples directory and can be enabled later.
12. Specify the location of a pickup folder for .zip or .tar files that contain add-ons
you want installed with the agent.
Any .zip or .tar files found in the specified location are extracted into the
<Agent_Home> directory.
13. Review your selections and continue to install the agent, and then complete the
installation.
Note: Installing the agent does not stop or start the application server, or configure the
application server startup script. Perform these tasks manually after installation. The
specific steps depend on the type of application server you are monitoring.
Install the Java Agent Silently
In silent mode, you invoke the agent installer from a command line and specify a
response file that contains installation instructions. The installation then runs in the
background without displaying any information about its progress. Because this
installation method enables you to install the agent without user interaction, it is most
commonly used to install agents on remote computers or to install multiple agents with
the same configuration.
If you have previously installed a Java agent, you can use the automatically-generated
response file to install additional agents. Alternatively, you can manually edit the sample
response file provided in the installation or create your own response file using a text
editor.


About Automatically Generated Response Files
Any time you run the Java agent installer (interactively or silently), the installer creates a
response file that records the installation options you selected. This automatically
generated response file is saved in the <Agent_Home>/install directory. The file name
indicates the date and time that the installer created the response file in the following
format:
autogenerated.responsefile.<year>.<month>.<day>.<hour>.<minutes>.<seconds>
For example, an installation done April 30, 2005 at 7:10:05 a.m. would have an
automatically generated response file with this name:
autogenerated.responsefile.2005.4.30.7.10.05
You can use this automatically generated response file for subsequent silent
installations with the same settings or edit it to use new settings.
About the sample response file
If you have not previously installed the Java agent or want to use default instead of
previously selected installation options, you can edit the default sample response file
included with the Java agent installer. The default sample response file is located in the
<Agent_Home>\install directory and named:
SampleResponseFile.Agent.txt
The sample response file provides default settings for most properties, but you must
manually edit the file before you can use it for silent installation.
Configure the response file properties and install the agent
Whether you use the automatically-generated response file, the default sample
response file, or create a custom response file, you must configure the properties in the
file appropriately before invoking the Java agent installer in silent mode. The properties
you set in the response file are equivalent to the selections you make when running the
installer interactively.
Note: For additional details about setting any property, see the comments in the
<Agent_Home>/install/SampleResponse.Agent.txt file.


Follow these steps:
1. Open the response file in a text editor.
2. Set the appropriate property values. The properties to configure are:
USER_INSTALL_DIR=<root_installation_directory>
Specifies the directory for installing the agent. In most cases, you should use
the application server root directory.
silentInstallChosenFeatures=Agent
Specifies the components you want to install.
appServer=Default
Specifies the type of application server to monitor. Valid values are Default,
JBoss, Tomcat, WebLogic, WebSphere, Sun, Oracle, or Interstage. The value is
case-sensitive.
This setting controls the ProbeBuilder Directives (PBDs) that are installed with
the agent and which additional monitoring options are valid.
(Optional) appServerHome=
Specifies the home directory of the application server. This property is not
needed if you have set the USER_INSTALL_DIR property to the application
server root directory.
(Optional) appServerJavaExecutable=
Specifies the path to the Java executable that is used to launch the application
server.
instrumentationLevel=Typical
Specifies whether you want to use Full or Typical instrumentation. The value is
case-sensitive.
agentName=Default Agent
Specifies the agent name that should be displayed in the Workstation.
processName=Default Process
Specifies the process name that should be displayed in the Workstation.
emHost=localhost
Specifies the host name of the computer running the Enterprise Manager that
the agent should connect to by default.
emPort=5001
Specifies the port number the agent should use to connect to its Enterprise
Manager.


(Optional) alternateAgentProfile=
Specifies the absolute path to an existing agent profile.
(Optional) pickupFolder=
Specifies the absolute path to a "pickup folder" that contains .zip or .tar files for
any add-ons you want to install with the agent.
(Optional) changeDetectorEnable=false
Specifies whether to enable the ChangeDetector agent extension. If you set this
property to false, ChangeDetector files are installed but not enabled. You may
enable it later.
(Optional) changeDetectorAgentID=
Specifies a name for the ChangeDetector agent extension. If you enable
ChangeDetector, uncomment and set this property.
(Optional) shouldEnable*
Specifies which additional CA APM monitoring solutions you want to enable. All
of the properties for optional CA APM monitoring solutions are set to false by
default. The options that are valid to enable depend on the application server
type.
3. Save the response file and close the text editor.
4. Invoke the installer in silent mode by specifying the path to the installer executable
and the absolute path to the response file.
<path_to_installer> -f <absolute_path_to_response-file>
The command you use to invoke the installer depends on the operating system
where you are running the command.
For example, on Windows, the command format looks like this:
IntroscopeAgent<version>windows.exe -f C:\temp\myResponseFile.txt
On Linux or UNIX, the command format looks like this:
./IntroscopeAgent<version>unix.bin -f /
Select the appropriate command format for your operating system.
5. Check the <Agent_Home>/install/Introscope_Agent_<version>_InstallLog.log file to
verify the agent has been installed successfully.
About the Java Agent Directory Structure


Install Manually Using Installation Archives
You can put the agent files on a system without running the Java agent installer
interactively or configuring a response file. Application server-specific archives let you
install the agent.
The installation archives contain all of the files that would have been installed had you
run the agent installer. After you copy an archive to a computer, you extract the
contents and configure the agent profile to identify the Enterprise Manager to connect
to and other properties. Use these files to deploy multiple agents in a batch job, or keep
the files as an archive of the default set of agent files.
To install the Java agent manually from an archive, verify that the computer where you
plan to install has 35 MB of free disk space.
Follow these steps:
1. Select the appropriate installation archive for your application server and operating
system.
2. Extract the contents of the archive into a location your JVM can access using a
command appropriate to your operating system. For example, on UNIX or Linux:
tar -xvf IntroscopeAgentFilesOnly-NoInstaller<version><app_server>.<os>.tar
3. Open the <Agent_Home>/core/config/IntroscopeAgent.profile file in a text editor
and configure the connection to the Enterprise Manager.
For more information about setting the properties for communication with the
Enterprise Manager, see Configuring the connection to the Enterprise Manager (see
page 59).

4. Configure any additional properties, then save and close the
5. Configure the application server with the location of the Java agent startup file and
the agent profile.
6. Restart the application server.
When you install the agent, the following directory structure is created in the root
installation directory:
wily
This directory is the <Agent_Home> directory, which contains Agent.jar that is used
to start the agent.
Within the <Agent_Home> directory are subdirectories that provide the libraries
and extension files that enable various features of the Java agent.


core
config
Contains the IntroscopeAgent.profile, ProbeBuilder Directive files (.pbd), and
ProbeBuilder List files (.pbl) files that control agent operations, metric data
collection, and the instrumentation process. The specific properties defined in
the IntroscopeAgent.profile and the .pbd and .pbl files that are deployed and
referenced in the agent profile depend on choices you made during
installation.
Within the config directory, the hotdeploy subdirectory enables you to deploy
new directives without editing the IntroscopeAgent.profile or restarting
applications. If files placed in the hotdeploy directory contain invalid syntax or
too many metrics, then Instrumentation can fail or application performance
can suffer.
ext
Contains the files for enabled agent extensions or features. For example, the
directory contains files for the application triage map, and LeakHunter.
connectors
Contains the CreateAutoProbeConnector.jar utility that is used to configure an
AutoProbe Connector that enables instrumentation in certain environments.
common
Contains the configuration and property files for the extensions.
examples
Contains folders and files for optional agent extensions, such as CA APM for
SOA. If you did not enable an extension at installation, you can use the files in
this directory to configure the extension later.

install
Contains log files that record the installation process and the files you can use
for silent installation. For example, the generated response file and the
SampleResponseFile.Agent.txt file are located in this directory.
This directory is not created if you manually install the Java agent from an
installation archive.
logs
Stores agent log files.
Configuring Detection of Synthetic Transactions


tools
Holds the following content:
The URLGrouper.jar command-line utility that analyzes web server log
files.
The list of the files of extensions. For example, configurePMI.bat,
CreateIU.jar, listServers.bat, MergeUtility.jar, NetInterface.jar, and
setPmiModules.jacl files are located in this directory.
The TagScript tools files: TagScript.jar, TagScript.bat, and TagScript.sh
command-line scripts.
UninstallerData
Contains a subdirectory with the executables and associated resources that are
used to uninstall the agent and related resources.
This directory is not created if you manually install the Java agent from an
installation archive.
version
Contains version information for optional CA APM monitoring solutions. This
information is installed regardless of what you enable at installation time.
Configuration settings for synthetic transaction monitoring are done using the
introscope.agent.synthetic.header.names parameter.
The introscope.agent.synthetic.header.names parameter value lists the HTTP header
parameters that are used to determine whether a monitored HTTP request is part of a
synthetic transaction. Commas separate the Individual parameter names. If this
parameter is undefined or the value is empty, synthetic transactions are not detected. If
multiple HTTP header parameter names are defined, they are examined in the specified
order. The first HTTP parameter with a value is used to define the synthetic transaction.
The node under which synthetic transactions are reported depends on the specific HTTP
header parameter that is used to detect each transaction as follows:
If the parameter value is anything other than lisaframeid or x-wtg-info, the HTTP
parameter value itself is used as the node name. Appropriate modification is done
to ensure that a valid node name is used.
If the parameter value is lisaframeid, the synthetic node name is CA LISA.
If the parameter value is x-wtg-info, the HTTP header parameter value is assumed
to contains a sequence of name and value pairs. The ampersand symbol separates
the pairs from each other. An equal sign separates the attribute name and value
within each pair. The values of group, name, ipaddress, and request_id with a |
node separator form the synthetic transaction node names.


For example, given the parameter:
introscope.agent.synthetic.header.names=Synthetic_Transaction,x-wtg-info,lisafram
eid
The following x-wtg-info header results in metrics being reported under node
SampleGroup|sample|192.168.193.1|start:
clear
synthetic=true&instance=ewing&name=sample&group=SampleGroup&version=4.1.0&ipaddre
ss=192.168.193.1&sequencenumber=1&request_id=start&executiontime=1226455047

Any attributes that are not defined in the x-wtg-info HTTP header parameter value have
default values that are supplied as follows:
group=unknownGroup
name=unknownScript
ipaddress=0.0.0.0
request_id=Action

If introscope.agent.synthetic.header.names is undefined, the following configuration
parameters are ignored.
introscope.agent.synthetic.node.name=Synthetic Users
The node under which transactions recognized as synthetic are reported. This node
is located under Frontends|Apps|<WebAppName> where <WebAppName> is the
web application name. This value defaults to Synthetic Users.

introscope.agent.non.synthetic.node.name=Real Users
The node under which transactions not recognized as synthetic are reported. This
node is located under Frontends|Apps|<WebAppName> where <WebAppName> is
the web application name. If undefined, no additional node under <WebAppName>
is created.

introscope.agent.synthetic.user.name=Synthetic_Trace_By_Vuser
The name of the HTTP header parameter whose value is used as the synthetic user
name. The synthetic user name is used to separate different synthetic transactions.
A node for each synthetic user name is created below the Synthetic User node. If
this configuration parameter is defined and an HTTP header parameter of this name
exists, synthetic transaction metrics are reported. The node under which the
transactions are reported is <Synthetic Users>|<Synthetic User> where:


The introscope.agent.synthetic.node.name configuration parameter
determines the <Synthetic Users> node name.
The HTTP header parameter value determines the <Synthetic User> node
name.
Note: Changes to these properties take effect immediately and do not require you to
restart the managed application.
Using the TagScript Utility
The CA TagScript utility can be used with HP Vugen to specify extraction of synthetic
user information.
To use the TagScript utility
1. Open the TagScript utility.
For Windows:
<Agent_Home>\wily\tools\TagScript.bat
For UNIX:
<Agent_Home>/wily/tools/TagScript.sh
You are asked which environment you want to modify scripts for.
2. Click one of the following options:
Performance Testing - for HP Loadrunner scripts
Production - for HP Business Process Monitor or Sitescope scripts
Un-tag - to reverse the tagging process
3. Navigate to the directory where your HP Vugen scripts are located. Double-click
each .c script to open them.
The HP Vugen script .c files are all backed up and replaced by modified versions.
4. If HP Vugen is open and the utility is executed, you are prompted to reload the
modified scripts. When prompted click Yes to All.
5. You can either close the TagScript utility or click the cancel button in the file
selection dialog. You are not required to close the TagScript utility and many users
do not close the utility while in HP Vugen. If a script is modified or any new scripts
are created, not closing the utility simplifies the process.
6. Verify that your scripts have been tagged in the following locations:
The new paragraphs of HP Vugen code are inserted at the beginning of each
script.
Tags appear before all lr_start_transaction, lr_end_transaction, and at the end
of the script.
Java 7 Autoprobe


7. (Optionally) You can track each virtual user in an HP Loadrunner performance test
with an individual set of blame stacks. To track each user, uncomment the following
line at the beginning of the script in the declaration paragraph:
web_add_auto_header(Synthetic_Trace_By_Vuser,vuserOverview)
Note: If this option is uncommented for a Production tagged script, an individual set
of blame stacks is created for each point of presence or synthetic generator.
Java 7 Autoprobe
Add -XX:-UseSplitVerifier if you use applications that are compiled and executed with
Java 1.7, otherwise the agent can cause bytecode verification errors similar to:
java.lang.VerifyError: StackMapTable error: bad offset,
or
java.lang.ClassFormatError: Illegal local variable table.
This instruction applies if you are using JVM AutoProbe through javaagent or
Xbootclasspath.
How to instrument applications
After you have installed the agent, you must configure the application server to
instrument your applications. The most common method for instrumenting applications
is to use JVM AutoProbe and the javaagent command line option. JVM AutoProbe
instruments applications dynamically during runtime. It is suitable for all J2EE
application servers that provide a hook in the bootstrap or application server class
loader for Introscope to see all bytecode loaded from the file system.
Most JVM providers support the javaagent option. If you are using a JVM that does not
provide support for this option you must use an alternate method for instrumentation.
Running ProbeBuilder manually is only required if you must instrument bytecode
statically before an application is started. You can run ProbeBuilder manually using the
ProbeBuilder wizard or from a command-line prompt. It instruments bytecode and
outputs a newly named, instrumented jar or class file. This newly instrumented
bytecode is then placed ahead on the classpath (or renamed in place) for the application
prior to its being started.
For more information about the alternatives to using JVM AutoProbe, see Appendix B:
Alternative Methods for Instrumentation.
Configuring the Application Server to Start the Java Agent


You must configure the application server you are instrumenting to include the path to
the agents primary .jar file and the agent profile. In most cases, you do this by editing
the application server's startup script, then restarting the application server. When the
application server restarts, the Java agent instruments the classes discovered for default
components of the JVM and application environment. The specific steps involved
depend on the application server.
Configure Apache Tomcat to use the Java agent
To configure Apache Tomcat to use the Java agent, you must edit the Tomcat startup
script. By default, the Tomcat startup script is catalina.sh or catalina.bat in the
$CATALINA_HOME/bin directory. Depending on the requirements of your web server,
you may use a different startup script or a different location for the startup script.
Follow these steps:
1. Navigate to the directory that has the Tomcat startup script. For example:
cd /apache-tomcat-6.0.18/bin
2. Open the Tomcat startup script in a text editor. For example:
vi catalina.sh
3. Locate the command line for setting Java options and add the following
command-line options to specify the path to the agent's startup .jar file and the
agent profile:
-javaagent:<PathToAgentJar>
-Dcom.wily.introscope.agentProfile=<PathToAgentProfile>
For example, if using Agent.jar, add code similar to the following before the
command that starts the server:
set JAVA_OPTS=%JAVA_OPTS% -javaagent:c:\apache-root\wily\Agent.jar
-Dcom.wily.introscope.agentProfile=
c:\apache-root\wily\core\config\IntroscopeAgent.profile
4. Save the startup script.
5. (Optional) Enable the reporting of Apache Tomcat JMX metrics by configuring the
agent profile to collect JMX metrics. Open the IntroscopeAgent.profile and set the
following property:
introscope.agent.jmx.enable=true
Note: If you want to see JMX metrics from Apache Tomcat in a console by using the
JMX remote management server with a platform-specific MBeanServer, add
com.wily.use.platform.mbeanserver=true to the IntroscopeAgent.profile. This
configuration is changed from previous versions of Introscope where the use of the
platform-specific MBeanServer was set in the command line.


6. Save and close the IntroscopeAgent.profile.
7. Restart the Tomcat server.
Customize Data Collection
After you install the Java agent on an Apache Tomcat server, you can modify a PBD to
customize data collection.
Follow these steps:
1. Navigate to the <Agent_Home>\wily\core\config directory.
2. Open the tomcat.pbd file in a text editor.
3. Modify the sections you want to customize.
4. Save and close the file.
Configure JBoss to Use the Java Agent
To configure JBoss to use the Java agent, you edit the JBoss startup script that
corresponds to your JBoss version:
JBoss startup script for JBoss 7 at a minimum
The default JBoss startup script for standalone mode is standalone.sh or
standalone.bat in the $JBOSS_HOME/bin directory. The default for domain mode is
domain.sh or domain.bat in the $JBOSS_HOME/bin directory.
JBoss startup script for JBoss 6 and previous versions
By default, the JBoss startup script is run.sh or run.bat in the $JBOSS_HOME/bin
directory.
Depending on the requirements of the web server, you can use a different startup script
or a different location for the startup script.

Follow these steps:
1. Navigate to the directory that has the JBoss startup script, for example:
cd /jboss.GA/bin
2. Open the JBoss startup script in a text editor, for example:
vi run.sh / vi standalone.sh


3. Locate the command line for setting Java options. To specify the path to the startup
.jar file for the agent and the path to the agent profile, add the following
command-line options:
For JBoss 7 at a minimum
-Djboss.modules.system.pkgs=com.wily,com.wily.*
For JBoss 6 and previous versions
For example, if you are using Agent.jar add code similar to the following examples
before the command that starts the server:
For JBoss 7 at a minimum
set JAVA_OPTS= %JAVA_OPTS%
-Djboss.modules.system.pkgs=com.wily,com.wily.*
-javaagent:%JBOSS_HOME%\wily\Agent.jar
-Dcom.wily.introscope.agentProfile=%JBOSS_HOME%\wily\core\config\Introsc
opeAgent.profile
For JBoss 6 and previous versions
set JAVA_OPTS= -javaagent:%JBOSS_HOME%\wily\Agent.jar
-Dcom.wily.introscope.agentProfile=%JBOSS_HOME%\wily\core\config\Introsc
opeAgent.profile %JAVA_OPTS%
4. Save the run.bat or standalone.bat file.
5. (Optional) Enable the reporting of JBoss JMX metrics by configuring the agent
profile to collect JMX metrics.
a. Open the IntroscopeAgent.profile in a text editor and set
introscope.agent.jmx.enable=true.
Note: To see JMX metrics from JBoss in a JConsole by using the JMX remote
management server with a platform-specific MBeanServer, add
com.wily.use.platform.mbeanserver=true to the IntroscopeAgent.profile. This
configuration is changed from previous versions of Introscope where the use of
the platform-specific MBeanServer was set in the command line.
b. Save and close the introscopeAgent.profile file.
c. For JBoss 7 at a minimum, navigate to your JBoss installation
<Agent_Home>/wily/common directory and move the WebAppSupport.jar file
into the <Agent_Home>/wily/core/ext directory.
d. For JBoss 6 and previous versions:
Navigate to your <Agent_Home>/wily/common directory and move the
WebAppSupport.jar file into the /server/<server_configuration>/lib
directory.


Navigate to the <Agent_Home>/wily/deploy directory and move the
introscope-jboss-service.xml file into your
/server/<server_configuration>/deploy directory.
Note: These paths assume that you are using the default configuration. If not,
move the files into the equivalent JBoss installation directory.
JBoss Application Server Logging Considerations
Consider the following information when you use a JBoss application server.
Symptom:
The following behavior is common for any agent that performs logging on the
application server:
For JBoss 6 systems, the application server does not create the boot.log file.
For JBoss 7 systems, the agent does not start or the following error appears:
The LogManager was not properly installed
Solution:
Use one of the following methods to resolve these issues:
Turn off the logging of the agent in the agent profile and start the server. Enable the
logging after the server starts.
Open the JBoss startup script in a text editor, and update with:
set JAVA_OPTS= %JAVA_OPTS%
-Djboss.modules.system.pkgs=org.jboss.logmanager,com.wily,com.wily.*
-Djava.util.logging.manager=org.jboss.logmanager.LogManager
-javaagent:%JBOSS_HOME%\wily\Agent.jar
-Dcom.wily.introscope.agentProfile=%JBOSS_HOME%\wily\core\config\IntroscopeAg
ent.profile
-Xbootclasspath/p:%JBOSS_HOME%\modules\org\jboss\logmanager\main\jboss-logman
ager-1.2.2.GA.jar;%JBOSS_HOME%\modules\org\jboss\logmanager\log4j\main\jboss-
logmanager-log4j-1.0.0.GA.jar;%JBOSS_HOME%\modules\org\apache\log4j\main\log4
j-1.2.16.jar


JBoss-specific PBDs and PBLs
When you install the Java Agent on a JBoss application server, the following
JBoss-specific PBDs and PBLs are available for you to customize data collection:
jboss4x.pbd
jsf.pbd
jsf-toggles-full.pbd
jsf-toggles-typical.pbd
jboss-full.pbl
jboss-typical.pbl
Configure JBoss 7 for Autonaming
The autonaming feature works with JBoss 7 only if you copy the webappsupport.jar file
to the appserver deployment folder.
Example
For a standalone server, copy webappsupport.jar to the
JBOSS7_HOME/standalone/deployments/ folder.
Configure Oracle WebLogic to Use the Java Agent
To configure Oracle WebLogic to use the Java agent, you edit the WebLogic startup
script. Depending on your requirements, the startup script you use can be specific to a
WebLogic domain. By default, the WebLogic startup script is startWebLogic.sh or
startWebLogic.cmd in the $WEBLOGIC_HOME/samples/domain/<domain_name>/bin
directory. You can use a different startup script. For example, you can use an
application-specific startup script or a different location for the startup script.
Follow these steps:
1. Navigate to the directory that has the WebLogic startup script you want to modify.
For example:
cd $WL_HOME/samples/domain/wl_server
2. Open the WebLogic startup script in a text editor. For example:
vi startWebLogic.sh
3. Locate the command line for setting Java options or the Java command line and add
the following command-line options. For example: -javaagent:<PathToAgentJar>
4. Save and close the WebLogic startup script or application-specific startup script.


Create a startup class for WebLogic
Creating a WebLogic startup class for an application server or cluster enables the Java
agent to collect additional information from the application server. If you configure a
startup class, the Java agent can automatically determine its name. The startup class
also enables the Java agent to report JMX metrics that are used to determine
application health in the Workstation.
Follow these steps:
1. Open the WebLogic Administrative Console.
2. In the left pane, expand the Environments folder.
3. Click the Startup & Shutdown folder.
The Startup and Shutdown page opens.
4. Click Configure a New Startup Class.
The Configuration tab is shown.
5. In the Name field, enter:
Introscope Startup Class
6. In the ClassName field, enter:
com.wily.introscope.api.weblogic.IntroscopeStartupClass
7. Click Create.
The Target and Deploy tab appears.
8. Select the boxes for the servers you want to make this startup class available to.
9. Click Apply and select the Run before application deployments option.
10. Add the location of the WebAppSupport.jar to the <server or application server>
startup classpath.
Enable cross-process tracing in WebLogic Server
Transaction trace sessions enable you to trace all of the operations that take place in a
transaction, including transactions that cross JVM boundaries on computers with
compatible JVM versions.


Cross-process transaction tracing is supported for synchronous transactions, for
instance servlets to EJBs, and asynchronous transactions.
To enable cross-process transaction tracing for WebLogic
1. Create a startup class as described in Creating a startup class for WebLogic (see
page 41).
2. Add "-Dweblogic.TracingEnabled=true" to the java command line for starting
WebLogic Server.
3. Open the IntroscopeAgent.profile in a text editor.
4. Locate and set introscope.agent.weblogic.crossjvm property to true. For example:
introscope.agent.weblogic.crossjvm=true
Java Agent Support for JMX Metrics
CA Introscope can collect management data that application servers or Java
applications expose as JMX-compliant MBeans and can present the JMX data in the
Investigator metric tree. WebLogic provides the following sources of JMX metrics:
RuntimeServiceMBean: per-server runtime metrics, including active effective
configuration
DomainRuntimeServiceMBean: domain-wide runtime metrics
EditServiceMBean: allows user to edit persistent configuration
CA Introscope polls only RuntimeServiceMBean for metrics. RuntimeServiceMBean
supports local access (an efficiency issue), and contains most of the data that is
expected to be relevant. However, CA Introscope can support any MBean built to the
Sun JMX specification.
Note: For more information about the Sun JMX specification, see
http://java.sun.com/products/JavaManagement/.


To support the collection of JMX metrics, the following keywords are defined and
enabled by default in the IntroscopeAgent.profile file for WebLogic:
ActiveConnectionsCurrentCount
WaitingForConnectionCurrentCount
PendingRequestCurrentCount
ExecuteThreadCurrentIdleCount
OpenSessionsCurrentCount
CA Introscope displays the JMX metrics in the Investigator tree under the following
node:
<Domain>|<Host>|<Process>|AgentName|JMX|
More information:
Enabling JMX Reporting (see page 187)

How to Set Up Resource Metrics in Weblogic
Various agents can report resource metric categories in the CA Introscope
Workstation. You can configure an Oracle WebLogic server so that a Java agent can
report resource metrics.
To set up resource metrics for Oracle WebLogic, follow these steps:
1. Install CA APM for Oracle WebLogic using the instructions in the CA APM for Oracle
WebLogic Server Guide.
2. Enable the CA APM for Oracle Databases option using the CA APM for Oracle
Databases Guide.
3. Follow the instructions for configuring resource metrics in the CA APM
Configuration and Administration Guide.
About WebLogic Diagnostic Framework (WLDF)
The WebLogic Diagnostic Framework (WLDF) is a monitoring and diagnostic framework
that defines and implements a set of services that run within the WebLogic Server
process and participate in the standard server life cycle. Using WLDF, you can create,
collect, analyze, archive and access diagnostic data generated by a running server and
the applications deployed within its containers. This data provides insight into the
run-time performance of servers and applications and enables you to isolate and
diagnose faults when they occur.


WLDF enables dynamic access to server data through standard interfaces and the
volume of data accessed at any given time can be modified without shutting down and
restarting the server.
How WLDF data is converted into Introscope metrics
In WLDF, information is organized into a set of Data Accessors, each with multiple
Columns. Introscope converts this WLDF information into an Introscope-specific metric
format and displays it in the Investigator under the following node:
<Domain>|<Host>|<Process>|AgentName|WLDF|
Information in the Data Accessors is defined by a domain name and one or more
key/value pairs. Introscope converts Data Accessor Columns into key and value
information displayed in alphabetical order in the Introscope-generated metrics. The
following example shows the syntax used:
<Domain>|<Host>|<Process>|AgentName|WLDF|<domain
name>|<key1>=<value1>|<key2>=<value2>:<metric>
For example, this table shows the information for the BYTECOUNT Column from the
HTTPAccessLog Data Accessor:

Domain name Key/Value pairs Metric names
WebLogic Name=HTTPAccessLog,
Type=WLDFDataAccessRuntime=Accessor,
WLDFRuntime=WLDFRuntime
BYTECOUNT
The Data Accessor information in the table above would be converted to the following
Introscope metric:
<Domain>|<Host>|<Process>|AgentName|WLDF|Weblogic|Name=HTTPAccessLog
|Type=WLDFDataAccessRuntime=Accessor|WLDFRuntime=WLDFRuntime:BYTECOUNT
Note that the key/value pairs are displayed alphabetically in the Introscope metric.


Enable WLDF reporting
By default, WLDF reporting is not enabled in Introscope. You can modify the agent
profile to enable reporting of WLDF metrics.
To enable WLDF reporting
1. Shut down the managed application, if it is running.
2. Configure a WebLogic startup class.
4. Locate and set the following property:
introscope.agent.wldf.enable=true
Configure WebLogic with JRockit JVM to Use the Java Agent
Valid for: WebLogic 9.0 at a minimum with JRockit JVM 1.5 at a minimum
Note: For information about the supported versions of WebLogic, see the Compatibility
Guide. If you are using an older version of WebLogic or JRockit JVM, you can create and
run an AutoProbe connector file (see page 335) to instrument applications.
If you are using WebLogic with JRockit JVM, use the following command-line options to
start the JVM:
JAVA_VENDOR=Bea
JAVA_OPTIONS=%JAVA_OPTIONS% -javaagent:PathToAgentJar
-Dcom.wily.introscope.agentProfile=PathToIntroscopeAgent.profile
Configure WebLogic with JRockit JVM to View Socket Metrics
Valid for: WebLogic 9.0 with JRockit JVM 1.5.0
Symptom:
Due to a third-party compatibility issue, you can experience a problem with viewing
metrics for socket clients.


Solution:
Turn on metrics for socket clients by using the following Managed Socket option.
Follow these steps:
1. Open the file toggles_typical.pbl or toggles_full.pbl in a text editor.
2. Turn on the Managed Socket option to trace socket metrics. For example:
#######################
# Network Configuration
# ================
#TurnOn: SocketTracing
# NOTE: Only one of SocketTracing and ManagedSocketTracing should be 'on'.
ManagedSocketTracing is provided to
# enable pre 9.0 socket tracing.
TurnOn: ManagedSocketTracing
TurnOn: UDPTracing
3. Save your changes.
The Managed Socket option is turned on.
Configure IBM WebSphere to Use the Java Agent
To configure IBM WebSphere to use the Java agent, you edit the WebSphere startup
script. Depending on your requirements, the startup script you use can be specific to an
application server node.
Follow these steps:
1. Navigate to $WEBSPHERE_HOME/profiles/<App Server Name>/config/cells/<Cell
Name>/nodes/<Node name>/servers/server1directory.
2. Edit the server.xml file. This file is the WebSphere default startup script and
location.
Note: Depending on the application server requirements, you can use a different
startup script. For example, you can use an application-specific startup script or a
different location for the startup script. In addition, different WebSphere
combinations on different operating systems or using different JVM vendors or JVM
versions can have special requirements.
Note: For more information, see the section that most closely describes your
WebSphere application server environment.


Configure WebSphere Application Server 6.1 on UNIX, Windows, OS/400, z/OS, IBM JVM 1.5
Valid for: WebSphere Application Server 6.1 on UNIX, Windows, OS/400, z/OS, IBM
JVM 1.5
The dynamic instrumentation feature requires class redefinition support. The use of
class redefinition can significantly affect performance when running on IBM JDK version
5. CA Introscope and IBM JDK version 5 customers that want to use dynamic
instrumentation should be aware of this performance overhead. When employing this
configuration, CA Technologies recommends using the dynamic instrumentation feature
only in a QA environment.
Note: For information about this performance overhead, see the IBM Java Diagnostics
Guide.

When you are running WebSphere 6.1 using an IBM JVM 1.5, use the alternate versions
of the Java agent .jar file and Java agent profile. These files, named AgentNoRedef.jar
and IntroscopeAgent.NoRedef.profile, are located in the <Agent_Home>/core/config
directory.
Note: If you are using the AllAppServer agent distribution, the alternate profile is named
IntroscopeAgent.websphere.NoRedef.profile.
As a result of using the previous files and syntax, metrics are no longer reported for:
System classes
NIO (Sockets and Datagrams)
SSL
The following features are also affected:
The socket instrumentation uses the ManagedSocket style for CA Introscope
earlier than 9.0.
Remote dynamic instrumentation is disabled
Changes to the PBD files require that you restart the instrumented JVM before
being applied
The deep inheritance and hierarchy support instrumentation is disabled.
When using the previous files and syntax, the agent reports class redefinition status by
the following actions:
Adds a metric named Agent Class Redefinition Enabled beneath the agent node in
the Investigator. The metric values are true or false.
Writes a log message to the agent log file.


When class redefinition is enabled, the log message is written at the WARN
level and the message reads:
Introscope Agent Class Redefinition is enabled. Enabling class redefinition
on IBM 1.5 JVMs is known to incur significant overhead.
When class redefinition is disabled, the log message is written at INFO level and
the message reads:
Introscope Agent Class Redefinition is disabled.
Writes a message to standard error (only when class redefinition is enabled), which
reads:
Warning: Introscope agent has been configured to support class redefinition.
IBM JVMs version 1.5 and higher are known to incur significant overhead with
redefinition enabled. To avoid this overhead please use AgentNoRedef.jar
instead of Agent.jar.
If you use a non-IBM JVM or an IBM JVM that is a version other than 1.5, the previous
metric and messages are not output.
You configure WebSphere Application Server to use the agent.
Follow these steps:
1. Open the WebSphere Administrator Console.
2. Navigate to Application Servers > your server > Server Infrastructure > Java and
Process Management > Process Definition > Java Virtual Machine.
3. Set the Generic JVM Argument field as follows:
-javaagent:<Agent_Home>/AgentNoRedef.jar
-Dcom.wily.introscope.agentProfile=<Agent_Home>/core/config/IntroscopeAgent.N
oRedef.profile
If you have both instrumented and noninstrumented applications on the same
computer, include the -Xshareclasses:none setting in the Generic JVM Argument.
This setting avoids errors on AIX.
Note: A unique directory is required when there is more than one version of
WebSphere using the same agent directory.


Configure WebSphere Application Server 6.1 on UNIX, Windows, Sun, HP, Other JVM 1.5
Valid for: WebSphere Application Server 6.1 on UNIX, Windows, Sun, HP, all other
JVM 1.5
Note: If you use a non-IBM JVM or an IBM JVM version other than 1.5, not all metrics
and messages are output.
You can configure the WebSphere Application Server to use the agent.
The following examples indicate which Java argument and .jar file to use with a specific
JVM when installing the Java Agent on WebSphere 6.1.
Follow these steps:
-javaagent:<Agent_Home>/Agent.jar
-Dcom.wily.introscope.agentProfile=<Agent_Home>/core/config/IntroscopeAgent.p
rofile
4. Restart the WebSphere application server.
Configure WebSphere Application Server 7.0 on UNIX, Windows, OS/400, JVM 1.5
Valid for: WebSphere Application Server 7.0 - UNIX, Windows, OS/400, JVM 1.5 at a
minimum
You can configure WebSphere Application Server 7.0 to use the agent. If you are
monitoring WebSphere 7.0, verify that you have build 7.0.0.8 at a minimum installed
before configuring the server to start the Java agent.
Follow these steps:
-javaagent:<Agent_Home>/Agent.jar
rofile


Configure WebSphere Application Server 7.0 on z/OS - JVM 1.5
Valid for: WebSphere Application Server 7.0 on z/OS - JVM 1.5 at a minimum
You can configure WebSphere Application Server 7.0 to use the agent. Verify that you
install build 7.0.0.8 at a minimum before you configure the server to start the Java
agent.
Follow these steps:
-Xbootclasspath/a:<Agent_Home>/Agent.jar -javaagent:<Agent_Home>/Agent.jar
rofile



Modifying Java2 Security Policy
For AutoProbe to run correctly in WebSphere environments with Java2 Security
enabled, it may be necessary to add permissions to your Java2 Security Policy.
To add permissions to your Java2 Security Policy
1. Open the file $WebSphere home/properties/server.policy in a text editor.
2. Add the following permissions to the file:
// permissions for Introscope AutoProbe
grant codeBase "file:${was.install.root}/-" {
permission java.io.FilePermission "${was.install.root}${/
}wily${/}-", "read";
permission java.net.SocketPermission "*", "connect,resolve";
permission java.lang.RuntimePermission "setIO";
permission java.lang.RuntimePermission "getClassLoader";
permission java.lang.RuntimePermission "modifyThread";
permission java.lang.RuntimePermission "modifyThreadGroup";
permission java.lang.RuntimePermission "loadLibrary.*";
permission java.lang.RuntimePermission "accessClassInPackage.*";
permission java.lang.RuntimePermission "accessDeclaredMembers";
};
grant {
permission java.util.PropertyPermission "*", "read,write";
};
Note: Line breaks are shown for user readability and are not needed when adding
the permissions to the server.policy file.
Configure a Custom Service in WebSphere
You can create or modify a custom service in the WebSphere Application Server. The
custom service enables the Java agent to collect additional information from the
application server. If you configure a custom service, the Java agent can automatically
determine its name. The custom service also enables the Java agent to report JMX and
Performance Monitoring Infrastructure (PMI) metrics. The Introscope Workstation on
the Application Overview tab uses these metrics to determine application health. The
custom service can encrypt the user credentials that you want to use to access JMX
metrics in CA Introscope.
Note: To get SIBus-related metrics or new PMI modules that have been added to
WebSphere Application Server, you disable existing the Custom Service and then create
a Custom Service.


Follow these steps:
2. Select the server that you want to configure and navigate to Server Infrastructure >
Administration > Custom Services.
3. Modify the custom service that you want or create one.
4. Complete the following fields on the Configuration page and click OK.
Enable service at server startup
Specifies that the service starts during the server startup.

External Configuration URL
Specifies the location of the configuration properties file. For JMX metrics
configuration, you use the jmxconfig.properties file, for example:
<Agent_Home>/wily/common/jmxconfig.properties.
Classname
Specifies the name of the custom service class, for example:
com.wily.introscope.api.websphere.IntroscopeCustomService
com.wily.powerpack.websphere.agent.PPCustomService
Display Name
Specifies the name to display in CA Introscope, for example: Introscope
Custom Service.
Classpath
Specifies the fully qualified pathname of the properties file, for example:
<Agent_Home>/wily/common/WebAppSupport.jar
<Agent_Home>/wily/common/PowerpackForWebSphere_Agent
5. Configure the user credentials for accessing JMX metrics:
a. Navigate to <Agent_Home>/wily/common and open the jmxconfig.properties
file in a text editor.
b. Uncomment and set values according to the property descriptions.
c. Save and close the file.
When the server starts, the custom service starts and the user credentials are
encrypted. Thereafter, whenever the server starts, it uses the encrypted password.


Example: Configure the jmxconfig.properties File for Password Encryption
The following text shows an example of the jmxconfig.properties set for password
encryption.
jmxUsername=dave
jmxPassword=mypassword
plainTextPassword=true
More information:
Configure WebSphere and WebLogic to Use JMX Reporting (see page 191)

Enable the Collection of WebSphere PMI Metrics
CA Introscope can provide WebSphere performance data by extracting WebSphere
Performance Monitoring Infrastructure (PMI) metrics using the PMI interface that
WebSphere provides.
You enable the PMI data collection in WebSphere before the data can be available to CA
Introscope. In WebSphere, all performance monitoring settings are off by default.
Follow these steps:
1. Configure a custom service for CA Introscope in WebSphere.
2. Enable PMI data collection in WebSphere.
3. Enable the reporting of PMI data in the IntroscopeAgent.profile.
After you have enabled PMI data collection in WebSphere, the PMI data can be
displayed as CA Introscope metrics. You can filter which metric categories to bring into
CA Introscope, depending on your needs.
Note: For WebSphere on z/OS, CA Technologies recommends that you use CA APM for
WebSphere z/OS. CA APM for WebSphere z/OS provides WebSphere-specific PBDs and
metrics. These PBDs and metrics use a low-overhead tracer technology to retrieve
WebSphere-specific metrics. CA APM for WebSphere z/OS does not require you to
enable PMI in WebSphere for z/OS. You can also enable PMI reporting in WebSphere for
z/OS, but this approach consumes more system resources.


Configure the reporting of WebSphere PMI metrics in Introscope
After you turn on Performance Monitoring Settings in WebSphere, you must enable PMI
data collection in Introscope, and enable the metric categories youd like to see
reported.
To configure the reporting of PMI metrics in Introscope
1. Shut down your managed application.
3. Locate the property, introscope.agent.pmi.enable, under the WebSphere PMI
Configurations section, and verify it is set to true.
4. Locate the following properties for high-level PMI metric categories:
introscope.agent.pmi.enable.threadPoolModule=true
introscope.agent.pmi.enable.servletSessionsModule=true
introscope.agent.pmi.enable.connectionPoolModule=true
introscope.agent.pmi.enable.beanModule=false
introscope.agent.pmi.enable.transactionModule=false
introscope.agent.pmi.enable.webAppModule=false
introscope.agent.pmi.enable.jvmRuntimeModule=false
introscope.agent.pmi.enable.systemModule=false
introscope.agent.pmi.enable.cacheModule=false
introscope.agent.pmi.enable.orbPerfModule=false
introscope.agent.pmi.enable.j2cModule=true
introscope.agent.pmi.enable.webServicesModule=false
introscope.agent.pmi.enable.wlmModule=false
introscope.agent.pmi.enable.wsgwModule=false
introscope.agent.pmi.enable.alarmManagerModule=false
introscope.agent.pmi.enable.hamanagerModule=false
introscope.agent.pmi.enable.objectPoolModule=false
introscope.agent.pmi.enable.schedulerModule=false
# introscope.agent.pmi.enable.jvmpiModule=false
Four categoriesthreadPool, servletSessions, connectionPool, and j2care set to
true by default. You can enable additional metric categories by setting the
appropriate properties to true or comment out categories to reduce the metrics
reported.
5. Save the changes and close the file.
6. Restart the managed application.
After youve enabled PMI collections in Introscope, available PMI metrics are displayed
under the WebSpherePMI node in the Investigator tree.


Configure the Reporting of Resource Metric Map Data for WebSphere
Various agents can report resource metric categories in the CA Introscope
Workstation. After you enable PMI data collection in WebSphere, you can configure the
application server to report the Resource Metric Map data.
Follow these steps:
1. Log in to the WebSphere console.
2. Select Monitoring and Tuning, Performance Monitoring Infrastructure.
3. Select the server you want to use for reporting, such as server1.
4. Click the Configuration tab, and select the Custom option.
5. Click the Custom link.
A configuration tree of options appears.
6. Select JDBC Connection Pools on the tree, and enable WaitingThreadCount on the
right pane.
7. Select ThreadPools on the tree, and enable ActiveCount on the right pane.
8. Save the configuration and restart the application server.
The application server is configured to report resource metrics.
Note: For more information about the Resource Metric Map data, see the CA APM
Enable cross-process tracing in WebSphere
Transaction trace sessions enable you to trace all of the operations that take place in a
transaction, including transactions that cross JVM boundaries on computers with
compatible JVM versions. Cross-process transaction tracing is supported for
synchronous transactions, for instance servlets to EJBs, and asynchronous transactions.
To enable cross-process transaction tracing in WebSphere
1. Create a custom service as described in Configuring a custom service in WebSphere
6.1 (see page 51).
2. Turn on the work area service.
From the administration page, Servers > Application servers, click on server1, click
on Business Process Services, click on Work Area Service, check the "Enable service
at server startup" box.
4. Locate and set introscope.agent.websphere.crossjvm property to true. For example:
introscope.agent.websphere.crossjvm=true


Logging considerations on WebSphere for z/OS
There are some things to consider when logging in a WebSphere z/OS environment. For
more information about Java Agent logging options, see Java Agent Monitoring and
Logging (see page 131).
Tagging log output as EBCDIC
WebSphere for z/OS changed its default encoding from EBCDIC CP1047 to ASCII
ISO8859-1. Because z/OS is normally an EBCDIC machine, any logging data written by
the Java Agent or AutoProbe must be tagged to use EBCDIC as the final output stream,
instead of ASCII.
To tag data as EBCDIC instead of ASCII
1. Open the IntroscopeAgent.profile, located in the <Agent_Home>\wily\core\config
directory.
2. Add these properties to the IntroscopeAgent.profile:
log4j.appender.console.encoding=IBM-1047
log4j.appender.logfile.encoding=IBM-1047
3. Save the IntroscopeAgent.profile.
Eliminate Startup Timing Issues With Logging Facilities
For WebSphere for z/OS, you can use a property to eliminate any startup timing window
exposures that can occur with the CA Introscope logging facilities.
Follow these steps:
1. Open the IntroscopeAgent.profile, located in the <Agent_Home>\wily\core\config
directory.
2. Add this property to the IntroscopeAgent.profile:
introscope.agent.logger.delay=100000
The value is in milliseconds, so the default delay in this example is 100 seconds.


Configure Oracle Application Server to use the Java agent
Oracle Application Server (OAS) uses one configuration file for the management of every
application, and therefore every JVM, that is managed by the Oracle Console. This file is
usually called opmn.xml.
To modify Oracle Application Server to use the Java agent
1. Shut down your application server and make a backup of opmn.xml.
2. Locate the section in the opmn.xml file for the application you want to instrument.
You will most likely want to instrument more than one application at this time.
3. Under <category id="start-parameters"> for the selected application, locate the
section called <data id="java-options".
4. Insert the following at the end of the java-options line, before the end "/> using the
appropriate path for your environment:
For example, the entire entry for one application would be on one line:
<data id="java-options" value="-server -XX:MaxPermSize=128M -ms512M -mx1024M
-XX:AppendRatio=3
-Djava.security.policy=$ORACLE_HOME/j2ee/home/config/java2.policy
-Djava.awt.headless=true -Dhttp.webdir.enable=false
-javaagent:$ORACLE_HOME/wily/Agent.jar
-Dcom.wily.introscope.agentProfile=$ORACLE_HOME/wily/core/config/IntroscopeAg
ent.profile/>
Configure GlassFish to Use the Java Agent
You can configure the Sun GlassFish open source application server project to use the
Java agent.
Note: For supported versions of Sun GlassFish, see the Compatibility Guide.
Follow these steps:
1. In GlassFish, navigate to the following location:
/appserver-install-dir/domains/domain1/config/


2. Open the domain.xml file in a text editor.
3. Add the full path for the Agent.jar file and the IntroscopeAgent.profile as JVM
options to the java-config element in the domain.xml file. For example:
<java-config ..........>
<jvm-options>-javaagent:/sw/wily/Agent.jar</jvm-options>
<jvm-options>-Dcom.wily.introscope.agentProfile=/sw/wily/core/config/Introsco
peAgent.profile</jvm-options>
.....>
4. Open the configuration properties file in a text editor and add wily properties.
For example, for Glassfish 3.2 the properties file is
<glassfish_home>/glassfish/config/osgi.properties.
The properties to edit are:
eclipselink.bootdelegation=oracle.sql, oracle.sql.*, com.wily.*
org.osgi.framework.bootdelegation=sun.*,com.sun.*,com.wily.*
5. Copy the Webappsupport.jar file from the <Agent_Home>/wily root directory to
the <Agent_Home>/wily/ext directory.
6. Start the application server.
Configure SAP Netweaver to Use the Java Agent
You configure JVM AutoProbe for SAP NetWeaver to use the Java agent.
Note: For supported versions of SAP NetWeaver, see the Compatibility Guide.
Follow these steps:
1. Start the SAP Configtool (configtool.bat).
2. Select the instance that you want to modify.
3. In the right pane, select VM Parameters > System.
4. Create a parameter (without providing -D). For example:
name: com.wily.introscope.agentProfile
value: <Agent_Home>/core/config/IntroscopeAgent.profile
5. Click the Additional tab and create a parameter, for example:
name: -javaagent:<Agent_Home>\Agent.jar
value: <leave empty>
6. Save your changes.
Configuring the connection to the Enterprise Manager


7. Restart the SAP server.
8. To verify that the changes made with the Configtool were made, open the following
file:
<drive>:\usr\sap\...\j2ee\cluster\instance.properties
9. Find the line beginning with ID<server_id>.JavaParameters, and confirm that it
contains the information you entered.
To report metrics, the agent must connect to an Enterprise Manager. The default
communication settings enable an agent to connect to a local Enterprise Manager using
port 5001. However, the agent and the Enterprise Manager do not typically reside on
the same system. You can modify the default settings when you install the agent, or
after installing the agent by modifying the IntroscopeAgent.profile.
Depending on your requirements, you can configure the communication between the
agent and Enterprise Manager to use:
Direct socket connections
HTTP tunneling connections or HTTP tunneling through a proxy server
HTTP over Secure Sockets Layer connections (HTTPS)
Secure Socket Layer (SSL) connections
Connect to the Enterprise Manager Using a Direct Socket Connection
The most common way for the agent to connect to the Enterprise Manager is through a
direct socket connection. CA Technologies recommends using a direct socket connection
to the Enterprise Manager if possible.
To configure a direct socket connection from the agent to the Enterprise Manager
2. Locate the introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT
property and specify the host name or IP address of the Enterprise Manager the
agent should connect to by default. For example:
introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=sfcollect01
If you use a cluster with more than one Enterprise Manager, be sure to specify a
Collector Enterprise Manager.
3. Set the introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT property
to the Enterprise Manager listening port. For example:
introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT=5001


4. Set the introscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT
property to the socket factory used for connections to the Enterprise Manager. For
example:
introscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT=com.wi
ly.isengard.postofficehub.link.net.DefaultSocketFactory
5. (Optional) Specify one or more backup Enterprise Managers for the agent to
connect to, if the connection to the primary Enterprise Manager is lost.
6. Save and close the IntroscopeAgent.profile file.
Connect to the Enterprise Manager with HTTP tunneling
If a direct socket connection to the Enterprise Manager is not feasible, you can configure
agents to connect to an Enterprise Manager over HTTP. This configuration allows
communication to pass through firewalls that only permit HTTP traffic.
Note: HTTP tunneling imposes more CPU and memory overhead on the application
server and Enterprise Manager than a direct socket connection.
To configure HTTP tunneling for an agent
2. Set the introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT property
to the host name or IP address of the Enterprise Manager the agent should connect
to by default. For example:
introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=webhost
to the HTTP listening port of the Enterprise Managers embedded web server. For
example:
This property should match the value specified in the
<EM_Home>/config/IntroscopeEnterpriseManager.properties file on the Enterprise
Manager for the introscope.enterprisemanager.webserver.port property. By default,
this port is 8081.
property to the HTTP tunneling socket factory. For example:
ly.isengard.postofficehub.link.net.HttpTunnelingSocketFactory


Configure a proxy server for HTTP tunneling
You can configure the HTTP tunneled agent to connect through a proxy server to the
Enterprise Manager. This configuration is necessary for a forwarding proxy server where
the agent is running behind a firewall that only allows outbound HTTP traffic routing
through the proxy server.
The proxy server configuration properties apply only if the agent is configured to tunnel
over HTTP. The proxy server configuration applies to any configured HTTP tunneled
connection on the agent, not to a single connection. This configuration is especially
important to consider when configuring failover between multiple Enterprise Managers,
where the connection to each Enterprise Manager is over HTTP.
Important! HTTP/1.1 is required to enable agent HTTP tunneling. In addition, the proxy
server must support HTTP Post.
Important! If the proxy is not reachable, the agent bypasses the proxy and makes a
direct connection with the Enterprise Manager. If the proxy is reachable but its
authentication fails, the agent keeps retrying to connect to the Enterprise Manager
through the proxy.
To configure a proxy server for HTTP tunneling
2. Set the introscope.agent.enterprisemanager.transport.http.proxy.host property to
the host name or IP address of the proxy server.
3. Set the introscope.agent.enterprisemanager.transport.http.proxy.port property to
the port number of the proxy server.
4. (Optional) If the proxy server requires user credentials for authentication, set the
following properties:
introscope.agent.enterprisemanager.transport.http.proxy.username=<user_name>
introscope.agent.enterprisemanager.transport.http.proxy.password=<user_passwo
rd>
Connect to the Enterprise Manager with HTTPS tunneling
The agent can connect to the Enterprise Manager using HTTP over Secure Sockets Layer
(SSL) by configuring properties in the IntroscopeAgent.profile file.
To configure a connection through HTTPS
to the host name or IP address of the target Enterprise Manager.


to the HTTPS listening port of the Enterprise Manager's embedded web server. For
example:
property to use an HTTPS tunneling socket factory. For example:
ly.isengard.postofficehub.link.net.HttpsTunnelingSocketFactory
Connect to the Enterprise Manager over SSL
If you are using direct Secure Socket Layer (SSL) connections, you can configure the
agent communication with the Enterprise Manager using SSL without HTTP tunneling.
To configure the agent to connect using SSL
2. Configure the agent to connect to the Enterprise Managers SSL listening port using
an SSL socket factory.
to the host name or IP address of the target Enterprise Manager.
to the Enterprise Manager's SSL listening port.
property to the SSL socket factory. For example, set the property to:
com.wily.isengard.postofficehub.link.net.SSLSocketFactory
6. If the agent needs to authenticate the Enterprise Manager, uncomment and set the
introscope.agent.enterprisemanager.transport.tcp.truststore.DEFAULT property to
the location of the truststore containing the Enterprise Managers certificate. If you
do not specify a truststore, the agent trusts all certificates.You can specify an
absolute path or a path relative to the agent profile. For example:
introscope.agent.enterprisemanager.transport.tcp.truststore.DEFAULT=/certs
7. Set the introscope.agent.enterprisemanager.transport.tcp.trustpassword.DEFAULT
property to specify the truststore password, if needed.
8. If the Enterprise Manager requires client authentication, set the
introscope.agent.enterprisemanager.transport.tcp.keystore.DEFAULT property to
the location of a keystore containing the agent's certificate. You can specify an
absolute path or a path relative to the agent profile. On Windows, backslashes must
be escaped. For example:
introscope.agent.enterprisemanager.transport.tcp.keystore.DEFAULT=C:\\keystor
e
Upgrade Multiple Agent Types


9. Set the introscope.agent.enterprisemanager.transport.tcp.keypassword.DEFAULT
property to the keystore password, if needed.
10. Set the introscope.agent.enterprisemanager.transport.tcp.ciphersuites.DEFAULT
property to a comma-separated list of allowed cipher suites.
If you do not specify a value for this property, the default cipher suites are used.
Configure Agent Load Balancing
In clusters where the workload is primarily metrics reported by agents, you can optimize
the overall cluster capacity by configuring MOM agent load balancing.
Note: For more information about MOM agent load balancing, see the CA APM
Upgrade Multiple Agent Types
Some environments have thousands of agents that are distributed across many different
application servers. For example, an environment can have 8,000 agents, with 3,000
agents on WebLogic, 2,000 on WebSphere, and 3,000 on JBoss. To make it easier to
upgrade agents across multiple application servers, you can use an agent superset
package. Agent superset packages are operating system-specific packages that contains
files for all supported application servers except SAP NetWeaver.
The following agent superset packages are available:
IntroscopeAgentFiles-NoInstaller<version>allappserver.windows.zip
IntroscopeAgentFiles-NoInstaller<version>allappserver.unix.tar
IntroscopeAgentFiles-NoInstaller<version>allappserver.zOS.tar
IntroscopeAgentFiles-NoInstaller<version>allappserver.os400.zip
Note: In these file names, <version> refers to the current release number of the Java
agent.
Each package contains the following files:
All application server-specific PBDs and PBLs
Uninstalling the Java agent


All application server agent profiles, with the application server name embedded in
the file name. For example:
IntroscopeAgent.weblogic.profile
IntroscopeAgent.websphere.profile
Note: The default IntroscopeAgent.profile is not included, see Step 3 for more
information.
All agent JAR files and platform monitors suitable for the operating system type
Follow these steps:
1. Select a superset package appropriate for the target operating system.
2. Extract the selected agent package into the home directory of the application
server.
Note: You can ignore the extra PBDs and PBLs in the <Agent_Home>/core/config
directory that refer to application servers you are not using.
3. Perform the following steps if you have not already configured an
IntroscopeAgent.profile:
a. Select the appropriate IntroscopeAgent.<application_server_name>.profile.
b. Rename it to IntroscopeAgent.profile.
c. Configure the file for use with your environment.
4. Perform the following steps If you have already configured an
a. Open the corresponding IntroscopeAgent.<application_server_name>.profile
file in an editor.
b. Look for new properties that you want to use.
c. Transfer these properties to your existing IntroscopeAgent.profile.
More information:
Install Manually Using Installation Archives (see page 30)

Uninstalling the Java agent requires you to know where the Java agent was installed for
each application being monitored.
If you used the Java agent installer to install the Java agent, you can use the uninstall
program to remove the installed agent files. Launch the uninstaller and follow the
on-screen directions.


To uninstall the Java agent from any supported JVM
1. Stop the application server.
Note: Monitored JVMs must be shut down before you run the uninstall program.
2. Open the application server startup script and remove the Java agent switches from
the Java command line. Depending on the application server, the startup options
include the following:
-Xbootclasspath
-javaagent:<path_to_the_agent_jar>
-Dcom.wily.introscope.agentProfile=<path_to_the_agent_profile>
3. Reboot the application server.
4. Run the Uninstall_Introscope_Agent program located in
<Agent_Home>/UnstallerData directory.
5. Manually delete the <Agent_Home> directory.
Uninstalling the Java agent from z/OS
The recommended way to uninstall the Java agent from z/OS is to delete the
<Agent_Home> directory using an rm -rf command. This is necessary because the
executable uninstaller does not run properly on z/OS due to a third party bug.
Important! Only remove files after shutting down the monitored JVM.
For an active Introscope installation on z/OS, it is important to keep the UninstallerData
folder intact. If you delete the UninstallerData folder, you will not be able to upgrade to
future versions of Introscope. Do not delete the UninstallerData folder unless you have
decided to uninstall the entire instance.


Chapter 3: Configuring Agent Properties

Most agent operations are configured using properties in the IntroscopeAgent.profile
file. This section describes the most common agent properties to set. Additional
properties may be appropriate for your environment. Different versions of the agent
may have different properties available for you to set or different default values.
How to modify communication with Enterprise Manager (see page 67)
How to configure backup Enterprise Managers and failover properties (see page 67)
How to enable and use additional GC metrics (see page 69)
How to enable and configure thread dumps (see page 69)
How to Configure an Agent to Collect Distribution Statistic Metrics (see page 72)
How to modify communication with Enterprise Manager
To report metrics, the agent must connect to an Enterprise Manager. After installing the
agent, you can modify the communication channel used by modifying the
IntroscopeAgent.profile.
Depending on your requirements, you can configure the communication between the
agent and Enterprise Manager to use:
Direct socket connections
HTTP tunneling connections or HTTP tunneling through a proxy server
HTTP over Secure Sockets Layer connections (HTTPS)
Secure Socket Layer (SSL) connections
The most common way for the agent to connect to the Enterprise Manager is through a
direct socket connection. CA Technologies recommends using a direct socket connection
to the Enterprise Manager, if possible. If you want to use a different type of connection,
see the appropriate section.
How to configure backup Enterprise Managers and failover
properties
When you install the agent, you specify the Enterprise Manager host name and port
number that the agent connect to by default. Optionally, you can also specify one or
more backup Enterprise Managers. If the agent loses the connection to its primary
Enterprise Manager, it can then attempt to connect to a backup Enterprise Manager.
How to configure backup Enterprise Managers and failover properties


To enable an agent to connect to a backup Enterprise Manager, you specify the
communication properties for the Enterprise Managers in the agent profile. If the
primary Enterprise Manager is not available, the agent tries to connect to the next
Enterprise Manager on its list of allowed connections. If the agent does not connect
with its first backup Enterprise Manager, it tries the next Enterprise Manager in the list.
The process continues trying to connect to each Enterprise Managers in the list, in
order, until it succeeds in connecting. If the agent is unable to connect to any of its
Enterprise Managers, it waits 10 seconds before retrying.
Follow these steps:
1. Open the Introscope.Agent.profile file in a text editor.
2. Specify one or more alternative Enterprise Manager communication channels by
adding the following properties to the agent profile for each backup Enterprise
Manager:
introscope.agent.enterprisemanager.transport.tcp.host.NAME
introscope.agent.enterprisemanager.transport.tcp.port.NAME
Introscope.agent.enterprisemanager.transport.tcp.socketfactory.NAME
Replace NAME with an identifier for the new Enterprise Manager channel. Do not
use DEFAULT or the name of an existing channel when creating a channel. For
example, to create two backup Enterprise Managers:
introscope.agent.enterprisemanager.transport.tcp.host.BackupEM1=paris
introscope.agent.enterprisemanager.transport.tcp.port.BackupEM1=5001
Introscope.agent.enterprisemanager.transport.tcp.socketfactory.BackupEM1=com.
custom.postofficehub.link.net.DefaultSocketFactory
introscope.agent.enterprisemanager.transport.tcp.host.BackupEM2=voyager
introscope.agent.enterprisemanager.transport.tcp.port.BackupEM2=5002
introscope.agent.enterprisemanager.transport.tcp.socketfactory.BackupEM2=com.
wily.isengard.postofficehub.link.net.DefaultSocketFactory
3. Locate the introscope.agent.enterprisemanager.connectionorder property and set it
to a comma-separated list of identifiers for the primary and backup Enterprise
Managers. The order in which you list the identifiers defines the order in which they
are contacted. For example:
introscope.agent.enterprisemanager.connectionorder=DEFAULT,BackupEM1,BackupEM
2
4. Locate the introscope.agent.enterprisemanager.failbackRetryIntervalInSeconds
property and specify how frequently the agent tries connecting to its primary
Enterprise Manager. The default interval is 120 seconds (2 minutes). For example:
introscope.agent.enterprisemanager.failbackRetryIntervalInSeconds=120
5. Save the changes and close the IntroscopeAgent.profile file.
6. Restart the application.
How to enable and use additional GC metrics


How to enable and use additional GC metrics
Garbage collection and memory management can have a significant effect on an
applications performance. Basic GC Heap metrics are available by default. There are also
optional metrics that you can enable to provide additional details about garbage
collection processing and memory pool usage. These additional metrics display under
the GC Monitor node in the Investigator when enabled. The GC Monitor metrics report
information to help you optimize memory pool allocation and garbage collection
processing. Therefore, these metrics are typically enabled when developing or testing
applications or when researching application performance issues. In most cases, the
metrics are not used for real-time application management in a production environment
and are disabled by default.
If you want to enable the GC Monitor metrics, you can open the agent profile in a text
editor and set the introscope.agent.gcmonitor.enable property to true. After you enable
the property, you can view details about the Garbage Collectors and Memory Pools for
the JVM you are monitoring. For more information about the metrics, see the CA APM
Workstation User Guide.
How to enable and configure thread dumps
Thread dumps can provide useful detailed information about what is happening within
the agent JVM. Thread dump functionality is provided in the Thread Dumps tab
associated with each agent node in the metric browser tree.
For information about collecting and analyzing thread dumps, see the CA APM
Workstation User Guide. Setting the Thread_Dump permission allows users to see the
Thread Dumps tab and use all the functionality. For more information, see the CA APM
Security Guide.
Both IntroscopeAgent.profile and IntroscopeEnterpriseManager.properties properties
are required to enable thread dumps. By default, the Thread Dumps tab and its
functionality are enabled. However, if either or both of the thread dump enable
properties are set to false, then users cannot see the Thread Dumps tab.
If you enable or disable thread dumps on a MOM, that configuration applies to all the
Collectors in the cluster. Therefore, if you disable thread dumps on a MOM, thread
dumps are disabled on all the Collectors too.
To enable thread dumps
1. Open the IntroscopeAgent.profile file in the <Agent_Home>/wily/core/config
directory and set this property:
introscope.agent.threaddump.enable=true
2. Save and close IntroscopeAgent.profile.



3. Open the IntroscopeEnterpriseManager.properties file located in the
<EM_Home>/config directory and set this property:
introscope.enterprisemanager.threaddump.enable=true
4. Save and close IntroscopeEnterpriseManager.properties.
For CA Introscope users to view the Deadlock Count metric, configure the
IntroscopeAgent.profile. You can perform additional configuration to display agent
Thread node metrics.
To enable Deadlock Count metric collection
directory.
2. Set this property to true to enable the Deadlock Count metric to be collected.
introscope.agent.threaddump.deadlockpoller.enable=true
3. (Optional) Set the full version of the PBL to display metrics in the agent Threads
node.
Specify the name of the PBL file in this property:
introscope.autoprobe.directivesFile.
For example, to use the full version of the standard PBL for WebLogic server,
set the property to:
introscope.autoprobe.directivesFile=weblogic-full.pbl
User can see metrics such as Active Threads counts and thread groups under
AgentName | Threads.
4. Save and close IntroscopeAgent.profile.
Configure thread dumps using both IntroscopeAgent.profile and
IntroscopeEnterpriseManager.properties properties.
To configure thread dumps
1. Open the IntroscopeEnterpriseManager.properties file located in the
<EM_Home>/config directory.
2. (Optional) Set this property to save thread dump files to a specific directory on the
Enterprise Manager. For example, TestThreadDumps.
introscope.enterprisemanager.threaddump.storage.dir=TestThreadDumps
3. (Optional) Set this property to purge thread dump files older than a specified
number of days. For example, older than 30 days.
introscope.enterprisemanager.threaddump.storage.clean.disk.olderthan.days=30
4. (Optional) Set this property to purge thread dump files after a specified number of
days. For example, after every two days.
introscope.enterprisemanager.threaddump.storage.clean.disk.freq.days=2


5. (Optional) Set this property to limit the maximum number of thread dump files that
can be stored on the Enterprise Manager. For example, 5000 files.
introscope.enterprisemanager.threaddump.storage.max.disk.usage=5000
Note: If:
* the number of thread dump files stored exceeds the limit set in the
introscope.enterprisemanager.threaddump.storage.max.disk.usage property
and
* there are no files older than the number of days set in the
introscope.enterprisemanager.threaddump.storage.clean.disk.olderthan.days
property
Then the Enterprise Manager does not store any thread dump files.
6. Save and close IntroscopeEnterpriseManager.properties.
7. Restart the Enterprise Manager.
If an Enterprise Manager goes down, you can copy thread dump files to another
Enterprise Manager so users can view thread dump data.
Important! Restart the Enterprise Manager when you add files to or remove files from
the thread dump directory. CA Technologies does not recommend moving thread dump
files from one Enterprise Manager to another.
To copy thread dump files from one Enterprise Manager to another
1. Navigate to the <EM_Home>/threaddumps directory on the Enterprise Manager
containing thread dump files (EM1).
2. Copy the thread dump files.
3. Paste the files into the <EM_Home>/threaddumps directory on the Enterprise
Manager where users are to view the thread dumps (EM2).
4. Restart both Enterprise Managers EM1 and EM2.
5. If needed, establish the agent connection and enable and configure thread dumps
on EM2.
EM2 users can select the agent node then click the Load Previous button in Thread
Dumps tab. A list displays the thread dumps moved from EM1.
How to Configure an Agent to Collect Distribution Statistic Metrics


How to Configure an Agent to Collect Distribution Statistic
Metrics
You can configure an agent to collect the distribution of response times analyzed by
BlamePointTracers to create Average Response Time metrics. Distribution statistic
metrics provide detailed descriptions that explain how specific operations vary. The
additional statistical distribution data of monitored response time values is available in
the getExtendedMetricData web service.
The CA APM agent can be configured to collect response time information for specific
operations. The response times are stored in a distribution statistics metric, which is
paired with the Average Response Time metric for the selected operations.
Follow these steps:
directory.
2. Specify the Average Response Time metrics for which you want to create paired
Distribution Statistics metrics by uncommenting and editing the
introscope.agent.distribution.statistics.components.pattern. For example, use this
expression:
Servlets\\|LoginServlet:.*
Distribution statistics for LoginServlet response times are generated.
3. (Optional) Create your expression using these guidelines:
a. Precede metric node separators with backslashes because the vertical bar and
period have special meanings within regular expressions.
b. Precede backslashes with another backslash because backslashes have special
meaning within the agent profile.
The regular expressions appear in the agent log after the special characters. For
example:
Servlets\|Login Servlet:.*
c. Match the summary level and individual metrics regular expressions. If the
metrics do not match, then the distribution statistics for summary level are
summarized or not created.
The following examples show expressions that match:
Servlets(\\|.*):.*
Servlets.*
5. Restart the agent.


Examples of Distribution Statistics Metrics
Depending on how you set the configuration property, distribution statistics can be
collected for:
Example of Servlet and JSP Distribution Metrics for Individual and Summary Level
(see page 73)
Example of Servlet and JSP Distribution Metrics (see page 74)
Example of Servlet and JSP Distribution Metrics for Individual and Summary Level
To collect distribution statistics for individual and summary levels for Servlets and JSPs,
use this pattern:
introscope.agent.distribution.statistics.components.pattern=(Servlets|JSP)(\\|.*)
:.*
The following image shows the Investigator collecting distribution statistics metrics for
the servlet and summary levels for Java agent nodes:



Example of Servlet and JSP Distribution Metrics
To collect distribution statistics for the Servlet and JSP levels but not Servlet or JSP
summary levels, use this pattern:
introscope.agent.distribution.statistics.components.pattern=(Servlets|JSP)\\|.*
The following image shows the Investigator collecting distribution statistics metrics for
the servlet, but not the summary levels for Java agent nodes:


Chapter 4: AutoProbe and ProbeBuilding
Options

AutoProbe and ProbeBuilding Overview (see page 75)
Configuring ProbeBuilding (see page 76)
AutoProbe and ProbeBuilding Overview
The Java agent inserts probes into the bytecode of applications you want to monitor.
ProbeBuilding is how you decide which probes to insert into applications using
ProbeBuilder Directives (PBDs) and ProbeBuilder Lists (PBLs). The default PBD and PBL
files included with the Java agent provide a basic level of metric collection. Modify the
default settings in these files to better tune metric collection specifically for your
environment.
You configure PBDs and PBLs to insert the probes for the metrics you want to collect in
your applications and environment. Then you use these files to instrument applications
automatically using JVM AutoProbe or manually using ProbeBuilder. Use JVM
AutoProbe to instrument applications. Configuring JVM AutoProbe, however, depends
on the application environment.
Note: For supported versions of JVMs, see the Compatibility Guide.
To instrument your applications on JVMs, use one of the following methods:
JVM AutoProbe, with -javaagent property. CA Technologies highly recommends
using JVM AutoProbe to instrument applications on JVMs.
Manual ProbeBuilding is an advanced instrumentation technique. Contact CA
Support before using this method.
Important! When instrumenting your applications, use only one method of
instrumentation.
Verify that you have installed and configured the Java Agent (see page 23).
Configuring ProbeBuilding


Unsupported Instrumentation Method
The AutoProbe for Application Servers method for instrumentation is not supported for
Java agents. You can use AutoProbe for Application Servers to instrument applications
that use older versions of the JVM (1.4 and earlier) with earlier versions of the Java
agent. CA Technologies recommends that you use JVM AutoProbe for applications that
use JVM 1.5 or later.
Note: For supported versions of the JVM, see the Compatibility Guide. Unless otherwise
indicated in the Compatibility Guide, the JVM supported is the JVM that the application
server version is shipped with.
ProbeBuilding technology performs the instrumenting process. Probes that are defined
in the ProbeBuilder Directive (PBD) files identify the metrics that an agent gathers from
web applications and virtual machines at runtime.
By default, AutoProbe uses the typical PBD that is set provided with the Java agent. This
setup results in the collection of a moderate number of metrics. The following options
let you customize the metric collection level and configure ProbeBuilding behaviors.
Full or typical tracing options (see page 76)
Dynamic ProbeBuilding (see page 77)
ProbeBuilding class hierarchies (see page 81)
Removing line numbers in bytecode (see page 84)
Full or typical tracing options
In Introscope, ProbeBuilder List (PBL) files govern which tracer groups are used in the
instrumentation process. The introscope.autoprobe.directivesFile property specifies one
or more PBL files.
Introscope provides two versions of each default PBLa full version which enables a
larger set of Tracer Groups than the typical version which results in more detailed
metric reporting, and a typical version that enables a smaller set of Tracer Groups. This
results in less detailed metric reporting and reduced overhead. By default,
introscope.autoprobe.directivesFile specifies the typical version of the default PBL file.


To change the tracing level between full and typical
1. Stop the managed application.
3. Specify the name of the PBL file you want to use in this property:
introscope.autoprobe.directivesFile.
For example, to use the Full version of the standard PBL for WebLogic Server, set
the property to:
introscope.autoprobe.directivesFile=weblogic-full.pbl
Dynamic ProbeBuilding
CA Introscope uses dynamic ProbeBuilding to implement new and changed PBDs
without restarting managed applications or the agent. Dynamic ProbeBuilding is useful
for making corrections to PBDs, or to change data collection levels during triage without
interrupting application service.
Important! Dynamic ProbeBuilding is only available for use with Java 1.5 at a minimum.
Dynamic ProbeBuilding is dependent on Java 1.5 capabilities and on the -javaagent
command.
Note: The Workstation allows you to perform dynamic instrumentation through the
Transaction Trace viewer. For more information, see the CA APM Workstation User
Guide.
Dynamic ProbeBuilding causes CA Introscope to look periodically for new PBDs and for
changed PBDs. To minimize overhead, CA Introscope selectively reinstruments classes
that the modified PBDs affect. To improve performance, the scope of dynamic agent
reinstrumentation is limited to reloading only those classes whose instrumentation has
changed when PBDs were edited.
When you edit a PBD or add a PBD to the hotdeploy directory, only user directives (such
as adding or removing directives for a class, or toggling tracer groups) are
reinstrumented.
Important! Only changes to directives using tracer groups are supported, for example,
changes in any directive like TraceAllMethods that have an IfFlagged switch. However,
CA Introscope only ships any out of the box directives with tracer groups or flags.
Changes to skips or transformations are not supported.
The following directives are not reinstrumented:
System directives such as adding a tracer or changing a new tracer mapping.
Arrays, interfaces, and classes specified in Skip directives, and any transformations.


You can configure the reinstrumentation process to:
Exclude all classes that particular classloaders load.
Limit the scope to specific class packages.
Note: Dynamic ProbeBuilding is not enabled by default.
If a class is reinstrumented so that it no longer reports data for a metric, the metric is
still displayed in the Investigator. Existing metrics do not disappear from the Investigator
window when their classes are reinstrumented.
Important! Due to a limitation in Java 1.5, access is not available to some class bytes,
with the following effects:
Modifications to the j2ee.pbd file may not be picked up, and metrics can continue
to be published under old names.
Some exceptions can appear in the agent log.
To avoid this issue, restart the application server after modifying the j2ee.pbd file.
When configuring dynamic ProbeBuilding, CA Technologies recommends that you base
your changes on tracer groups.
Example: Control the Level of Instrumentation for the Tracer Group XYZ.
This example demonstrates how you can control the level of instrumentation for a
tracer group.
Follow these steps:
1. Create two tracer groups:
XYZTracingregular tracing options
XYZTracingLitefewer components are traced
2. Switch between them: Turn off XYZTracing and turn on XYZTracingLite.
3. View the impact that dynamic ProbeBuilding has on your environmental
performance.
4. Adjust the tracing groups accordingly.
Adjustment affects all classes being traced as part of each tracer group.


Configure Dynamic ProbeBuilding
To configure dynamic ProbeBuilding, you edit the IntroscopeAgent.profile.
Follow these steps:
1. Navigate to the <Agent_Home>/wily/core/config directory.
3. Verify that the property, introscope.autoprobe.enable, is set to true.
4. Uncomment and set the following properties:
introscope.autoprobe.dynamicinstrument.enabled=true
This property enables dynamic ProbeBuilding. This property takes effect when
you restart the managed application.
introscope.autoprobe.dynamicinstrument.pollIntervalMinutes=1
The polling interval in minutes to check for PBD changes. The default is set to
one-minute intervals. This property takes effect when you restart the managed
application.
introscope.autoprobe.dynamicinstrument.classFileSizeLimitInMegs=1
Some classloader implementations have been observed to return huge class
files. This behavior prevents memory errors. This property takes effect when
you restart the managed application.
introscope.autoprobe.dynamic.limitRedefinedClassesPerBatchTo=10
Too many classes redefinitions at a time can be very CPU intensive. When
changes in PBDs trigger a redefinition of many classes, this property attempts
to batch the process at a comfortable rate.
5. Save changes to the IntroscopeAgent.profile and close the file.
6. Restart the managed applications (if appropriate).


Dynamic Instrumentation Impacts Performance for IBM JDK
Valid for: CA Introscope with IBM JDK version 5
Note: No performance overhead exists with the use of class redefinition when using CA
Introscope with IBM JDK version 6.
Symptom:
Dynamic instrumentation requires class redefinition support. Use of class redefinition
can significantly impact performance. IBM provides technical information about this
performance overhead in the Java Diagnostics Guide. CA Introscope and IBM JDK
customers that want to take advantage of dynamic instrumentation should keep in mind
this performance overhead.
Solution:
CA Technologies recommends using the dynamic instrumentation feature only in a
production environment.
More information:
Configure WebSphere Application Server 6.1 on UNIX, Windows, OS/400, z/OS, IBM JVM
1.5 (see page 47)
Socket metrics (see page 132)

Dynamic ProbeBuilding Versus Dynamic Instrumentation
Dynamic ProbeBuilding and dynamic instrumentation are different:
Dynamic ProbeBuilding (see page 77) is based on manual changes you make to PBD
files and manual configurations you make in the IntroscopeAgent.profile. If you
update or change a PBD file and save it in the correct location, dynamic
ProbeBuilding implements the changes. Dynamic ProbeBuilding requires you to
configure the IntroscopeAgent.profile, and change the PBD files you want updated.
All changes are permanent (until you manually update or change the files again).
Dynamic instrumentation is performed from the Workstation transaction trace
viewer. The changes to instrumentation you select using the interface are made
automatically for you, and are often only temporary. Instrumenting a method
dynamically means inserting the instrumentation during runtime. You can
dynamically instrument one, more, or all of the methods during a transaction trace
session. Then, you can view metrics the newly instrumented methods return. This
method lets you do dynamic application performance tuning. Dynamic
instrumentation does not require any changes to the IntroscopeAgent.profile. If you
decide to make instrumentation changes permanent, a PBD is created and saved in
the correct location for you.


Note: For more information about how to use dynamic instrumentation from the
Workstation transaction trace viewer, see the CA APM Workstation User Guide.
Important! CA Introscope stores a dynamic instrumentation cache on the
application server host computer. For dynamic instrumentation to work, the user
account which the application server uses to access the Enterprise Manager must
have write access to the <Agent_Home> and <Agent_Home>/logs directories on the
application server host computer.
ProbeBuilding Class Hierarchies
For JVMs earlier than 1.5, CA Introscope does not automatically instrument classes in
the deeper levels of class hierarchy. CA Introscope only instruments the classes that
explicitly extend a probed class.
With a supported JVM, you can configure CA Introscope to instrument multiple
subclass levels of a probed class. The product updates the tracer groups in the
associated internal directive appropriately and dynamically instruments the classes. The
directive changes are written to a log file.
Note: For information about supported JVMs, see the Compatibility Guide.
If you prefer to update your PBDs manually, you can disable directive updates and you
can use the log file to determine appropriate updates.
More information:
Instrumenting and Inheritance (see page 115)

Enable instrumentation of multiple levels of subclasses
Follow these steps to configure Introscope to dynamically update internal directives.
To enable instrumentation of multiple levels of subclasses
1. Verify that dynamic instrumentation is enabled as described in Dynamic
ProbeBuilding (see page 77).
2. Open the IntroscopeAgent.profile.
3. To enable instrumentation of multiple levels of subclasses, uncomment this
property setting:
introscope.autoprobe.hierarchysupport.enabled=true


Support for Multiple Inheritance, Interfaces, and Abstract Methods
The Java agent supports instrumentation by interface and by inheritance. This ability
extends to dynamic instrumentation.
The Java agent supports the instrumentation of methods through a call to subclasses by
using the API getMethodCalls. The getMethodCalls lets you better understand the
consequences of instrumentation of inherited methods or interface methods by
providing the following information:
If the class defining the method is an interface.
The number of classes that a possible instrumentation of the method affects. This
number is the number of subclasses or the number of implementing classes.
If the method is called within a specific stack trace.

A tracer to instrument interfaces and abstract methods is available with the Java Agent,
using the following syntax:
TraceOneMethodWithlabelIfInherits: <class> <method> <Label> <Tracer Group> <Tracer
Type> <Resource>
This tracer instruments a method of any class implementing interface or extending
superclass in the following cases:
If the method is defined in the interface.
If the method is abstract in the superclass.
Important! Using this tracer could have a heavy impact on the performance of your
system. Test the impact of this tracer during your system startup and instrumentation
processes before deploying to a larger agent configuration.
More information:
Creating custom tracers (see page 106)



Configure periodic polling for uninstrumented subclasses
When multi-level subclass instrumentation is enabled, Introscope will check for
uninstrumented subclasses at application startup.
To configure Introscope to poll for uninstrumented subclasses
2. Uncomment this property setting:
introscope.autoprobe.hierarchysupport.runOnceOnly=false
3. To change the frequency with which Introscope polls for uninstrumented subclasses
from its default value of 5, uncomment this property and set it to the desired
polling frequency:
introscope.autoprobe.hierarchysupport.pollIntervalMinutes
4. Optionally, you can limit the number of times Introscope polls uninstrumented
subclasses by uncommenting this property and setting it to the desired limit:
introscope.autoprobe.hierarchysupport.executionCount
The default of this property is 3 minutes.
Disable directive updates
If multi-level subclass instrumentation is enabled, when Introscope detects
uninstrumented subclasses, by default, it updates internal directives appropriately to
ensure the classes are instrumented. If you prefer to update PBDs manually, you can
disable internal directive updates by uncommenting this property in the
introscope.autoprobe.hierarchysupport.disableDirectivesChange=true


Controlling directive logging
When multi-level subclass instrumentation is enabled, you must uncomment the
following properties in the IntroscopeAgent.profile to have multi-level subclass
instrumentation logs created. When these properties are configured, a log file named
pbdupdate.log is created in the <Agent_Home>/wily directory (by default), or in the
custom location (if specified). The multi-level instrumentation details are written to the
agent logs.
log4j.additivity.IntroscopeAgent.inheritance=false
log4j.logger.IntroscopeAgent.inheritance=INFO,pbdlog
log4j.appender.pbdlog.File=pbdupdate.log
log4j.appender.pbdlog=com.wily.introscope.agent.AutoNamingRollingFileAppender
log4j.appender.pbdlog.layout=com.wily.org.apache.log4j.PatternLayout
log4j.appender.pbdlog.layout.ConversionPattern=%d{M/dd/yy hh:mm:ss a z} [%-3p] [%c]
%m%n_
You must restart the managed application before changes to these properties take
effect.
Removing line numbers in bytecode
When you instrument application bytecode, the bytecode line numbers are preserved
by default. Preserving bytecode line number information is helpful when using
debuggers or when obtaining stack trace information.
You can turn off this feature by adding a system property on the Java command line.
Turning off this feature removes all line numbers when AutoProbe or ProbeBuilder
instruments the application code.
To remove line numbers in bytecode when using AutoProbe or ProbeBuilder
Define this system property on the Java command line with the -D option:
com.wily.probebuilder.removeLineNumbers=true


Chapter 5: ProbeBuilder Directives

This section describes how to create and modify ProbeBuilder Directives.
ProbeBuilder Directives Overview (see page 85)
Using the IntroscopeAgent.profile, PBLs, and PBDs together (see page 103)
Applying ProbeBuilder Directives (see page 103)
Using Blame Tracers to mark blame points (see page 117)
ProbeBuilder Directives Overview
ProbeBuilder Directive (PBD) files tell the Introscope ProbeBuilder how to add probes,
such as timers and counters, to instrument an application. PBD files govern what
metrics your agents report to the Introscope Enterprise Manager.
Note: All metrics are calculated using the time set by your system clock. If the system
clock is reset during a transaction, the elapsed time reported for that transaction can be
misleading.
Introscope includes a set of default PBD files. You can also create custom Introscope
PBD files to track any classes or methods to obtain specific information about your
applications.
Two kinds of files specify ProbeBuilder Directives:
ProbeBuilder Directive (PBD) files
A ProbeBuilder Directive (PBD) file contains directives used by ProbeBuilder to
instrument your applications. This file determines which metrics the agents report
to the Enterprise Manager.
ProbeBuilder List (PBL) files
A ProbeBuilder List (PBL) file contains a list of multiple PBD filenames. Different PBL
files can refer to the same PBD files.
Important! PBDs and PBLs only support ASCII characters. Placing other characters (such
as Unicode characters) in PBDs or PBLs could result in problems with AutoProbe.
When you install the Java agent and you are using Introscope AutoProbe, the relevant
PBD and PBL files for your specific application server are included. These files are
located in the <Agent_Home>\wily\core\config directory.


More information:
Default PBD Files (see page 87)
Default PBL files (see page 90)

Components traced by the default PBDs
The default Introscope PBD files implement tracing of the following Java components:
Oracle JDBC
JSP Tag Libraries
JSP IO Tag Libraries
JSP DB Tag Libraries
Struts
Servlets
JavaServer Faces (JSF)
JavaServer Pages (JSP)
Enterprise JavaBeans (EJBs)
Java Database Connectivity (JDBC)
Network Sockets
Remote Method Invocation (RMI)
Extensible Markup Language (XML)
Java Transaction API (JTA)
Java Naming and Directory Interface (JNDI)
Java Message Service (JMS)
Common Object Request Broker Architecture (CORBA)
User Datagram Protocol (UDP)
File Systems
Threads
System Logs
Thrown and Caught Exceptions (off by default)


Occasionally, too many Java classes are selected for monitoring in a ProbeBuilder
Directive (PBD) file, causing the Java agent to start incorrectly, or to experience a
"hang". If this happens, use the AutoProbe.log file to identify the classes that caused the
Java agent to hang and add a skip directive to the PBD file, skipping the classes that may
have caused the problem. For more information about adding skip directives to your
custom PBD files, see Skip directives (see page 77).
Default PBD Files
The Java agent includes the following default PBD files:
appmap.pbd
Provides tracer directives for application triage map instrumentation.
appmap-ejb.pbd
Provides tracer directives for application triage map EJB instrumentation.
appmap-soa.pbd
Provides tracer directives for application triage map SOA instrumentation for SPM
supported Java SOAP stacks.
Note: For more information, see the CA APM for SOA Implementation Guide.
bizrecording.pbd
Provides tracer definitions and directives to setup agent business recording.
biz-trx-http.pbd
Provides tracer directives for business-centric HTTP instrumentation.
di.pbd
Provides directives ProbeBuilder to not instrument Apache Derby implementation
classes.
errors.pbd
Configures Error Detector by specifying what code-level events constitute serious
errors. By default, only front-end and back-end errors are considered serious. That
is, only errors that manifest as a user-facing error page or that indicate a problem
with a back-end system (ADO.NET, Messaging, and so on).
j2ee.pbd
Provides tracer groups for common Java Enterprise Edition components. Use either
toggles-full.pbd or toggles-typical.pbd to TurnOn specific tracing.
java2.pbd
Provides tracer groups for common Java 2 components. Use either toggles-full.pbd
or toggles-typical.pbd to TurnOn specific tracing.


jsf.pbd
Provides tracer groups for Java Server Face (JSF) components.
jsf-toggles-full.pbd
Provides on/off switches in the form of TurnOn directives for the tracing provided in
the jsf.pbd. Most tracer groups are turned on.
jsf-toggles-typical.pbd
the jsf.pbd.
jvm.pbd
Provides directives which implement support for various Java Virtual Machines. You
use this property with the Introscope default files.
leakhunter.pbd
Provides instrumentation settings for CA APM LeakHunter, a leak detection utility.
Typically, you do not modify the contents of this file.
lisa.pbd
Provides tracer directives for CA LISA instrumentation for the CA APM integration.
oraclejdbc.pbd
Provides tracer groups for Oracle JDBC components. Comment or uncomment the
TurnOn directives to alter the set of Oracle JDBC components that are traced.
ServletHeaderDecorator.pbd
Enables the servlet header decorator, which is part of the integration with CA CEM.
smwebagentext.pbd
Provides tracers for the SiteMinder Web Agent Introscope plug-in.
soaagent.pbd
Provides tracers for TransactionMinder Agent, part of the CA SOA Security Manager
(SOA Agent for Web Server and Application Server).
spm-correlation.pbd
Provides directives that control the correlation of transaction traces across
components. This file is required to enable cross-process transaction tracing when
you use CA APM for SOA.

struts.pbd
Provides directives which monitor Apache struts. Use this property with the
Introscope default files.
summary-metrics-6.1.pbd
Provides directives necessary for JSP tracing, Servlet tracing, and EJB tracing in
pre-7.0 Introscope instances.


taglibs.pbd
Provides directives that monitor classes that should be traced as JSP tag libraries,
Jakarta I/O libraries, and DGTags tag libraries.
TIBCO pbd
The Java agent is installed with several PBDs related to monitoring TIBCO through
the SOA extension.
toggles-full.pbd
other directives files. Most tracer groups are turned on.
toggles-typical.pbd
other directives files. Only a small section of tracer groups is turned on.
webMethods pbds
The Java agent is installed with several PBDs related to monitoring webMethods
through the CA APM for webMethods Broker.
WebSphere MQ pbds
The Java agent is installed with several PBDs related to monitoring WebSphere MQ
connectors and messaging system through the CA APM for IBM WebSphere MQ.
Note: For more information, see the CA APM for IBM WebSphere MQ Guide.
The Java agent also installs application server-specific PBDs, which vary depending on
the application server you are monitoring.
More information:
Default Tracer Groups and Toggles Files (see page 90)
Turning tracer groups on or off (see page 99)

Default PBD Files From Previous Releases
The agent uses the PBD and PBL files from the current release by default. However, the
product provides the PBD and PBL files from previous releases in the
<Agent_Home>/wily/examples/legacy directory. Each of the file names in this directory
have a -legacy suffix, for example, default-full-legacy.pbl.


Default PBL files
There are two PBL files available with each agent:
default-full.pbl (default)
References PBD files in which most tracer groups are turned on. Introscope uses
this set by default to demonstrate full Introscope functionality.
default-typical.pbl
A subset of tracer groups in the referenced PBD files are turned on. The typical set
includes common settings, and is the set you can customize for a particular
environment.
The Java agent also installs application server-specific PBLs, which vary depending on
the application server you are monitoring.
Default Tracer Groups and Toggles Files
Tracer groups are defined in PBD files. They cause the reporting of information about a
set of classes. In PBD files, tracer group information is referred to by the term flag. For
example, TraceOneMethodIfFlagged or SetFlag are defining tracer group information.
A tracer group consists of a set of tracers that is applied to a set of classes. For example,
there are tracer groups which report the response times and rates for all RMI classes.
You can refine the gathering of metrics on your systems by turning on or off certain
tracer groups. This method affects overhead usage, either increasing or decreasing it,
depending on how you configure the tracer groups.
Tracer groups are modified in the toggles-full.pbd and the toggles-typical.pbd files,
which are referred to by the default-full.pbl and default-typical.pbl files. The following
list is of default tracer groups and their default configurations:
AgentInitialization
Agent Initialization Configuration
Full: on
Typical: on
ApacheStandardSessionTracing
HTTP Session Configuration
Full: on
Typical: on


AuthenticationTracing
Authentication Configuration
Full: on
Typical: on
CorbaTracing
CORBA method invocations
Full: on
Typical: on
DBCPTracing
DBCP Configuration
Full: on
Typical: on
DBCPv55Tracing
DBCP Configuration
Full: on
Typical: on
EJB2StubTracing
EJB 2.0 Configuration
Full: on
Typical: on
EJB3StubTracing
EJB 3.0 Configuration
Full: on
Typical: on
EntityBean3Tracing
Entity EJB 3.0 method invocations
Full: on
Typical: on
EntityBeanTracing
Entity EJB method invocations
Full: on
Typical: on


HTTPServletTracing
HTTP servlet service responses
Full: on
Typical: on
If you are using Application Server AutoProbe, turn on this tracer group:
HTTPAppServerAutoProbeServletTracing.
InstanceCounts
Counts number of instances of the object type that is identified with the tracer
group.
Full: on
Typical: on
Nothing is traced until classes are identified with this tracer group.
J2eeConnectorTracing
J2EE connector information
Full: on
Typical: on
JavaMailTransportTracing
Mail sending times
Full: on
Typical: on
JDBCQueryTracing
JDBC queries
Full: on
Typical: on
JDBCUpdateTracing
JDBC updates
Full: on
Typical: on
JMSConsumerTracing
JMS message processing times
Full: on
Typical: on


JMSListenerTracing
JMS message processing times
Full: on
Typical: on
JMSPublisherTracing
JMS message broadcast times
Full: on
Typical: on
JMSSenderTracing
JMS message broadcast times
Full: on
Typical: on
JSPTracing
JSP service responses
Full: on
Typical: on
MessageDrivenBean3Tracing
Message-driven EJB 3.0 method invocations
Full: on
Typical: on
MessageDrivenBeanTracing
Message-driven EJB method invocations
Full: on
Typical: on
NIOSocketTracing
Generates a set of metrics for each socket connection under the
NIO|Channels|Sockets node, with more metrics under the Backends node.
Full: on
Typical: on


NIOSocketSummaryTracing
Generates a single set of metrics covering all NIO socket connections
Full: on
Typical: on
These metrics include connections that are excluded from NIOSocketTracing
metrics by agent properties. These metrics also include some internal NIO sockets
which are always excluded from NIOSocketTracing metrics.
NIOSelectorTracing
Prevents certain internal JVM uses of NIO channels from being counted in NIO
channel metrics.
Users do not have control of this option.
NIODatagramTracing
Generates a set of metrics for each datagram "connection".
Full: on
Typical: on
See NIODatagramTracing metrics on page 269 for more information.
NIODatagramSummaryTracing
Generates a single set of metrics measuring all NIO datagram activity.
Full: on
Typical: on
These metrics include connections that are excluded from NIODatagramTracing
metrics by agent properties. These metrics also include some internal NIO sockets
which are always excluded from NIODatagramTracing metrics.
PersistentSessionTracing
HTTP session configuration
Full: on
Typical: on


RMIClientTracing
RMI client method invocations
Full: on
Typical: on
RMIServerTracing
RMI server method invocations
Full: on
Typical: on
ServerInfoTracing
Server info configuration
Full: on
Typical: on
SessionBean3Tracing
Session EJB 3.0 method invocations
Full: on
Typical: on
SessionBeanTracing
Session EJB method invocations
Full: on
Typical: on
SocketTracing
Network socket bandwidth and SSL tracking
Full: on
Typical: on
StrutsTracing
Execution times of actions in the Struts framework
Full: on
Typical: on
SuperpagesSessionTracing
Full: on
Typical: on


ThreadPoolTracing
Thread Pool Configuration
Full: on
Typical: on
UDPTracing
User Datagram Protocol (UDP) socket bandwidth
Full: on
Typical: on
UnformattedSessionTracing
Full: on
Typical: on
EJB3MethodLevelTracing
EJB 3.0 activity at the method level
Full: on
Typical: off
EJBMethodLevelTracing
EJB activity at the method level
Full: on
Typical: off
FileSystemTracing
File system bytes written and read
Full: on
Typical: off
JAXMListenerTracing
JAXM message sends
Full: on
Typical: off
JNDITracing
JNDI lookup times
Full: on
Typical: off


JSPDBTagsTagLibraryTracing
Jakarta DB Tags custom tag library for reading and writing from a SQL database
Full: on
Typical: off
JSPIOTagLibraryTracing
Jakarta IO custom tag library for various input and output tasks
Full: on
Typical: off
JTACommitTracing
Commit times using JTA
Full: on
Typical: off
ThreadTracing
Number of active threads by class
Full: on
Typical: off
XMLSAXTracing
Time spent parsing XML document
Full: on
Typical: off
XSLTTracing
XML transformation time
Full: on
Typical: off
CatchException
Exception configuration
Full: off
Typical: off
FormattedSessionTracing
HTTP Session configuration
Full: off
Typical: off


HTTPAppServerAutoProbeServletTracing
HTTP Servlets configuration
Full: off
Typical: off
HTTPSessionTracing
HTTP Session configuration
Full: off
Typical: off
JSPTagLibraryTracing
Processing time of custom JSP tags
Full: off
Typical: off
ManagedSocketTracing
Network configuration
Full: off
Typical: off
ThrowException
Exception configuration
Full: off
Typical: off
Generally, do not edit the default toggles PBD files. However, you can refine the
gathering of metrics by turning on or off certain tracer groups. Tracer groups can be
modified in the toggles files by:
Turning on/off tracer groups to save on the system overhead
Adding classes to a tracer group
Tracer groups report information only when turned on (uncommented) and are
activated with the keyword TurnOn.
Note: The Java agent supports EJB (2.0 at a minimum) instrumentation. To tailor your
metric collection, turn on or off the tracer groups that are associated with EJB. Support
for EJBs in the application triage map extends only to Session and Entity beans. Message
Driven beans are not supported.


Setting toggles to gather additional metric information
The following toggles, when turned on, cause the collection of additional metrics across
all APIs for CA Technologies-provided tracer groups that are enabled. You must add
these toggles to your full or typical toggle file to change the configuration.
DefaultStalledMethod Tracing
Stalled method tracing
Full: on
Typical: on
DefaultConcurrentInvocationTracing
Concurrent invocation information
Full: on
Typical: off
DefaultRateMetrics
Invocation rate metrics
Full: off
Typical: off
Turning tracer groups on or off
You can refine the gathering of metrics on your systems by turning on or off certain
tracer groups.
To turn a tracer group on
1. Locate the toggles-full.pbd or toggles-typical.pbd file (depending on which file type
(<appserver>-full.pbl or <appserver>-typical.pbl is in use by AutoProbe or the Java
agent). These files are found within the <appserver home>wily/core/config
directory or <Introscope_Home>/core/config/systempbd directory.
2. Locate the tracer group to turn on, and uncomment the line by removing the pound
sign from the beginning of the line. The directive in the following example is turned
on, and will cause the tracing of all HTTP Servlets.
TurnOn: HTTPServletTracing
Note: Any uncommented (turned on) directive for a tracer group causes the tracer
group to be used.
To turn a tracer group off
Comment the tracer group by placing a pound sign at the beginning of the line, as in
the following example:
#TurnOn: HTTPServletTracing


Adding classes to a tracer group
You can turn on tracing for a particular class by adding the class to an existing tracer
group. To identify a class as being part of a tracer group, use one of the Identify
keywords.
For example, to add the class, com.myCo.ejbentity.myEJB1, to the tracer group,
EntityBeanTracing:
IdentifyClassAs: com.myCo.ejbentity.myEJB1 EntityBeanTracing
The identify keywords are:
IdentifyInheritedAs
IdentifyClassAs
IdentifyCorbaAs
EJB subclass tracing
By default, entity and session EJB-related directives add probes only for EJBs that
directly and explicitly implement the entity, session, or message-driven EJB interfaces.
Often, an applications EJBs are subclasses of classes which directly and explicitly
implement the entity or session EJB interface. These are not tracked by default by
Introscope.
For EJB subclasses to be tracked by Introscope, they must be added to the appropriate
tracer group. To do this, add entries that refer to the direct ancestors of the EJB
subclasses to be tracked.
From these models, replace <entity.bean.ancestor.class> or
<session.bean.ancestor.class> with the fully-qualified class name of the immediate
ancestor of the EJBs to be instrumented.
For entity EJBs:
IdentifyInheritedAs: <entity.bean.ancestor.class> EntityBeanTracing
For session EJBs:
IdentifyInheritedAs: <session.bean.ancestor.class> SessionBeanTracing


The examples below are based on this class hierarchy:
mySessionEJB implements javax.ejb.SessionBean
mySessionEJBsubclass1 extends mySessionEJB
mySessionEJBsubclass1a extends mySessionEJBsubclass1
mySessionEJBsubclass1b extends mySessionEJBsubclass1
mySessionEJBsubclass2 extends mySessionEJB
The tracer group, SessionBeanTracing, causes the tracking of mySessionEJB:
The following tracer traces mySessionEJBsubclass1 and mySessionEJBsubclass2.
IdentifyInheritedAs: mySessionEJB SessionBeanTracing
The following tracer traces mySessionEJBsubclass1a and mySessionEJBsubclass1b.
IdentifyInheritedAs: mySessionEJBsubclass1 SessionBeanTracing
Note: This example does not use packages. If your code is in a package, it needs to
include the package name with the class name.
EJB 3.0 annotations
The following directive allows you to group any class containing the given class-level
annotation into tracer groups. This directive supports EJB 3.0. EJBs conforming to the
3.0 specifications do not explicitly implement any well-known interface, but instead are
entirely enabled via annotations. To easily identify EJB 3.0 classes, use this directive:
IdentifyAnnotatedClass <annotation-name> <flag-name>
To use this directive, create a directive class and directive parser class for the new
directive. You must then add a matcher class to examine your bytecode to determine if
a class contains a given annotation.
Note: This directive does not support method-level annotations.
EJB Support for the Application Triage Map
CA Introscope supports out-of-the-box tracing of EJB (2.0 at a minimum) session and
entity beans, specifically for use in the Workstation application triage map. CA
Technologies recommends using this out-of-the-box functionality in test environments
only, as this configuration affects the start-up time of the agent.
If deploying this functionality to your production environments, configure EJB tracers for
specific things. The out-of-the-box functionality can be too broad.


Use the following directive to instruct ProbeBuilder to flag a class that is inheriting from,
or implementing, the superclass or interface:
IdentifyDeepInheritedAs
For EJB 2.0 application triage map support, the following directives are in the j2ee.pbd
file:
IdentifyDeepInheritedAs: javax.ejb.EJBObject EJB2StubTracing
IdentifyDeepInheritedAs: javax.ejb.SessionBean SessionBeanTracing
IdentifyDeepInheritedAs: javax.ejb.EntityBean EntityBeanTracing
IdentifyDeepInheritedAs: javax.ejb.MessageBean MessageBeanTracing
These directives let the ProbeBuilder identify the EJB stubs on the client side, and beans
on the server side that the application triage map use.
For EJB 3.0 application triage map support, the following directive is in the j2ee.pbd file:
IdentifyInheritedAnnotatedClassAs
The directive matches all classes that implement interfaces directly, or through a super
interface.
In the context of the application triage map, the following additional directive is set in
j2ee.pbd:
IdentifyInheritedAnnotatedClassAs: javax.ejb.Remote EJB3StubTracing
EJB Naming
You can name called backends, generic frontends, and monitored components that deal
with EJBs. The name formatter lets you configure a suitable name for EJB (2.0 at a
minimum) client stubs and bean implementations.
The EjbNameFormatter classes define an EJB-related metric name, application triage
map application name, or node name. You use following placeholders:
For EJB client stubs: {classname}, {interface}, and {method}
For EJB beans: {classname}, {bean}, {interface}, and {method}
The following metric names are used by default:
EJB Bean frontend: EJB|{interface}
EJB Client stub backend: EJB|{interface}
Application triage map owner name for EJB bean: {interface}
Application triage map node name for EJB Client Stub: Client {interface}
Application triage map node name for EJB Bean: Server {interface}
Using the IntroscopeAgent.profile, PBLs, and PBDs together


These names are default EJB name formatters. They are used in the j2ee.pbd and
appmap-ejb.pbd files. You use the same name formatters, but different metric names.
For example, you could modify existing tracer directives to use a more appropriate
name, but keep the same flags:
...
# Default commented out:
#TraceComplexMethodsIfFlagged: EJB2StubTracing EJB2BackendTracer "{interface}"
#Add the EJB application name to backend marker as well as called method
TraceComplexMethodsIfFlagged: EJB2StubTracing EJB2BackendTracer
"MyCustomerBeanApp-{interface}-{method}"
...
SetTracerClassMapping: EJB2BackendTracer
com.wily.introscope.agent.trace.BackendTracer
com.wily.introscope.probebuilder.validate.ResourceNameValidator
SetTracerParameter: EJB2BackendTracer nameformatter
com.wily.introscope.agent.trace.ejb.Ejb2StubNameFormatter
Note: The EJB context tracer is set on setContext() method of EJB 2.0 beans. This tracer
is an internal CA Introscope tracer for the EJB 2.0 bean name formatter, which allows
the name formatter to function correctly.
Using the IntroscopeAgent.profile, PBLs, and PBDs together
When the Java agent is first installed, you set an instrumentation level, either full or
typical. This setting refers to the ProbeBuilder List (PBL) files default-typical.pbl and
default-full.pbl (see Default PBL files (see page 90) for more information).
Applying ProbeBuilder Directives
The way in which you apply PBDs depends on the method you choose to use. CA
Technologies recommends you use JVM AutoProbe to implement your PBDs. You can
also use the ProbeBuilder Wizard or the command line ProbeBuilder to implement your
PBDs.
Using JVM AutoProbe
When you are ready to implement a PBD file, add it to the hotdeploy directory.
AutoProbe looks for PBD files in the directory that contains the IntroscopeAgent.profile
file (by default, this is the <Agent_Home>/wily/core/config directory), and the
<Agent_Home>/wily/core/config/hotdeploy directory. AutoProbe resolves file names
relative to these directories. If you have moved the location of your wily directory, be
sure to map the file path to the correct directory.


To implement PBDs using AutoProbe
1. Save modified standard PBD or PBLs to the <Agent_Home>/wily/core/config
directory.
2. Copy custom PBDs into the <Agent_Home>/wily/core/config/hotdeploy directory.
Any PBDs added to this directory will be implemented without having to update or
modify the introscope.autoprobe.directivesFile property in the
Note: If you have enabled dynamic instrumentation, the PBDs in the hotdeploy
directory are picked up live from the folder. No reboot is required. For more
information about dynamic instrumentation, see Dynamic ProbeBuilding (see
page 77).
Using the ProbeBuilder Wizard or command-line ProbeBuilder
When you are ready to implement a PBD file, add it to the hotdeploy directory. The
Command-line ProbeBuilder looks for any custom directive files in the same directory
where ProbeBuilder is run from, and the <Agent_Home>/wily/core/config/hotdeploy
directory. The Command-line ProbeBuilder resolves file names relative to these
directories.
The steps to implement ProbeBuilder Directives using the ProbeBuilder Wizard or
command-line ProbeBuilder are the same as using JVM AutoProbe. See Using JVM
AutoProbe (see page 103) for more information.
Instrumenting with new and changed PBDs
For new or changed directives to take effect, your applications must be instrumented
using the latest PBDs. This process varies depending on the ProbeBuilding method you
use.
JVM 1.5 systems using JVM AutoProbe via -javaagent
You can configure dynamic instrumentation, allowing changed PBDs to take effect
without application or Java Agent restart. This enables you to perform PBD corrections,
or perform triage-driven instrumentation without interrupting application service. For
more information see Dynamic ProbeBuilding (see page 77).


New and Changed ProbeBuilder Files
Valid for: JVMs earlier than 1.5 or installations using Xbootclasspath
New and changed ProbeBuilder Directive files or ProbeBuilder List files take effect the
next time the application server loads the application classes.
If your managed applications are not running when you add or change directives, the
applications are instrumented using the updated directives the next time you start the
applications.
If your managed applications are running, it is necessary to load, or reload, the managed
application classes.
How you cause the classes to reload depends upon the application server you use. Most
application servers require a restart.
Using the ProbeBuilder Wizard
To use the ProbeBuilder Wizard
1. The Custom Directives screen will list the PBD files you placed in the hotdeploy
directory described in Using the ProbeBuilder Wizard or command-line
ProbeBuilder (see page 104).
2. Select the custom directives files to use.
Using the command-line ProbeBuilder
Important! CA Technologies recommends using the command-line ProbeBuilder as your
last option for Introscope-enabling your latest PBDs.
To use the command-line ProbeBuilder
1. Stop your managed application.
2. Run the command-line ProbeBuilder or the ProbeBuilder Wizard, supplying the
custom PBD and PBL files in the command line.
3. Configure the application to use the new files.
4. Start the managed application.
5. If they are not already running, start the Enterprise Manager and the Workstation.
Creating custom tracers


You can further refine your metric collection by creating custom PBD files. Creating
custom directives, by creating tracers to track application specific measurements,
require the use of specific syntax and keywords. To write custom tracers, you must
define:
The directive type (indicating generically how many class(es) or method(s) to trace)
The specific class(es) or method(s) to trace
The type of information to trace in the class(es) or method(s) (for example, a time, a
rate, or a count)
The fully-qualified metric name (including the resource path) under which to
present this information
Custom PBDs are stored in the <Agent_Home>/wily/core/config/hotdeploy directory.
Any PBDs added to this directory will be implemented without having to update or
modify the introscope.autoprobe.directivesFile property in the IntroscopeAgent.profile.
If you have enabled dynamic instrumentation, the PBDs in the hotdeploy directory are
picked up live from the folder. No reboot is required. For more information about
dynamic instrumentation, see Dynamic ProbeBuilding (see page 77).
Once a custom PBD is created, Introscope treats it as if it is an out-of-the-box PBD. You
can set alerts on the metrics created, save them to SmartStor, or use them in the
creation of custom dashboards in the Introscope Workstation.
Note: Be sure to choose methods to trace carefully, as more methods traced means
more overhead.
Using a custom BlamePointTracer tracer for common metrics
A BlamePointTracer is the most commonly used tracer. This tracer generates five
separate metrics for associated methods or classes:
Average Response Time (ms)
Concurrent Invocations
Errors Per Interval
Responses Per Interval
Stall Count


The following is an example of a BlamePointTracer. A BlamePointTracer has been set for
a method called search in class petshop.catalog.Catalog. PetShop|Catalog|search is the
name of the node under which the BlamePoint metrics will be displayed in the
Introscope Investigator.
TraceOneMethodOfClass: petshop.catalog.Catalog search BlamePointTracer
"PetShop|Catalog|search"
Directive names and arguments used in tracer syntax
In addition to simple keywords that associate tracers into groups or enable/disable
groups, PBD files contain tracer definitions. For Introscope to recognize and process
tracers, you must use a specific syntax when constructing custom tracers. A tracer is
composed of a directive and information about the method or class to trace, in the
following format:
<directive>: [arguments]
where [arguments] is a list, and is directive-specific.
Note: Depending on the directive used, only a subset of these parameters are required.
<directive>
The most common directives to use are the following trace directives:
TraceOneMethodOfClass
Traces a specified method in the specified class.
TraceAllMethodsOfClass
Traces all methods in the specified class.
TraceOneMethodIfInherits
Traces one method in all direct subclasses or direct interface implementations of
the specified class or interface.
TraceAllMethodsIfInherits
Traces all methods in all direct subclasses or direct interface implementations of the
specified class or interface.
TraceOneMethodIfFlagged
Traces one method if the specified class is included in a tracer group that has been
enabled with the TurnOn keyword.


TraceAllMethodsIfFlagged
Traces all methods if the specified class is included in a tracer group that has been
enabled with the TurnOn keyword.
Note: Only concrete, implemented methods can be traced and report metric data while
running. An abstract method specified in a custom tracer results in no metric data being
reported.
The expected syntax for trace directives usually consist of the following arguments:
<Tracer-Group>
The group to which the tracer is associated.
<class>
A fully qualified class or interface name to trace. Fully qualified classes include the
full assembly name of the class as well as the name, for example:
[MyAssembly]com.mycompany.myassembly.MyClass
The assembly name must be enclosed in [] brackets.
<method>
The method name (for example, MyMethod)
OR
the full method signature with return type and parameters (for example,
myMethod;[mscorlib]System.Void([mscorlib]
System.Int32). For more information on method signatures, see Signature
differentiation (see page 112).)
<Tracer-name>
Specifies the tracer type to be used. For example, BlamePointTracer. See the Tracer
name table below for descriptions of tracer names.
<metric-name>
Controls how the collected data is displayed in the Introscope Workstation.
The following examples describe three ways to specify the name and location of a
metric at different levels of the metrics tree.
metric-namethe metric appears immediately inside the agent node.
resource:metric-namethe metric appears inside one resource (folder) below the
agent node.
resource|sub-resource|sub-sub-resource:metric-namethe metric appears more
than one resource (folder) level deep below the agent node. Use pipe characters (|)
to separate the resources.


Commonly used tracer names and examples
The following list describes the most commonly used tracer names and what they trace.
BlamePointTracer
Provides a standard set of metrics including average response time, per interval
counts, concurrency, stalls, and errors for a blamed component.
ConcurrentInvocationCounter
Reports the number of times a method has started but not yet finished. The result
is reported under the metric name specified in the tracer, <metric-name>, in the
Investigator tree. An example use of this tracer would be counting the number of
simultaneous database queries.
DumpStackTraceTracer
Dumps a stack trace to the instrumented application's standard error for methods
to which it is applied. The exception stack trace thrown by the Dump Stack Tracer is
not a true exceptionit is a mechanism for printing the method stack trace.
This feature is useful for determining callpaths to a method.
Important! This feature imposes heavy system overhead. It is strongly
recommended that this tracer only be used in a diagnostic context where a sharp
increase in overhead is acceptable.
MethodCPUTimer
Average CPU time (in milliseconds) used during method execution and reports it
under <metricname> in the metrics tree.
Note: This tracer requires a platform monitor on the supported platform.
MethodTimer
Average method execution time in milliseconds and reports it under the metric
name specified in the tracer, <metric-name>, in the metrics tree.
PerIntervalCounter
Number of invocations per interval. This interval will change based on the view
period of the consumer of the data (for example, the View pane in the Investigator).
It is reported under the metric name specified in the tracer, <metric-name>, in the
Investigator tree.


Using quotes in custom tracer definitions
Custom tracers can contain metric names with spaces in them. When using spaces in
your custom metric names, CA Technologies recommends putting quotes ("") around all
metric names.
Important! Do not place quotes around class names. This causes the custom tracers to
malfunction. For example:
Correct
IdentifyClassAs: MyClass MyTracers
Incorrect
IdentifyClassAs: "MyClass" MyTracers
If you create a metric name that contains a class name, you must use quotes around the
whole metric name. Metric names are allowed to have spaces, and all spaces in metric
names must be contained within quotes. For example, the metric name
"{classname}|Test One Node" should be represented as follows:
Correct
TraceOneMethodIfFlagged: MyTracers AMethod BlamePointTracer "{classname}|Test One
Node"
Incorrect
TraceOneMethodIfFlagged: MyTracers AMethod BlamePointTracer {classname}|Test One
Node
Important! Introscope will not monitor classes having invalid class file names. For
example, in the class file name:
org/jboss/seam/example/seambay/AuctionImage$JaxbAccessorM_getData_setData_[B:
The _[B: causes the class file name to be invalid. You cannot use an open square
brackets ([) as part of the Java class file name. When Introscope encounters such classes
having invalid class names, it fails to instrument them and reports them as an error
message in the agent logs.
The following sections are examples of method tracers. In the following example,
quotes ("") are used around the metric names. CA Technologies highly recommends
putting quotes around all metric names when you create custom metric names.
Average tracer example
This tracer tracks the average execution time of the given method in milliseconds.
TraceOneMethodOfClass: com.sun.petstore.catalog.Catalog search BlamedMethodTimer
"Petstore|Catalog|search:Average Method Invocation Time (ms)"


Rate tracer example
This tracer counts the number of times the method is called per second, and reports this
rate under the specified metric name.
TraceOneMethodOfClass: com.sun.petstore.catalog.Catalog search
BlamedMethodRateTracer "Petstore|Catalog|search:Method Invocations Per Second"
Per interval counter tracer example
This method tracer counts the number of times the method is called per interval, and
reports the per interval count under the specified metric name.
TraceOneMethodOfClass: com.sun.petstore.catalog.Catalog search PerIntervalCounter
"Petstore|Catalog|search:Method Invocations Per Interval"
The interval is determined by the monitoring logic in the Enterprise Manager, such as
the Graph frequency.
The preview pane in the Investigator defaults to 15 second intervals.
Counter tracer example
This tracer counts the total number of times the method is called.
TraceOneMethodOfClass: com.sun.petstore.cart.ShoppingCart placeOrder
BlamedMethodTraceIncrementor "Petstore|ShoppingCart|placeOrder:Total Order Count"
Combined counter tracers example
These tracers combine incrementor and decrementor Tracers to keep a running count.
TraceOneMethodOfClass: com.sun.petstore.account.LoginEJB login
MethodTraceIncrementor "Petstore|Account:Logged In Users"
TraceOneMethodOfClass: com.sun.petstore.account.LogoutEJB logout
MethodTraceDecrementor "Petstore|Account:Logged In Users"
Advanced single-metric tracers
Directives and tracers track methods, classes, and sets of classes. A single-metric tracer
reports a specific metric for a specific method, which is the smallest unit that Introscope
can track. Single-metric tracers can be created in several ways: through the method
signature, by substituting keywords, or by manipulating the metric name parameters.


Signature differentiation
Tracers can be applied to a method based on the method signature.
To trace a single instance of a method with a specific signature, append the signature to
the method name (including return type) specified using the internal method descriptor
format.
For example, myMethod(Ljava/lang/String;)V traces the instance of the method with a
string argument and void return type.
For complete information about this format, see the Sun Java Virtual Machine
Specification.
Metric name keyword-based substitution
Keyword-based substitution allows runtime substitution of values into the metric name.
The parameters in the metric name in the tracer are substituted at runtime for the
actual values into the metric name. This feature can be used with any directive.
{method}
Name of the method being traced
{classname}
Runtime class name of the class being traced
{packagename}
Runtime package name of the class being traced
{packageandclassname}
Runtime package and class name of the class being traced
Note: If Introscope processes a class which does not have a package, it will replace
{packagename} with the string "<Unnamed Package>".
Keyword-based substitution: Example 1
If the metric name for a tracer in the pbd file is:
"{packagename}|{classname}|{method}:Response Time (ms)"
and the tracer is applied to method myMethod with a runtime class of myClass that is in
package myPackage, the resulting metric name would be:
"myPackage|myClass|myMethod:Response Time (ms)"


Keyword-based substitution: Example 2
If a tracer with a metric name in the .pbd file of
"{packageandclassname}|{method}:Response Time (ms)"
was applied to the same method, the resulting metric name would be
"myPackage.myClass|myMethod:Response Time(ms)"
Note: The . between the package and class instead of the | in the first example.
Metric-name-based parameters
You can create a single-method tracer that creates a metric name based on parameters
passed to a method using the TraceOneMethodWithParametersOfClass keyword, using
this format:
TraceOneMethodWithParametersOfClass: <class-name> <method> <tracer-name>
<metric-name>
Parameters can be used in the metric name. This is accomplished by substituting the
value of parameters for placeholder strings in the metric name. The placeholder strings
to use are "{#}" where # is the index of the parameter to substitute. The indices start
counting at zero. Any number of parameter substitutions can be used in any order. All
parameters are converted to strings before substitution into the metric name. Object
parameters other than strings should be used with caution because they are converted
using the toString() method.
Important! If you are unclear about what string the parameter will be converted to, do
not use it in the metric name.


Metric-name-based example
A Web site uses a class named order, with a method named process. The method has
parameters for different kinds of orders, either book or music.
You can create a tracer like this:
TraceOneMethodWithParametersOfClass: order process(LJava/lang/string;)V
MethodTimer "Order|{0}Order:Average Response Time (ms)"
This tracer produces metrics like these:
Order
BookOrder
MusicOrder
You can also use the TraceOneMethodWithParametersIfInherits keyword.
Skip directives
Certain packages, classes, or methods can be skipped by AutoProbe or ProbeBuilder by
using skip directives. By default, the Java agent and fundamental Java classes and
packages are skipped by AutoProbe or ProbeBuilder.
Counting object instances
The InstanceCounts tracer group counts the number of instances of the particular object
types associated with it (for information on associating object types with the
InstanceCounts tracer group using the standard IdentifyClassAs and IdentifyInheritedAs
directives, see Adding classes to a tracer group (see page 100)). Any instances explicitly
allocated in your code will be counted. Subtypes will also be counted. Objects created
through different mechanisms, such as deserialization or cloning, might not be counted.
Tracing using this tracer group could potentially incur incremental performance (and
memory) impact, depending entirely on the number of instances counted.
Note: CA Technologies testing has shown that in practice, the number of instances has
to be quite large for an impact to be noticeable.


Turning on InstrumentPoint directives
There are two types of directives identified by the keyword, InstrumentPoint: those that
trace exceptions, and one that causes agent initialization when the application starts up
(instead of when the first Probe is run).
Exceptions
The following directives are used to turn on tracing of exceptions either where thrown
or caught. They can cause performance degradation so they are not turned on by
default. To turn either of these on, uncomment the appropriate line:
#InstrumentPoint: ThrowException
#InstrumentPoint: CatchException
Agent initialization
The agent initialization instrument point directive does not cause additional overhead
and is turned on by default in both full and typical PBD sets.
InstrumentPoint: AgentInitialization
If multiple ProbeBuilder Directive files are used, any settings (such as tracer groups,
Skips, InstrumentPoints, Custom Method Tracers) turned on in any file take effect.
Combining custom tracers
You can use multiple tracers that affect the same metric, in effect combining them. This
is most commonly used with incrementors and decrementors.
This example creates a metric named Total Purchases. With a class cart and methods
buyBook and buyCD, create the following tracers:
TraceOneMethodOfClass cart buyBook PerIntervalCounter "Total Purchases"
TraceOneMethodOfClass cart buyCD PerIntervalCounter "Total Purchases"
This increments the metric Total Purchases when someone buys a piece of merchandise.
Instrumenting and Inheritance
Valid for: JVMs earlier than 1.5
CA Introscope does not automatically instrument classes in the deeper levels of a class
hierarchy in JVMs earlier than 1.5. When the subclasses of a probed class more than one
level deep are loaded, the new and overridden methods are not automatically
instrumented. Classes that explicitly name a probed interface as being implemented are
instrumented even though they implement the interface indirectly.


For example, assume a class hierarchy in which ClassB extends ClassA, and ClassC
extends ClassB, like so:
Interface/ClassA
ClassB
ClassC
When you instrument ClassA, ClassB is also instrumented because it explicitly extends
ClassA. However, CA Introscope does not instrument ClassC because ClassC does not
explicitly extend ClassA. To instrument ClassC, explicitly identify ClassC.
In Java environments earlier than 1.5, to ensure that subclasses are instrumented,
follow the EJB subclass tracing (see page 100) instructions.
If you use JVM 1.5, you can configure CA Introscope to instrument multiple subclass
levels of a probed class. (see page 81)
Java Annotations
CA Introscope allows the use of Java 1.6 annotations when creating custom metrics.
Note: For more information about Java annotations, see the documentation at the
Java developer website.
Use IdentifyAnnotatedClassAs to place the class in a tracer group, and then use
TraceAllMethodsIfFlagged directives to instrument the methods in the class. For
example:
SetFlag: AnnotationTracing TurnOn: AnnotationTracing
IdentifyAnnotatedClassAs: com.test.MyAnnotation AnnotationTracing
TraceAllMethodsIfFlagged: AnnotationTracing BlamePointTracer
"Target|MyTarget|{classname}"
In the example, com.test.MyAnnotation is the annotation name. When creating your
own annotations, use a term in your code. Classes containing the annotation name are
identified.
Using Blame Tracers to mark blame points


The Blame Technology for CA Introscope works in a managed Java application to
enable you to view metrics at the application tiers: the frontends and backends of your
application. This capability, referred to as boundary blame, allows users to triage
problems to an application frontend or backend. This information is also used in the
Workstation application triage map to mark the edges of your applications.
For information about how CA Introscope determines frontends and backends, and
about options for configuring URL Groups to control how metrics for frontends are
aggregated, see Configuring Boundary Blame (see page 157).
The following sections describe how you can use tracers to explicitly mark the frontends
and backends in your application.
Blame Tracers
Introscope provides tracers for capturing front and backend metrics: FrontendMarker
and BackendMarker. These tracers explicitly mark a frontend and backend, respectively.
You can use FrontendMarker and BackendMarker to instrument your own code, for
instance code that accesses a backend, to cause Introscope to capture and present
metrics for custom components in the Investigator tree.
If no components are instrumented with the FrontendMarker tracer (or its subclasses
HttpServletTracer and PageInfoTracer), no frontend metrics are generated and no
component will be marked as a frontend for transactions.
When more than one component within a transaction is instrumented with the
FrontendMarker tracer (or its subclasses), only the first designated component will
generate Frontend metrics.
Note: When using frontend tracers, the name of the application given in the frontend
tracer must match the name given for the application triage map tracers, keeping in
mind that both are case-sensitive. For example, if you name the frontend tracer
AppOne and the application triage map tracer refers to this tracer as APPONE,
information about AppOne will not be displayed correctly in the Workstation application
triage map.
To prevent specific classes from being marked as frontend, the PBD parameter
is.frontend.unless can be specified. For information on the PBD directive
is.frontend.unless, see the Custom FrontendMarker directive (see page 118).
If no BackendMarker is configured, Introscope will infer a backendany component
that opens a client socket will be a default backend if none is explicitly marked.


It is useful to use BackendMarker:
to assign a desired name to an item that Introscope detects as a backend.
to mark custom Java sockets that Introscope does not instrument.
for native sockets that are called through the Java Native Interface (JNI), to identify
a Java/JNI bridging method as the backend.
FrontendMarker and BackendMarker are instances of BlamePointTracer which provides
metrics such as average response time, per interval counts, concurrency, stalls, and
errors for a blamed component. A BlamePointTracer can be applied to middle
components for a more granular Blame Stack.
High agent CPU overhead from deep nested frontend transactions
Servlets are configured by Introscope to be seen as frontends. A typical transaction
starts with a servlet, which may call an EJB, which calls a back-end. Its possible for
servlets to call other servlets in a nested way, which Introscope sees as nested
frontends. In most cases, this does not add to agent CPU overhead.
However, deep transactions having nested frontend levels (for example 40 levels deep)
may result in high CPU overhead. For example, if a servlet repeatedly calls itself in a
transaction (continuous recurring calls) or calls multiple other servlets, you may see an
increase in agent CPU overhead. If the overhead is unacceptable, contact CA Support.
Custom FrontendMarker Directive
The PBD parameter is.frontend.unless enables some classes to not be marked as a
frontend component. The FrontendMarker (or its subclasses, such as HttpServletTracer)
instruments these classes. Set the parameter as a comma-separated list of absolute
class names. This parameter can be useful when the initial component is a generic
dispatcher. And this dispatcher forwards the request to a more specific component
which handles the type of request received. The second component would therefore be
a better frontend marker. The default is an empty list. PBD parameters are not dynamic.
A restart of the instrumented application server is required when the value of this
parameter is changed.
Important! Separate class names with a comma, not spaces. The use of spaces
invalidates the SetTracerParameter directive.
Any classes designated in the parameter list that the tracer instruments and to which
this parameter is applied:
Are designated as frontends.
Do not generate metrics under the Frontends node in the Investigator.


For example, you want to prevent the classes NotAFrontend and AnotherNonFrontend
from being treated as frontends in the package com.ABCCorp. These classes are
instrumented with a FrontendMarker named MyFrontendTracer. You use the following
PDB directive:
SetTracerParameter: MyFrontendTracer is.frontend.unless
com.ABCCorp.NotAFrontend,com.ABCCorp.AnotherNonFrontend
Blame Tracers in standard PBDs
Two of the standard PBDs provided with Introscope, j2ee.pbd and sqlagent.pbd,
implement Boundary Blame Tracing.
HttpServletTracer in j2ee.pbd is an instance of FrontendMarker.
SQLBackendTracer in sqlagent.pbd is an instance of BackendMarker.
The following Blame Tracers used in previous versions of Introscope still exist, but are
not typically used in Introscope PBDs:
BlamedMethodTimer
BlamedMethodRateTracer
BlamedMethodTraceIncrementor
BlamedMethodTraceDecrementor
Boundary Blame and Oracle Backends
In the current version of Introscope, Oracle databases are not detected based on the
socket connectionSQL Agent must be available for Introscope to detect Oracle
backends automatically.
To enable Introscope to detect Oracle backends in the absence of SQL Agent, make the
following modification to oraclejdbc.pbd:
In this portion of oraclejdbc.pbd:
#Socket data from the Oracle driver reports too many metrics
SkipPackagePrefixForFlag: oracle.jdbc. SocketTracing
SkipPackagePrefixForFlag: oracle.net. SocketTracing


Comment out the skips as shown in the following example:
#Socket data from the Oracle driver reports too many metrics
#SkipPackagePrefixForFlag: oracle.jdbc. SocketTracing
#SkipPackagePrefixForFlag: oracle.net. SocketTracing
Note: For more information, see the Knowledge Base article Disabling Database Name
Formatting in 7.1 (KB 1240) at http://ca.com/support.


Chapter 6: Java Agent Naming

This section has information about agent naming, related environmental and
deployment considerations, and options for automatically naming your agents.
Understanding the Java Agent name (see page 121)
Agent naming considerations for clustered applications (see page 124)
Specifying an agent name using a Java system property (see page 124)
Specifying an agent name using a system property key (see page 125)
Obtaining an agent name from the application server (see page 125)
Automatic agent naming (see page 126)
Enabling cloned agent naming in clustered environments (see page 129)
Application triage map and the agent name (see page 130)
Understanding the Java Agent name
Each Java Agent running in your Introscope environment has a name, whether you
assigned one explicitly, configured a method of automatically assigning a name, or
simply started an instrumented application that the Java Agent monitors. The Java
Agent name is central to many views in the Introscope Workstation and Investigator,
and it is key to the process of associating monitoring logic with target applications.


When an agent report metrics to an Enterprise Manager, a node is created for that
agent in the Investigator tree. When you configure management logic in the
Workstationfor instance, Dashboards, Alerts, and Actionsthe agent name is a
component in the regular expressions that identify the applications to which the
management logic applies. The Investigator tree below shows agents named
domain1//Adminserver, running on host qw32vtest01 under the WebLogic process.

How the Agent Determines its Name
The Java Agent uses the following sequence to determine a name:
1. If the Java Agent determines a name using the first method, it accepts that name
and connects to the Enterprise Manager.
2. If the Java Agent does not determine a name using the first method, it tries the
second method, and so on.
3. If the Java Agent does not determine a name using any method, it names itself
"UnnamedAgent."
Method 1: Agent name specified in a Java system property
The agent name is defined using a Java system property on the command line. Using this
method will override any other agent naming method. See Specifying an agent name
using a Java system property (see page 124).
Method 2: Agent name specified in a system property key in the IntroscopeAgent.profile
The agent name is obtained from a Java system property specified in a property in the
IntroscopeAgent.profile. See Specifying an agent name using a system property key (see
page 125).


Method 3: Agent name obtained automatically from the application server
If you use certain versions of WebLogic or WebSphere, the agent name can be
automatically obtained from the application server using automatic agent naming
functionality. You can configure a time delay, to give the agent as much time as
necessary to determine its name before connecting to the Enterprise Manager. See
Obtaining an agent name from the application server (see page 125).
Method 4: Agent name specified explicitly in the agent profile
The agent name is defined in the IntroscopeAgent.profile, in the property
introscope.agent.agentName. This was the standard method for naming agents in early
Introscope versions. Use this option if you already have an agent profile for every
application.
Method 5: Agent name determined to be "Unknown agent"
If the agent is unable to determine a name using one of the methods listed above, then
the agent names itself "UnnamedAgent".
How Introscope resolves agent naming conflicts
The fully qualified agent namecomprised of host name, process name and agent
nameis typically unique to each agent in an Introscope environment. Agents with the
same agent name usually have a unique fully-qualified agent name because their host
name and process names are likely to be different. Multiple agents will have the same
fully-qualified agent name only if they reside on the same host, monitor the same
process, and have the same agent name.
If an agent tries to connect to an Enterprise Manager to which an agent with the same
fully-qualified agent name is already connected, the Enterprise Manager appends a
unique identifier to the name of the newly connecting agent. The identifier consists of a
percent (%) character and a digit. This mechanism ensures that multiple agents that
connect using the same fully-qualified name can be uniquely identified for the duration
of the connection. The Enterprise Manager renames the first duplicate agent to connect
by appending "%1" to its agent name.
For instance, assume that two agents with the fully qualified agent name:
hostPA|processNIM|PodAgent
connect to the Enterprise Manager, one after the other. The Enterprise Manager
renames the second agent:
PodAgent%1
If other agents with the same fully qualified name connect, they are renamed, in
succession, PodAgent%2, PodAgent%3, PodAgent%4, and so on, where the digit
following the percent character is the next number in sequence.
Agent naming considerations for clustered applications


When a renamed agent disconnects, the suffix it was assigned can be re-used. For
example, if PodAgent%1 disconnects while PodAgent remains connected, the next agent
with the fully qualified name hostPA|processNIM|PodAgent to connect will be renamed
PodAgent%1.
Reuse of the suffix identifier makes it possible that the Enterprise Manager might assign
the same suffix to a particular agents name from connection to connection. However,
on subsequent connections, a given agent could just as well be renamed differently.
Having an agents name vary from connection to connection is problematic when
querying historical datait is preferable to configure a naming strategy that avoids the
Enterprise Manager renaming agents.
Agent naming considerations for clustered applications
If you run multiple instances of the same application, Introscope attempts to resolve
identical agent names, including custom metric agents, by appending the agent name
with a character and a random number. CA Technologies recommends, however, that
you tell Introscope how to resolve the naming.
The options for resolving identical agent naming are:
Tell Introscope that the agents in question are cloned agents by enabling cloned
agent naming (described in Enabling cloned agent naming in clustered
environments (see page 129).)
Define unique agent names yourself and make separate agent profiles for each
agent (described in Configuring unique names for application instances (see
page 129).)
Let Introscope uniquely name each agent using its own naming scheme (described
in How Introscope resolves agent naming conflicts (see page 123).)
Specifying an agent name using a Java system property
To specify an agent name using Java system property
On the Java command line, supply the desired name using this property:
-Dcom.wily.introscope.agent.agentName=
Specifying an agent name using a system property key


Specifying an agent name using a system property key
This method is the second the agent uses to look for its name. Use this method if you
want the agent to be named from the value of an existing Java system property in your
deployment.
To specify an agent name using the System Property Key
2. Under the Agent Name section, specify the Java system property that will provide
the agent name in this property:
introscope.agent.agentNameSystemPropertyKey
Note: If the Java system property specified here does not exist, this property will be
ignored.
Obtaining an agent name from the application server
You can configure the agent to extract the application server instance name
automatically from the application server, and use that information to name itself. This
eliminates the need to configure individual agent names in a separate agent profile file.
The agent can also rename itself if there are changes in the application server
environment. This enables you to deploy an agent profile across a large number of
environments that might consist of a mix of application server platforms.
Application servers that support agent naming
Automatic agent Naming is supported when you use Introscope with these supported
application server versions:
JBoss
WebLogic 9.x
WebSphere 6.1.x distributed
WebLogic 10.0
WebSphere 7.0.x distributed
WebLogic 10.3
Automatic agent naming


The name of the application server displayed in the Introscope Workstation is
determined by a Java J2EE API. This sometimes causes the name of the application
servers to display differently in the Workstation because all application servers
implement the API differently. The names of multiple application servers may be
formatted differently in the Workstation, and even the same application server name
may be formatted differently from release to release.
When automatic agent naming is enabled, the agent starts and looks for name
information from the application server. The agent waits until an agent name is
obtained before attempting to connect to the Enterprise Manager.
When the agent locates naming information, Introscope edits the information to make
the agent name compliant with agent naming rules.
Agent names on supported application servers are comprised of several pieces of
information, which differ according to application server.
For JBoss, the agent name is based on the configuration name that is specified
when the server is started.
For WebLogic, the agent name is comprised of:
Domain (data center) + cluster + instance (of WLS)
For WebSphere, the agent name is comprised of:
cell (domain) + process (instance of WAS)
When information is obtained, segments are separated by forward slashesfor
example:
medrec/MyCluster/MedRecServer
Any forward slashes in the segment name are converted to underscores. For example, if
a Domain is named Petstore/West, it is converted to Petstore_West.
Note: Introscope edits the information that is used to construct the agent name
according to the following rules:
characters such as pipes, colons, or percentage signs are replaced with underscores
names that begin with any character other than a letter are prepended with the
letter "A"
empty names are replaced with "UnnamedAgent" to distinguish them from the
"UnknownAgent" condition


To enable automatic agent naming
1. In the IntroscopeAgent.profile, set introscope.agent.agentAutoNamingEnabled to
true.
2. Make the following application server-specific changes:
For WebLogic, create an Introscope Startup Class. See Configuring a startup
class for WebLogic (see page 41).
For WebSphere, create an Introscope Custom Service. See Configuring a
custom service in WebSphere (see page 51).
For JBoss, create an XML file. See Configuring JBoss (see page 37).
Automatic agent naming and renamed agents
Using automatic agent naming, the agent always tries to obtain the most current
application-server-specific agent name. The agent periodically checks for a new name.
If a change to the application server configuration results in an agent name change, the
agent automatically renames itself. In the Investigator tree, the agent appears to
disconnect. The disconnected agent remains in the Investigator tree, and unmounts
automatically after the unmount time period has elapsed, or can be unmounted
manually.
The renamed agent reconnects to the Enterprise Manager and appears in the
Investigator tree. The agent logs these changes.
See Advanced automatic agent naming options (see page 127), for information on
configuring automatic agent naming properties for Enterprise Manager connection
delay, and rename checking interval time.
Advanced automatic agent naming options
There are several properties you can change to control automatic agent naming for your
environment.


Initial Enterprise Manager Connection Delay
When using the automatic agent naming feature, the agent waits up to a configurable
amount of time before connecting to the Enterprise Manager while trying to find agent
name information. The default delay is 120 seconds.
To change the delay value
2. Under the Agent Name section, configure the desired delay in the property
introscope.agent.agentAutoNamingMaximumConnectionDelayInSeconds.
Agent Rename Check Interval
When using the automatic agent naming feature, the agent periodically checks to see if
the naming information from the application server has changed. The default interval is
ten minutes.
To change this interval
2. Under the Agent Name section, configure the desired interval in the
introscope.agent.agentAutoRenamingIntervalInMinutes property.
Turning Off Agent Log File Automatic Naming
By default, when the agent name is found automatically, either by information provided
by a Java system property or application server, the log files associated with that agent
are named automatically using that same information. However, you can turn off this
automatic log naming, and continue to use the agent log name specified in the
To turn off agent log file automatic naming
2. Set the property, introscope.agent.disableLogFileAutoNaming, to a value of true.
Enabling cloned agent naming in clustered environments


Enabling cloned agent naming in clustered environments
If two agents exist with the same name monitoring the same host and process and are
not uniquely named by a user, the name is appended with a number. Cloned agent
naming enables you to correlate an agent with a particular application instance in a
clustered application.
You are running cloned agents if you:
are running agents that share a host, process, or Java Agent name with one or more
other agents, or
are running two or more agents that are using the same agent profile.
To enable cloned agent naming
1. Stop your managed application and the Java Agent.
2. Open the IntroscopeAgent.profile and set the following property to true:
introscope.agent.clonedAgent=true
4. Restart your managed application and the Java Agent.
Cloned agent naming scenario
With the Java Agent cloning property turned on, if you have four Java Agents, all named
AgentX, the Enterprise Manager names the agents AgentX-1, AgentX-2, AgentX-3 and
AgentX-4. If AgentX-1 disconnects and then reconnects, it will still use AgentX-1 as its
name. With this naming, you will never have more Java Agent names in the database
than the number of Java Agents originally cloned.
Configuring unique names for application instances
If you monitor multiple instances of an application on the same machine, you can
configure unique agent names explicitly.
To configure unique agent names
1. Create a separate agent profile for each application.
2. Uniquely name each agent in the agent profile.
3. Specify which agent profile each application should use.
Application triage map and the agent name


Application triage map and the agent name
The application triage map in the Workstation uses the agent name as part of several
unique identifiers when defining the front- and backends of applications, as well as for
storing information about application components in Introscope databases. If agent
names change some aspects of the application triage map may also change. For
example, during initial registration of an agent, the Enterprise Manager may assign a
duplicated agent name a unique name by appending %<sequence number> to the agent
name (e.g. MyAgent%1). If parts of the application triage map are relying on the
duplicated agent name for information, aspects of the map could change.
While this does not affect the proper functioning of either the agent or the Enterprise
Manager, it may reduce the overall capacity of your system; as such, CA Technologies
recommends users be aware of the naming conventions employed in their Introscope
environments. Users may want to designate specific agent names to sidestep this issue.


Chapter 7: Java Agent Monitoring and
Logging

While Introscope monitors your applications, Introscope can also monitor the health
and activity of the Java agent itself. This section contains information on monitoring the
health of the agent, as well as logging options for the Java agent.
Configuring agent connection metrics (see page 131)
Socket metrics (see page 132)
Configuring logging options (see page 136)
Managing ProbeBuilder Logs (see page 140)
Configuring agent connection metrics
By default, Introscope generates metrics on the connection status of agents connected
to an Enterprise Manager, which you can monitor. You can monitor agent connection
metrics to determine the current state of the connection between an agent and the
Enterprise Manager.
Agent connection metrics appear in the Workstation Investigator under the Enterprise
Manager process (the custom metric host):
Custom Metric Host (Virtual) \ Custom Metric Process(Virtual) \ Custom Metric Agent
(Virtual) (*SuperDomain*) \ Agents \ <HostName> \ <Agent Process Name> \ <Agent Name>
\ ConnectionStatus
Connection metrics have these values:
0No data about the agent is available
1agent is connected
2agent is slow to report
3agent is disconnected
An agent disconnecting also generates a "Whats Interesting" event. As with other
events, users can query for agent disconnects using the historical query interface. Agent
disconnect events are part of the data used in assessing application health in the
Overview tab in the Workstation Console.
Socket metrics


Once an agent disconnects from the Enterprise Manager, Introscope continues to
generate disconnected state metrics until the agent is timed out. When an agent times
out, no additional connection metrics are generated or reported to the Enterprise
Manager.
Follow these steps:
1. On the computer that the Enterprise Manager is installed on, open the
IntroscopeEnterpriseManager.properties file located in the
<Introscope_Home>/config directory.
2. Modify this property:
introscope.enterprisemanager.agentconnection.metrics.agentTimeoutInMinutes
The time increment is in minutes.
3. Save the IntroscopeEnterpriseManager.properties
Note: For information about Enterprise Manager properties, see the CA APM
Socket metrics
Socket and Secure Sockets Layer (SSL) metric collection is enabled by default in the
agent.
Note: JVMs using the Agent.NoRedef.jar do not have socket metrics reported. See
AutoProbe for WebSphere 6.1 (see page 47) for more information.
Restricting socket and SSL metric collection
Socket and SSL metric collection is enabled by default, but you can restrict some metric
collection to target more relevant information or scale back on overhead.
To restrict socket and SSL metric collection
2. In the Agent I/O Socket Metrics section, edit the property values to contain the list
of those hosts or ports for which metrics are required:
If an invalid host or port is included in the parameter values, a warning message is
written to the agent log and that value is ignored. If, as a result, the list contains no
entries, no restriction will apply.
introscope.agent.io.socket.client.hosts
A comma separated list of hosts; only 'client' socket metrics for specified hosts
will be generated. Hosts may be specified by name or textual representation of
IP address (in either IPv4 or IPv6 forms).
Socket metrics


Note: Duplicate host names are discarded. In cases where multiple host names
map to a single IP, only one of the names is retained. The property will,
however, match client connections with any of the set of synonymous names.
introscope.agent.io.socket.client.ports
A comma separated list of port numbers; only 'client' socket metrics for
specified ports will be generated.
Note: Duplicate ports are discarded.
introscope.agent.io.socket.server.ports
Comma separated list of port numbers, only 'server' socket metrics for
specified ports will be generated.
The above properties are dynamic; you do not need to restart your applications for
changes to take effect.
Fine-tuning socket and SSL metric collection
While you can restrict socket and SSL metric collection, as detailed in Restricting socket
and SSL metric collection (see page 132), you can further refine your metric collection by
turning on and off specific tracer groups. This helps to target the information you want
as well as reduce overhead costs.
To fine-tune socket and SSL metric collection
1. Open the java2.pbd file, located in the <Agent_Home>/wily/core/config directory.
2. In the I/O Socket Tracer Group or Network Tracer Group sections of java2.pbd,
determine the tracers you want to turn on or off, and comment or uncomment
them. For example, to suppress the metrics for input bandwidth, you comment out
the following:
#TraceOneMethodWithParametersIfFlagged: SocketTracing read
InputStreamBandwidthTracer "Input Bandwidth (Bytes Per Second)"
Note: Tracers with names ending with MappingTracer must not be commented out.
3. Save the java2.pbd file.
SSL, NIO, and Socket Tracing in the Application Triage Map
The application triage map in the Workstation Investigator can display instrumented
client socket connections. The agent records details about external systems being used
in transactions and sends this information to the Enterprise Manager. Then this
information appears graphically in the application triage map.
Socket metrics


The tracers in the appmap.pbd, located in the <Agent_Home>/wily/core/config
directory, extend existing SSL, NIO, and socket instrumentation. The tracers let the
agent send more component information to the application triage map. The tracers are
enabled by default. You can disable them by commenting out specific tracers in the
Trace Sockets and Trace NIO Sockets sections of the appmap.pbd.
Change the Component Name in the Application Triage Map
Component names, when displayed in the application triage map, include destination
host and port identities, in the same way socket client metric names are displayed. The
host identity in a component name can be configured to either a host name or a host IP
address. The default is the host name. You can change a displayed component name.
Follow these steps:
1. Open appmap.pbd, located in the <Agent_Home>/wily/core/config directory.
2. In the Trace Sockets and Trace NIO Sockets sections, select the tracers that you
want to modify.
3. Change {hostname} to {hostip} in the relevant tracers.
For example, the original tracers use the default {hostname}:
TraceOneMethodWithParametersIfFlagged: SocketTracing read AppMapSocketTracerBT
"System {hostname} on port {port}"
TraceOneMethodWithParametersIfFlagged: SocketTracing write
AppMapSocketTracerBT "System {hostname} on port {port}"
To display the host IP, use {hostip} instead:
TraceOneMethodWithParametersIfFlagged: SocketTracing read AppMapSocketTracerBT
"System {hostip} on port {port}"
TraceOneMethodWithParametersIfFlagged: SocketTracing write
AppMapSocketTracerBT "System {hostip} on port {port}"
4. Save the appmap.pbd file.
Socket metrics


Turning off socket and SSL metric collection
If collecting socket and SSL metrics is not required, they can be turned off entirely.
To turn off socket and SSL metric collection
1. Open either toggles-full.pbd or toggles-typical.pbd files (open the file you are using
in your deployment).
2. Comment out (place a pound or hash sign,#, at the beginning of the line)
SocketTracing:
3. Save the file you modified.
For more information about turning on and off tracer groups, in the toggles-full.pbd and
toggles-typical.pbd files, see Default tracer groups and toggles files (see page 90) and
Turning tracer groups on or off (see page 99).
Backwards Compatibility
Valid for: Tracers earlier than Release 9.0
The Java agent socket tracers are more configurable than in tracers in releases earlier
than 9.0. However, you can revert to earlier tracers (and disable the socket tracing
features and configuration options).
If you are using tracers earlier than Release 9.0, you can configure the Java agent:
To collect socket metrics and earlier tracers (see page 135).
To collect input and output bandwidth metrics (see page 136).
Collect Socket Metrics
Valid for: Tracers earlier than Release 9.0
To collect socket metrics using tracers, configure the Java agent.
Follow these steps:
1. Open either toggles-full.pbd or toggles-typical.pbd files (open the file that you are
using in your deployment).
2. Comment out (place a pound or hash sign,#, at the beginning of the line)
SocketTracing:
3. Uncomment ManagedSocketTracing:
TurnOn: ManagedSocketTracing
4. Save the file.
Configuring logging options


Collect Input and Output Bandwidth Metrics
Valid for: Socket tracers earlier than Release 9.0
If you are using socket tracers and you require input and output bandwidth metrics,
configure the Java agent.
Follow these steps:
2. Locate the Agent Socket Rate Metrics section and change the following property to
true:
introscope.agent.sockets.reportRateMetrics=true
Note: Only functions if ManagedSocketTracing is enabled and SocketTracing is
disabled.
When the Java Agent is installed on an application server, after the server starts up a log
directory is created here: <Agent_Home>/wily/logs. The application server process must
have full read/write/execute permissions on the <Agent_Home> directory. To
accomplish this, install the Java agent on the same operating system as the user who
runs the application server process. Or, install the Java agent as a different user, then
use the chmod command to grant the necessary permissions.
The Java agent has the option to run in verbose mode. Verbose mode records higher
levels of details about actions and agent interactions with your environment. This
information is useful in solving issues with your environment or agent functionality.
Introscope uses Log4J functionality for these functions. If you want to use other Log4J
functionality, please consult the Log4J documentation.
Running the agent in verbose mode
Running the agent in verbose mode records higher levels of information to the agent
log.
To run the agent in verbose mode


2. Modify this property, replacing the existing INFO with
VERBOSE#com.wily.util.feedback.Log4JSeverityLevel:
log4j.logger.IntroscopeAgent=VERBOSE#com.wily.util.feedback.Log4JSeverityLeve
l, console, logfile
Note: Changes to this property should take effect within one minute and do not
require the managed application to be restarted.
Redirecting agent output to a file
The property that controls the agent logging in verbose mode also controls where the
agent log is output and the location of this log file (see Running the agent in verbose
mode (see page 136) for more information).
To redirect agent output to a file
2. Find the property: log4j.logger.IntroscopeAgent
The options for this property are:
console: the information in the logfile is sent to the console
logfile: the information in the logfile is sent to a logfile. If this is selected, the
location of the log file is configured using the log4j.appender.logfile.File
property. See Changing the name or location of the agent logfile (see
page 138).
For example, if you wanted the agent to report in verbose mode to just a logfile, the
property would be set to:
l,logfile
If you wanted the agent to report to both a logfile and console, you would include
both logfile and console in the property.
Note: By default the agent log, IntroscopeAgent.log is written to the
<Agent_Home>/wily/logs directory. If you configured agent autonaming options,
the agent log files are also automatically named, as described in Agent log files and
automatic agent naming (see page 138).


Changing the name or location of the agent logfile
You can also change the location and name of a log file by modifying a property.
To change the name or location of the logfile
2. Locate the log4j.appender.logfile.File property.
If logfile was specified in the log4j.logger.IntroscopeAgent property, the location of
the log file is configured using the log4j.appender.logfile.File property. See step 2 in
Redirecting agent output to a file (see page 137) for more information.
Note: System properties (Java command line -D options) can be included as part of
the file name. For example, if a Java command starts with
-Dmy.property=Server1, then
log4j.appender.logfile.File=../../logs/Introscope-${my.property}.log is expanded to:
log4j.appender.logfile.File=../../logs/Introscope-Server1.log.
3. Set the location and name of the log file, using a fully qualified path to the new
location and file. For example:
log4j.appender.logfile.File=C:/Logs/AgentLog1.log
Agent log files and automatic agent naming
If you use the automatic agent naming functionality, by default the log files associated
with an agent are named automatically using the same information used to name the
agent.
Automatic agent naming affects the log file in the following way:
If the original name of the logfile does not end in .log, a period and log is added.
All characters that are not letters or digits will be replaced by underscores
If advanced Log4J functionality is used, the agent logfile automatic naming
capability might not work.
The following examples show how an agent logfile is named. The examples use an agent
name of DOM1//ACME42, where DOM1 is the WebLogic domain, and ACME42 is the
instance of the agent.


When an agent log file is created (named AutoProbe.log by default), if the agent name is
not yet available, a timestamp is included in the filename:
AutoProbe.20040416-175024.log
Once the agent name becomes available, the logfile is renamed using the agents
automatic name:
AutoProbe.DOM1_ACME42.log
You can disable automatic log naming - see Advanced automatic agent naming options
(see page 127) for more information.
Rolling up logs by date or size
You can roll up logs based on size or date, retaining a specified number of days of
information and purging the rest.
To roll up log files
1. Open the IntroscopeAgent.profile, and locate the Logging Configuration section.
2. Modify the following properties:
log4j.logger.IntroscopeAgent
log4j.appender.logfile.File
log4j.appender.console.layout
log4j.appender.console.layout.ConversionPattern
log4j.appender.logfile
log4j.appender.logfile.MaxFileSize
log4j.appender.logfile.MaxBackupIndex
Note: You must restart the managed application before changes to this property
take effect.
For example, the following configuration will keep up to three backup/rolled logs,
and each will be up to 2 kilobytes in size:
l, console, logfile
log4j.appender.logfile.File=logs/IntroscopeAgent.log
log4j.appender.console.layout=com.wily.org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{M/dd/yy hh:mm:ss a z} [%-3p]
[%c] %m%n
log4j.appender.logfile=com.wily.introscope.agent.AutoNamingRollingFileAppende
r
log4j.appender.logfile.MaxFileSize=2KB
log4j.appender.logfile.MaxBackupIndex=3
Managing ProbeBuilder Logs


Managing ProbeBuilder Logs
ProbeBuilder logs all the classes it sees, all the classes it instruments, as well as all the
classes it does not add instrumentation for the probes it added during the
instrumentation process and the PBDs it used. In addition, it logs the classes it did not
instrument due to skips.
Command-line ProbeBuilder and ProbeBuilder Wizard log name and location
The command-line ProbeBuilder and ProbeBuilder Wizard log file location is determined
by where you specify Java classes with the ProbeBuilder Wizard or with the
Command-Line ProbeBuilder. For a directory, the log file is located inside the
destination directory. For a file, the log file is located next to the destination file.
The ProbeBuilder log file is called:
<original-directory-or-original-file>.probebuilder.log
<original-directory> or <original-file> is the Java class location that you specify with the
ProbeBuilder Wizard or with the Command-Line ProbeBuilder.
Only the most recent log is kept; all previous log files are overwritten.
AutoProbe log name and location
AutoProbe will always attempt to log the changes it makes. By default the AutoProbe
log file is named AutoProbe.log.
To change the name or location of the AutoProbe log
2. Locate the introscope.autoprobe.logfile property and modify the log name and
location, using a fully qualified file path. Non-absolute names are resolved relative
to the location of the IntroscopeAgent.profile file.
Note: When loading the agent profile from a resource on a classpath, AutoProbe is
unable to write to the AutoProbe log file, because the IntroscopeAgent.profile file is
located within a resource.
You must restart the managed application before changes to this property take
effect.


Chapter 8: Configuring LeakHunter and
ErrorDetector

This section describes how to enable and configure LeakHunter and ErrorDetector.
LeakHunter (see page 141)
Enabling and disabling LeakHunter (see page 144)
Configuring LeakHunter properties (see page 144)
Running LeakHunter (see page 147)
Identifying potential leaks with collection IDs (see page 147)
LeakHunter log file (see page 148)
Using LeakHunter (see page 150)
ErrorDetector (see page 151)
Enabling ErrorDetector in the Java Agent (see page 152)
Configuring ErrorDetector options (see page 153)
Advanced error data capture (see page 154)
Defining new error types (see page 154)
Using ErrorDetector (see page 156)
LeakHunter
Introscope LeakHunter is an add-on component designed to help locate the source of
potential memory leaks by watching for collection instances that appear to be
increasing in size over timethat is, if the number of objects stored in the collection
increases over time.
Memory leaks that occur in programs that run for short periods (minutes or hours)
might not present a major issue. However for applications that are running 24 hours a
day (like a web site), a small memory leak can soon escalate into a big problem.
Introscope LeakHunter is designed to track information about a noticed memory leak by
being turned on, discovering collection classes, and then being turned off after
information is gathered. This method of using LeakHunter creates only a small,
temporary amount of overhead.
LeakHunter


How LeakHunter works
After you have enabled LeakHunter, you also define a timeout period during which
LeakHunter looks for new potential leaks. If you use AutoProbe, you simply restart the
managed application. If you use ProbeBuilder Wizard or Command-line ProbeBuilder,
you must re-instrument the application using the leakhunter.pbd (in addition to any
previously used PBD files).
If LeakHunter finds a collection that is growing in size over time, it:
identifies the collection with a unique ID
reports information about the collection to the Enterprise Manager as metric data
reports information about the collection to a log file on the agent machine
continues to track and report data for that collection
Should LeakHunter notice that a collection no longer appears to be leaking, it reports
that fact to both the Enterprise Manager and the log file, but continues tracking and
reporting data for that collection.
LeakHunter continues looking for potential leaks and monitors already identified
potential leaks until the timeout period expires. After a timeout period expires,
LeakHunter stops looking for potential leaks in newly allocated collections, and only
continues checking collections that are already identified as potential leaks. This
significantly reduces LeakHunter overhead and allows additional monitoring of the
potential leaks. LeakHunter continues monitoring the identified potential leaks until the
managed application is shut down.
To find the source of a memory leak, you can either browse the metric data in the
Introscope Investigator, or view the log file.
What LeakHunter tracks in Java
Introscope LeakHunter tracks these collections in a Java implementation:
Implementations of java.util.Collection:
java.util.ArrayList
java.util.LinkedList
java.util.TreeSet
java.util.HashSet
java.util.LinkedHashSet
java.util.Vector
java.util.Stack
LeakHunter


Implementations of java.util.Map:
java.util.Map
java.util.SortedMap
java.util.HashMap
java.util.TreeMap
java.util.IdentityHashMap
java.util.Hashtable
java.util.Properties
java.util.LinkedHashMap
What LeakHunter does not track
Introscope LeakHunter does not track:
a leak not caused by a collection.
custom collection implementations or other data structures with an increasing
number of references.
a leaking collection that is not instrumented.
In addition to the above, LeakHunter does not track the following in a Java
implementation:
any subclasses created for collections described in What LeakHunter tracks in Java
(see page 142). However, you can update the ProbeBuilder Directive (PBD) file to
get this information. See the leakhunter.pbd file for more information.
Note: If you are using Application Server AutoProbe, LeakHunter does not automatically
track collections allocated by the application server. To track these collections, you must
statically instrument the application server, or use JVM AutoProbe.
System and version requirements
Introscope LeakHunter has the same system requirements as the Java Agent.
By default, LeakHunter is not enabled after installation. You must enable LeakHunter to
use its functionality.
Enabling and disabling LeakHunter


Enabling and disabling LeakHunter
LeakHunter is run as an agent extension, so no class paths need to be updated. By
default, LeakHunter is not enabled after installation. You must enable LeakHunter to use
its functionality.
To enable LeakHunter
1. Open the agent profile, IntroscopeAgent.profile.
2. Under the LeakHunter Configuration heading, locate the property
introscope.agent.leakhunter.enable, and enter a value of true.
3. Save the agent profile.
4. Open *.typical.pbl or *.full.pbl (open the file you have listed in the
IntroscopeAgent.profile property introscope.autoprobe.directivesFile ) and
uncomment leakhunter.pbd.
Note: When using ProbeBuilder Wizard, copy the leakhunter.pbd file to the
<Agent_Home>\wily\core\config\hotdeploy directory.
Important! By default, agent extensions such as LeakHunter are located and referenced
from the <Agent_Home>\wily\core\ext directory. However, you can change the location
of the agent extension directory in the agent profile. If you change the location of the
\ext directory, be sure to move the contents of the \ext directory as well.
To disable LeakHunter
2. Under the LeakHunter Configuration heading, locate the property
introscope.agent.leakhunter.enable, and enter a value of false.
Configuring LeakHunter properties
LeakHunter configuration properties are located in the agent profile,
To configure LeakHunter


2. Configure the following LeakHunter properties as desired:
introscope.agent.leakhunter.logfile.location
Specifies the location of LeakHunter.log file. The file name is relative to the
<IntroscopeAgent.profile> directory. If the property is commented out or left
blank, no log file is written.
The default value is logs/LeakHunter.log.
introscope.agent.leakhunter.logfile.append
Specifies whether to replace the log file (value of false) or append an existing
log file (value of true) on application restart.
The default value is false.
introscope.agent.leakhunter.leakSensitivity
Specifies the sensitivity level for detecting memory leaks. A higher leak
sensitivity setting will result in more potential leaks being reported and a lower
sensitivity will result in fewer potential leaks reported.
The property value must be an integer from 1-10.
The default sensitivity level is 5.
introscope.agent.leakhunter.timeoutInMinutes
Specifies the period of time in minutes during which LeakHunter looks for new
potential leaks. The property value must be a non-negative integer. A value of
zero means no timeout.
The default value is 120 minutes.


introscope.agent.leakhunter.collectAllocationStackTraces
Specifies whether to collect allocation stack trace information. Turning on this
option has the potential to create higher system CPU usage and memory.
Changes to this property take effect immediately and do not require the
managed application to be restarted.
The default value is false.
introscope.agent.leakhunter.ignore.n
Specifies the particular collections that you want LeakHunter to ignore.
For Generic collections, use a syntax that includes the generic type
qualification, for example: System.Collections.Generic.List`1
Changes to this property take effect immediately and do not require the
The default values for these properties (where n=0-4) are:
introscope.agent.leakhunter.ignore.0=org.apache.taglibs.standard.lang.jst
l.*
introscope.agent.leakhunter.ignore.1=com.bea.medrec.entities.RecordEJB_xw
cp6o__WebLogic_CMP_RDBMS
introscope.agent.leakhunter.ignore.2=net.sf.hibernate.collection.*
introscope.agent.leakhunter.ignore.3=org.jnp.interfaces.FastNamingPropert
ies
introscope.agent.leakhunter.ignore.4=java.util.SubList
3. Save changes to the agent profile.
Important! The IntroscopeAgent.profile contains properties that control which packages
are ignored by LeakHunter. These properties are enabled by default. If you comment
out these properties, exceptions are reported in the agent logs.
Ignoring collections that cause poor performance
Collections whose size() method executes in time proportional to the number of objects
in the Collection will have poor performance. In other words, if the collection is such
that the size() method takes longer and longer to execute (for example, a badly
implemented LinkedList where to get the size of the list we traverse through each
element of the list and count), this will have negative effects on application
performance.
Such collections should be ignored, using the ignore property in
IntroscopeAgent.profile, as shown in Configuring LeakHunter properties.
Running LeakHunter


Running LeakHunter
LeakHunter is run as an agent extension.
To run LeakHunter
Using JVM AutoProbe: after LeakHunter has been enabled, restart the application.
Using Command-Line ProbeBuilder: re-instrument the managed application using
the leakhunter.pbd (and any previously used .pbds). Restart the managed
application.
Using ProbeBuilder Wizard: run ProbeBuilder Wizard and select the leakhunter.pbd
from the custom pbd list. Restart the managed application after implementing new
.jars for use.
Identifying potential leaks with collection IDs
LeakHunter identifies each potential leak with a unique collection ID that can be used to
correlate metric data from the Investigator tree with data in the log file, as well as
provide stable names across applications.
The unique collection IDs have a specific syntax, based on one of these types:
<method>-<4 digit hash code>#<unique number>
<field>-<4 digit hash code>#<unique number>
<method>: the name of the method where the collection was allocated
<field>: name of field to which the collection was assigned
<4 digit hash code>: the hash code of the full name of the class containing the
method or field name
#<unique number>: number appended to potential leaks with the same method and
hash code to ensure unique collection IDs during the run of the agent.
The collection ID for a potential leak will be stable across runs of the application.
Examples of these collection IDs are:
theLookupTable-6314#1
getLoginID-1234#1
getLoginID-1234#2
getLoginID-1234#3
verifyCart-5678#1
verifyCart-0012#1
LeakHunter log file


LeakHunter log file
The LeakHunter log file contains information about potential leaks identified by
Introscope LeakHunter in your managed application. Each entry contains information
about a potential leak. The LeakHunter log file includes entries for four scenarios:
when a potential leak is first identifiedsee Potential leak first identified log entry
(see page 148)
when an identified leak appears to have stopped leakingsee Identified potential
leak stops leaking log entry (see page 149)
when a previously identified leak appears to have started leaking againsee
Identified potential leak is leaking again log entry
when a LeakHunter timeout has occurredsee LeakHunter timeout log entry
Potential leak first identified log entry
This type of LeakHunter log entry contains information about a potential leak when it is
first identified:
Current timestamp (when written to the log)
Collection ID
Class of the collection
Allocation Method of the collection
Allocation time of the collection
Allocation stack trace of the collection
Field name to which the collection was assigned
Current size of the collection
LeakHunter log file


Note: The current size of a leaked collection recorded in the LeakHunter log file is not
dynamically updated. The log file identifies the size of a leak when the leak is first
identified. To see up-to-date information about the size of a leaked connection, click the
Leak tab in the Introscope Workstation.
Example: Log file entry if a potential leak is detected
The following example illustrates an entry in the Java log file when a potential leak is
first identified:
5/2/09 9:55:06 AM PDT
Potential leak identified
Assigned ID: testInst-2604#1
Collection Class: java.util.Vector
Allocation Method: sonOfLH_test.testInst()
Allocation Timestamp: 5/2/09 9:54:21 AM PDT
Allocation Stack Trace:
Unknown
Field Name(s):
sonOfLH_test.v3
sonOfLH_test.v4
sonOfLH_test.v5
Current Size: 44
Identified potential leak stops leaking log entry
This type of LeakHunter log entry contains information about a potential leak that has
stopped leaking:
Collection ID
Example: Log file entry if a potential leak stops leaking
The following example illustrates an entry in the Java log file when a potential leak
appears to have stopped leaking:
4/27/10 1:18:12 PM PDT
Potential leak no longer appears to be leaking
Assigned ID: createNewInstance-2815#3
Collection Class: java.util.HashMap
Current Size: 70
Using LeakHunter


This type of entry contains information about a potential leak that has started leaking
again:
Collection ID
Example: Log file entry if a potential leak resumes leaking
The following example illustrates an entry in the Java log file when a potential leak
appears to have started leaking again:
4/27/10 1:21:42 PM PDT
Potential leak appears to be leaking again
Assigned ID: createNewInstance-2815#3
Collection Class: java.util.HashMap
Current Size: 79
LeakHunter timeout log entry
This type of LeakHunter log entry contains the number of potential leaks that will
continue to be tracked.
Example: Log file entry when a timeout occurs
LeakHunter timeout occurred at 4/27/10 1:32:12 PM PDT
LeakHunter will only continue to track the 3 potential leaks
Using LeakHunter
For more information on how to use LeakHunter, see the CA APM Workstation User
Guide.
ErrorDetector


ErrorDetector
Introscope ErrorDector allows application support personnel to detect and diagnose the
cause of serious errors, which can prevent users from completing web transactions.
These kinds of application availability issues typically result in error messages to the
user such as "404 Not Found" and others, yet the error message lacks specific
information that IT personnel need to isolate the root cause of the problem. Introscope
ErrorDetector allows you to monitor these serious errors as they occur in live
applications, determine the frequency and nature of the errors, and deliver specific
information about the root cause to developers.
ErrorDetector is the only application management solution that helps ensure superior
user experiences and improves transaction integrity by enabling root cause analysis of
serious application errors.
Introscope ErrorDetector allows IT teams to:
Determine the frequency of anomalous transactions
Determine whether logged exceptions affect users
See exactly where errors occurred within the transaction path
Obtain the information needed to reproduce, diagnose and eliminate serious errors
Introscope ErrorDetector is integrated with Introscope, allowing you to monitor errors
in the Introscope Workstation. When application errors do occur, you can also use the
Live Error Viewer to examine detailed information about each error.
Types of errors
CA Technologies has defined a set of criteria to describe "serious" errors, based on
information contained in the J2EE specifications. ErrorDetector considers both errors
and exceptions to be errors. The most common type of error is a thrown exception.
Some examples of common errors are:
HTTP errors (404, 500, etc.)
Note: Occasionally, HTTP 404 errors originate in a web server instead of an
application server. If this occurs, ErrorDetector cannot detect the web server error
through the agent.
SQL statement errors
network connectivity errors (timeout errors)
backend errors (for example, cant send a message through JMS, can't write a
message to the message queue)
Enabling ErrorDetector in the Java Agent


What CA Technologies considers to be important errors might differ from what you
consider important errors. If you do not consider some of the errors ErrorDetector
tracks to be important, you can choose to ignore them. If there are additional errors you
want to track, you can use error tracers to create new directives to trace them.
How ErrorDetector works
Introscope includes a ProbeBuilder Directive (PBD) file called errors.pbd with the agent
installation. The tracers in this PBD capture serious errors.
ErrorDetector is installed automatically with your agent installation. Once ErrorDetector
is installed, you configure Introscope to use the errors.pbd and enable ErrorDetector
functionality. If you are using ProbeBuilder Wizard or Command-line ProbeBuilder, you
must re-instrument the application using the errors.pbd (in addition to any previously
used PBD files).
The agent collects error information as defined in the errors.pbd file.
From the Workstation, you can view:
error metric data in the Investigator
live errors in the Live Error Viewer
error details in the Error Snapshot, which shows component-level information on
how the error occurred
ErrorDetector is integrated with Transaction Tracer, enabling you to see exactly why and
how serious errors occurred within the context of the transaction path. Moreover, all
errors and transactions are persisted to Transaction Event Database, enabling you to
spot trends through analysis of historical data.
Introscope defines a transaction as the invocation and processing of a service. In the
context of a web application, it is the invocation and processing of a URL sent from a
web browser. In the context of a web service, it is the invocation and processing of a
SOAP message.
Enabling ErrorDetector in the Java Agent
The property introscope.agent.errorsnapshots.enable should be set to true in the
IntroscopeAgent.profile to enable the Java Agent to capture error data. By default, the
Java Agent is enabled to capture error data.
To enable/disable ErrorDetector data capture in the Java Agent
Configuring ErrorDetector options


2. Confirm the introscope.agent.errorsnapshots.enable property to true.
Note: To disable ErrorDetector, set this property to false.
3. If you are using ProbeBuilder Wizard, copy the errors.pbd file to the
<EM_Home>/config/custompbd directory, and re-instrument your applications.
Configuring ErrorDetector options
You can configure ErrorDetector to limit the maximum number of errors the agent
sends to the Enterprise Manager, or specify the errors to ignore.
Enabling ErrorDetector captures error data without incurring much overhead. The
out-of-the-box throttle is set at 10 errors per 15 seconds. If you want to capture more
errors during this time period, you can increase the throttle, but be prepared to incur
more overhead.
To change the ErrorDetector throttle (optional)
2. Enter a new value for the introscope.agent.errorsnapshots.throttle property.
Note: Changes to this property take effect immediately and do not require the
You can configure the agent to ignore errors you dont want to track. The information
you specify to "tag" the error can be the exact error message, or any portion of the
message, along with the "wildcard" asterisk character.
To ignore certain errors (optional)
2. For the introscope.agent.errorsnapshots.ignore property, define the value to be
whatever information you think will identify that type of error.
For example, the following ignore property would ignore any errors with the term
"IOException" found anywhere within it:
introscope.agent.errorsnapshots.ignore.0=*IOException*
Advanced error data capture


3. To ignore additional errors, add additional ignore properties sequentially. For
example, to ignore two types of errors, the properties would look like this:
introscope.agent.errorsnapshots.ignore.0=*IOException*
introscope.agent.errorsnapshots.ignore.1=*HTTP Error Code *500*
4. Save changes to the agent profile.
Note: Changes to this property take effect immediately and do not require the
Advanced error data capture
Although ErrorDetector captures many common error types by default, CA Technologies
provides options for customizing the error detection mechanism to suit your needs.
You can use the following error-related tracers to create ProbeBuilder Directives (PBDs)
to capture errors.
ExceptionErrorReporter reports standard exceptions.
ThisErrorReporter reports a current object as an error.
HTTPErrorCodeReporter captures HTTP error codes and associated error messages.
MethodCalledErrorReporter reports if specific methods get called.
You should place any new directives you create with these tracers in the errors.pbd file
in the <Agent_Home>/wily directory.
Defining new error types
Errors are defined for ErrorDetector using PBDs. There are several special tracers that
check for the presence of an error and capture (or construct) the error message. By
placing these tracers at the right points, you can teach ErrorDetector about a new kind
of error in your application or its infrastructure.
ExceptionErrorReporter
The ExceptionErrorReporter tracer can be used to check for exceptions being thrown
from the instrumented method. If an exception is thrown, this tracer treats it as an error
and gets the error message from the exception. This is the most common definition of
an error.
Defining new error types


In order to capture error messages, the ExceptionErrorReporter tracer must be used
with a "...WithParameters" directive. For example:
TraceOneMethodWithParametersOfClass: com.bank.CustomerAccount getBalance
ExceptionErrorReporter "CustomerAccount:Errors Per Interval"
This directive specifies that any exception that is thrown from the getBalance() method
on the CustomerAccount constitutes an error.
Note that you must use a "...WithParameters" directive to increment the errors per
interval metric, but you only have to specify a "...WithParameters" directive once for
any method to make the parameters available for all tracers on that method. For
example, you can specify:

TraceOneMethodWithParametersOfClass: com.myClass myMethod BlamePointTracer

This directive makes the parameters for the com.myClass myMethod method available
to other tracers, including the ExceptionErrorReporter tracer.
MethodCalledErrorReporter
The MethodCalledErrorReporter tracer is used on methods where the very act of the
method being called means that an error has occurred. For example:
TraceOneMethodOfClass: com.bank.CheckingAccount cancelCheck
MethodCalledErrorReporter "CustomerAccount:Canceled Checks Per Interval"
This directive specifies that whenever the cancelCheck() method is called (for any
reason), this is an error. The error message simply states the class and method that was
called.
If you do not know which methods may throw an exception or error, try using the
ThisErrorReporter tracer.
ThisErrorReporter
The ThisErrorReporter tracer is similar to the MethodCalledErrorReporter, but it
constructs the error message by calling toString() on the instrumented objects. This
tracer is most useful to put on the constructor of an exception class. For example:
TraceOneMethodWithParametersOfClass: ezfids.util.exception.EasyFidsException [set
the init variable for your book] ThisErrorReporter
"Exceptions|{packageandclassname}:Errors Per Interval"
Note: To capture error messages, the ThisErrorReporter tracer must be used with a
"...WithParameters" directive.
Using ErrorDetector


The directives specify that whenever the constructor ("init" or ".ctor") of an
InvalidPINException is called, this constitutes an error. The error message is determined
by calling toString() on the InvalidPINException which generally returns the error
message that the application developer specified.
This tracer is good to use when you have a custom error management system based on
your own exception types.
Note: Introscope cannot instrument any code in the java.* packages so placing this
tracer on java.lang.Exception or java.sql.SQLException does not work.
HTTPErrorCodeReporter
The HTTPErrorCodeTracer tracer reports error codes from Servlets and JSPs. It is a per
interval counter that counts incidents of:
HTTP response codes 400 or higher.
javax.servlet.http.HttpServletResponse subclass invocations of sendError or
setStatus, for codes 400 or higher in Java environments.
See errors.pbd for example usage.
Using error tracer directives with caution
Be judicious in using the tracers described in the previous sections. The best practice is
to incur the overhead associated with error tracing to report only the most serious
problems, such as unrecoverable problems arising from backend systems.
The default errors.pbd is designed to report serious errors, while minimizing overhead
as much as possible. Overuse of error tracing, for instance, applying
ExceptionErrorReporter to every monitored method can result in a high volume of "false
positives." For example, if a user enters "California" in a numeric field, this may cause a
NumberFormatException, which you would not want to report as a serious problem.
Using ErrorDetector
For more information on how to use ErrorDetector, see the CA APM Workstation User
Guide.


Chapter 9: Configuring Boundary Blame

This section describes default Java Agent blame reporting behaviors, and related
configuration options.
Understanding Boundary Blame (see page 157)
Using URL groups (see page 157)
Using Blame tracers (see page 163)
Understanding Boundary Blame
Blame Technology lets you work in a managed Java Application to view metrics at the
front and backends of your application. This capability, referred to as boundary blame,
lets users triage problems.
How Introscope detects backends varies depending on the application. For database
activity, Introscope uses the SQL Agent to detect backends. If the SQL Agent is
unavailable, Introscope detect backend activity because backends such as client/server
databases, JMS servers, and LDAP servers are accessed through a socket. If you have
Oracle backends and do not use the Introscope SQL Agent, see Boundary Blame and
Oracle backends (see page 119).
For information about how boundary blame is presented in the Introscope Investigator,
see the CA APM Workstation User Guide.
Using URL groups
You can use URL Groups to monitor browser response time for sets of requests whose
path prefix begins with a string you define. The path prefix is the portion of the URL that
follows the hostname. For example, in this URL:
http://burger1.com/testWar/burgerServlet?ViewItem&category=
11776&item=5550662630&rd=1
the path prefix is:
/testWar
You can define a URL group for any useful category of requests that can be derived from
a URLs path prefix. For example, depending on the form of your application URLs, you
could define URL groups for each customer your application supports, for each major
application, or for sub-applications. This enables you to monitor performance in the
context of committed service levels, or for mission-critical portions of your application.
Using URL groups


Example: How URL group properties are defined
The following example is an excerpt from a Java Agent profile, showing how URL Groups
are defined:
introscope.agent.urlgroup.keys=alpha,beta,gamma
introscope.agent.urlgroup.group.alpha.pathprefix=/testWar
introscope.agent.urlgroup.group.alpha.format=foo {host} bar {protocol} baz {port}
quux {query_param:foo} red {path_substring:2:5} yellow {path_delimited:/:0:1} green
{path_delimited:/:1:4} blue {path_substring:0:0}
introscope.agent.urlgroup.group.beta.pathprefix=/nofilterWar
introscope.agent.urlgroup.group.beta.format=nofilter foo {host} bar {protocol} baz
{port} quux {query_param:foo} red {path_substring:2:5} yellow {path_delimited:/:0:1}
green {path_delimited:/:1:4} blue {path_substring:0:0}
introscope.agent.urlgroup.group.gamma.pathprefix=/examplesWebApp
introscope.agent.urlgroup.group.gamma.format=Examples Web App
Defining keys for URL groups
The property introscope.agent.urlgroup.keys defines a list of the IDs, or keys, of all of
your URL Groups. The key for a URL Group is referenced in other property definitions
that declare an attribute of the URL group. The following example defines the keys for
three URL Groups:
introscope.agent.urlgroup.keys=alpha,beta,gamma
If you define URL Groups so some URLs fall into multiple groups, the order in which you
list the keys for the URL Groups in the property is important. The URL Group with the
narrower membership should precede the URL Group with broader membership. For
example, if the IP Group with key alpha has the path prefix
/examplesWebApp and the URL Group with key delta has the path prefix
/examplesWebApp/cleverones, delta should precede alpha in the keys parameter
Defining membership of each URL group
The property introscope.agent.urlgroup.group.groupKey.pathprefix specifies a pattern
against which the path prefix of a URL is matched, defining which requests fall within
the URL Group.
Using URL groups


Example: Mapping a group key to a URL path prefix
This property definition assigns all requests in which the path portion of the URL starts
with /testWar to the URL Group whose key is alpha:
introscope.agent.urlgroup.group.alpha.pathprefix=/testWar
Requests that match the specified pathprefix include:
http://burger1.com/testWar/burgerServlet?ViewItem&category=
11776&item=5550662630&rd=1
http://burger1.com/testWar/burgerServlet?Command=Order&item=5550662630
Example: Creating URL groups by application path
A company that provides call center services could monitor response time for functional
areas by setting up a URL Group for each application function. If customers access
services with this URL:
http://genesystems/us/application_function/
where application_function corresponds to applications such as OrderEntry, AcctService,
and Support, the pathprefix property for each URL group would specify the appropriate
application_function.
Note: You can use the asterisk symbol (*) as a wildcard in pathprefix properties.
Define the name for a URL group
The property introscope.agent.urlgroup.group.groupKey.format determines the names
under which response time metrics for a URL group whose key is groupKey appear in the
Introscope Workstation.
Typically, the format property is used to assign a text string as the name for a URL. The
following example causes metrics for the URL Group with key alpha to appear in the
Workstation under the name Alpha Group:
introscope.agent.urlgroup.group.alpha.format=Alpha Group
Advanced naming techniques for URL groups (optional)
You can derive a URL Group name from request elements such as the server port, the
protocol, or from a substring of the request URL. This is useful if your application
modules are easily differentiated by inspecting the request. This section describes
advanced forms of the format property.
Using URL groups


Using host as a URL group name
To organize metrics for a URL group under names that reflect the hostname of the HTTP
server associated with requests, define the format parameter like this:
introscope.agent.urlgroup.group.alpha.format={host}
When format={host}, statistics for these requests would appear under the metric names
us.mybank.com and uk.mybank.com respectively:
https://us.mybank.com/mifi/loanApp......
https://uk.mybank.com/mifi/loanApp.....
Using protocol as a URL group name
To organize statistics for a URL group under names that reflect the protocol associated
with requests, define the format parameter like this:
introscope.agent.urlgroup.group.alpha.format={protocol}
When format={protocol} statistics are grouped in the Investigator under metric names
that correspond to the protocol portion of request URLs. For example, statistics for
these requests would appear under the metric name https:
https://us.mybank.com/cgi-bin/mifi/scripts......
https://uk.mybank.com/cgi-bin/mifi/scripts......
Using port as a URL group name
To organize statistics for a URL group under names that reflect the port associated with
requests, define the format parameter like this:
introscope.agent.urlgroup.group.alpha.format={port}
When format={port}, statistics are grouped under names that correspond to the port
portion of request URLs. For example, statistics for these requests would appear under
the name 9001.
https://us.mybank.com:9001/cgi-bin/mifi/scripts......
https://uk.mybank.com:9001/cgi-bin/mifi/scripts......
Using URL groups


Using parameter as a URL group name
To organize statistics for a URL group in Investigator under metric names that reflect the
value of a parameter associated with requests, define the format parameter like this:
introscope.agent.urlgroup.group.alpha.format={query_param:param}
When format={query_param:param} statistics are grouped in Investigator under metric
names that correspond to value of the parameter specified. Requests without
parameters are listed under <empty>. For example, given this parameter definition:
introscope.agent.urlgroup.group.alpha.format=
{query_param:category}
Statistics for these requests would appear under the metric name "734"
http://ubuy.com/ws/shoppingServlet?ViewItem&category=734
&item=3772&tc=photo
http://ubuy.com/ws/shoppingServlet?ViewItem&category=734
&item=8574&tc=photo
Using a substring of the request path as a URL group name
To organize statistics for a URL group under names that reflect a substring of the path
portion of request URLs, define the format parameter like this:
{path_substring:m:n}
where m is the index of the first character, and n is one greater than the index of the
last character. String selection operates like the java.lang.String.substring() method. For
example, given this setting:
{path_substring:0:3}
Statistics for the following request would appear under the metric node /ht:
http://research.com/htmldocu/WebL-12.html
Using URL groups


Using a delimited portion of the request path as a URL group name
To organize statistics for a URL group under names that reflect a character-delimited
portion request URL path, define the format parameter like this:
{path_delimited:delim_char:m:n}
where delim_char is the character that delimits the segments in the path, m is the index
of the first segment to select, and n is one greater than the index of the last segment to
select. For example, given this setting:
introscope.agent.urlgroup.group.alpha.format={path_delimited:/:2:4}
statistics for the requests of this form:
http://www.buyitall.com/userid,sessionid/pageid
would appear under the metric name /pageid
Note that:
a delimiter character, in this example, the forward slash (/) counts as a segment
the segment count starts at 0
The segments as delimited by the slash character in the URL example are:
0=/, 1=userid,sessionid, 2=/, and 3=pageid
You can specify multiple delimiters as necessary. For example, given this setting:
introscope.agent.urlgroup.group.alpha.format={path_delimited:/,:3:4}
statistics for requests of the form shown above would appear under the metric name
sessionid with the segments as delimited by the slash and the comma in the URL
example are:
0=/, 1=userid, 2=, 3=sessionid, 4=/, and 5=pageid
Using multiple naming methods for URL groups
You can combine multiple naming methods in a single format string, as shown below:
introscope.agent.urlgroup.group.alpha.format=red {host} orange {protocol} yellow
{port} green {query_param:foo} blue {path_substring:2:5} indigo
{path_delimited:/:0:1} violet {path_delimited:/:1:4} ultraviolet
{path_substring:0:0} friend computer
Using Blame tracers


Running the URLGrouper
URLGrouper is a command-line utility that analyzes a web server log file in Common
format. Using URLGrouper gives you a starting point for defining your own URL Groups.
Note: You can use the URLGRouper utility to analyze your Web server log file.
URLGrouper outputs a set of property settings for potential URL Groups, based on the
contents of the Web server log file. The asterisk symbol (*) can be used with
URLGrouper as a wildcard.
To run URLGrouper
1. Open a command shell.
2. Enter this command
java -jar urlgrouper.jar logfile
where logfile is the full path to your web server log file.
3. Property definitions for a set of URL Groups are output to STDOUT.
4. To configure the proposed URL Groups, copy the property statements produced by
URL Grouper into the IntroscopeAgent.profile.
Using Blame tracers
You can use tracers to explicitly mark the frontends and backends in your application.
For more information, see Using Blame Tracers to mark blame points (see page 117).


Chapter 10: Configuring Transaction Trace
Options

This section has information about default Transaction Tracing behaviors and related
configuration options.
New Mode of Transaction Tracing (see page 165)
Controlling automatic Transaction Tracing behavior (see page 168)
Cross-Process Transaction Tracing (see page 170)
Extending transaction trace data collection (see page 170)
Configuring Component Stall Reporting (see page 173)
New Mode of Transaction Tracing
CA APM Introscope 9.1 introduced a new agent transaction architecture and new
tracers. This new mode for transaction tracing replaces the transaction blame stacks
that the previous tracers used in earlier releases. The new mode implementation
optimizes computation and processing to improve the agent performance.
CA APM Introscope 9.1 allows you to configure and run agents with transactional blame
stacks as before. This legacy mode of the agent is fully supported but agent
enhancements in future releases will be implemented for the new mode only. The
following capabilities are not available when running the agent in legacy mode.
Agent CPU usage and response time optimizations in version 9.1
Dynamic Instrumentation for the .NET agent
SQL Agent properties
introscope.agent.sqlagent.sql.artonly
introscope.agent.sqlagent.sql.turnoffmetrics
Important! Running legacy tracers with the CA APM agent in new mode is known as
mixed mode configuration. Do not run in mixed mode because memory consumption
and service interruption can occur.


If you are running legacy mode tracers, run the agents in legacy mode. The following
message is written to the agent log to inform customers when they have incompatible
legacy tracers.
An agent tracer using legacy APIs has been detected. Running legacy tracers with
the agent in new mode is not recommended. Please contact support who can refer
you to documentation on how to upgrade your legacy tracers. In the interim, please
configure the agent to use legacy mode or use the pre-configured version of the
legacy agent package. For example, the legacy package for the Oracle WebLogic agent
on UNIX is IntroscopeAgentFiles-Legacy-NoInstallerx.x.x.xweblogic.unix.tar
Revert to run the agent in legacy mode until you are running new mode tracers.
If you are using the following out-of-the-box extensions, configure your agents to run in
legacy mode:
CA APM for IBM Websphere Portal
CA APM for IBM CICS Transaction Gateway
CA APM for IBM z/OS
CA APM for CA SiteMinder Web Access Manager
CA APM Integration for CA LISA
Configuring the Agent to Use Legacy Mode Transaction Tracing
Converting to the legacy mode transaction tracing only applies to versions of CA APM
that have the new mode. CA APM versions before version 9.1 do not have new mode.
Two options are available to configure agents to use the legacy mode:
Deploy the preconfigured legacy agent packages. These packages are available in
the files-only, no-installer, packages of the agent. For example, the legacy package
for the Oracle WebLogic agent on UNIX is:
IntroscopeAgentFiles-Legacy-NoInstaller<version_number>weblogic.unix.tar
Manually configure for the legacy mode.
Follow these steps:
1. Stop the monitored application.
2. Archive and delete existing log files in the <Agent_Home>/logs directory to prepare
for new logs.
3. Back up the existing .pbl and .pbd files in the <Agent_Home>/core/config directory.
4. Back up the existing <Agent_Home>/core/config/IntroscopeAgent.profile.
5. Copy the legacy .pbl and .pbd files from the <Agent_Home>/examples/legacy
directory to the <Agent Home>/core/config directory.


6. Open <Agent_Home>/core/config/IntroscopeAgent.profile and make the following
changes:
Add property introscope.agent.configuration.old=true
Update the agent property introscope.autoprobe.directivesFile to point to the
appropriate legacy .pbl and .pbd files that have been copied over. For example,
replace spm.pbl with spm-legacy.pbl
7. Restart the monitored application.
To configure a particular CA APM extension to use the legacy mode in a typical
installation, see the .pbl and .pbd files that are listed in the following table.

Extension Name Legacy typical pbl or pbd file
CA APM for Oracle Weblogic Server ppweblogic-legacy.pbd
spm-legacy.pbl
CA APM for Oracle Weblogic Portal powerpackforweblogicportal-legacy.pbl
spm-legacy.pbl
CA APM for Oracle Service Bus OSB-typical-legacy.pbl
spm-legacy.pbl
CA APM for IBM Websphere Portal powerpackforwebsphereportal.pbl
spm-legacy.pbl
CA APM for IBM Websphere MQ webspheremq-legacy.pbl
CA APM for IBM Websphere Process
Server /Business Process Management
wps-legacy.pbd
wesb-legacy.pbd
spm-legacy.pbl
CA APM for TIBCO BusinessWorks tibcobw-typical-legacy.pbl
spm-legacy.pbl
CA APM for Software AG Webmethods
Integration Server
webmethods-legacy.pbl
spm-legacy.pbl
CA APM for CA SiteMinder Web Access
Manager
smwebagentext.pbd
CA APM for IBM CICS Transaction
Gateway
PPCTGTranTrace.pbd
PPCTGServer-typical.pbd
PPCTGClient-typical.pbd
CA APM for IBM z/OS zos-full.pbl
CA APM for CA LISA lisa-typical.pbl
CA APM for SYSVIEW CTG_ECI_Tracer_For_SYSVIEW-legacy.pbd
HTTP_Tracer_For_SYSVIEW-legacy.pbd
WS_Tracer_For_SYSVIEW-legacy.pbd

Controlling automatic Transaction Tracing behavior


Automatic Transaction Tracing enables historical analysis of potentially problematic
transaction types without explicitly running Transaction Traces. Introscope offers two
types of automatic Transaction Tracing:
Transaction Trace sampling that is enabled by default, based on your URL groupings
Configurable automatic trace sampling that gathers trace information regardless of
URL groupings
Transaction Trace component clamp
Introscope sets a component clamp to limit the size of traces. The default is 5,000
components. When this limit is reached, a warning appears in the log, and the trace
stops.
You can clamp a component transaction that exceeds expected component counts, for
example, when a servlet executes hundreds of object interactions and backend SQL
calls. Without the clamp, Transaction Tracer views it as one transaction, continuing
infinitely. Without a clamp in place in certain extreme situations, the JVM can run out of
memory before the trace completes.
The property for clamping transactions is located in the IntroscopeAgent.profile file:
introscope.agent.transactiontrace.componentCountClamp=5000
Traces producing clamped components are marked with an asterisk. For more
information about viewing these traces, see the CA APM Workstation User Guide.
Important! If the Transaction Trace component clamp size is increased, the memory
required for Transaction Traces can increase. For more information, see the CA APM


Transaction trace sampling
Transaction trace sampling is enabled by default. As appropriate you can disable this
behavior. For more information on Transaction Trace properties, see Transaction tracing
(see page 307).
When you configure automatic trace sampling, you specify the number of transactions
to trace, during a time interval you specify.
Note: These properties are located in the Enterprise Manager properties file. Before
changing the defaults for the sampling.perinterval and sampling.interval properties,
consider the potential for increased load in the Enterprise Manager with higher
sampling rates. The Enterprise Manager will push this configuration to all agents
connected to it. You can also configure these properties in an agent. Configuring these
properties in the agent will overwrite the configuration set by the Enterprise Manager
for an individual agent.
To configure automatic trace sampling, modify these properties
introscope.agent.transactiontracer.sampling.enabled
Set to false to disable Transaction Trace sampling. The default value is true.
introscope.agent.transactiontracer.sampling.perinterval.count
Specifies the number of transactions to trace, during the interval you specify.
The default number of transactions is 1.
introscope.agent.transactiontracer.sampling.interval.seconds
Specifies the length of time to trace the number of transactions you specify.
The default interval is every 2 minutes.
Agent Heap Sizing
The agent uses Java heap memory to store collected data. You can view the agent GC
heap usage in the GC Heap overview.
The following information applies to the heap usage:
For a highly utilized heap, you may need to increase the heap allocation when you
install the agent.
For deep or long lasting transactions on your monitored applications, the
Transaction Trace sampling can require more heap memory.
Note: From more information about GC heap usage, see the CA APM Workstation User
Guide. If you are operating a high-performance CA Introscope environment, contact CA
Professional Services for the appropriate agent JVM heap settings.
Cross-Process Transaction Tracing


More information:
Transaction Trace component clamp (see page 168)
Transaction trace sampling (see page 169)

Cross-Process Transaction Tracing
The CA Introscope Java agent can trace transactions that cross JVM boundaries on
WebLogic Server or WebSphere. The environment must be comprised of compatible
versions of the application server by the same vendor. Cross-process transaction tracing
is supported for synchronous transactions, for instance servlets to EJBs, and
asynchronous transactions.
Important! Cross-process transaction tracing is available only with CA Introscope 9.0
agents at a minimum.
More information:
Enable cross-process tracing in WebLogic Server (see page 41)
Enable cross-process tracing in WebSphere (see page 55)

Extending transaction trace data collection
You can configure the Introscope Transaction Tracer to obtain additional information,
including User ID data for Servlet and JSP invocations, and other transaction trace data
such as HTTP request headers, request parameters, and session attributes. To capture
this information, you must define the criteria in the IntroscopeAgent.profile.
About User ID data
To configure the Java Agent to identify User IDs for Servlet and JSP invocations, you
must first obtain information on how your managed application specifies user IDs. The
Application Architect who developed the managed application can probably provide this
information.


Introscope Transaction Tracer can identify User IDs from managed applications that
store User IDs in one of these ways:
HttpServletRequest.getRemoteUser()
HttpServletRequest.getHeader (String key)
HttpSession.getValue (String key), where returned object is either a String
representing the UserID, or an Object whose toString() returns to the UserID
If your managed application stores User IDs using one of these methods, see Configuring
Agent to collect additional transaction trace data (see page 171), to configure Java
Agent settings to collect User ID data.
About servlet request data
Using Introscope, you can collect transaction trace data that matches user-configurable
parameters. For example, you can specify the agent to collect transaction trace data for
transactions that contain the User-Agent HTTP request header.
Introscope can record this servlet request information:
request headers
request parameters
session attributes
To record this servlet request information for your managed application, see
Configuring Agent to collect additional transaction trace data (see page 171) to
configure Java Agent settings to collect this data.
Configuring Agent to collect additional transaction trace data
You can configure the Java agent to collect additional transaction trace data such as
User ID, HTTP request headers, HTTP request parameters, or HTTP session attributes.
Follow these steps:
2. Locate the Transaction Tracer properties under the Transaction Tracer
Configuration heading.


Collecting user id data
To configure the Java agent to identify User IDs
1. Configure the properties that correspond to the method your managed application
uses to store User IDs.
Note: Ensure that only one set of properties are not commented, or the wrong
properties might be used.
2. For HttpServletRequest.getRemoteUser(), uncomment the property:
introscope.agent.transactiontracer.userid.method=HttpServletRequest.getRemote
User
3. For HttpServletRequest.getHeader (String key), uncomment the following pair of
properties, and define a key string for the second property:
introscope.agent.transactiontracer.userid.method=HttpServletRequest.getHeader
introscope.agent.transactiontracer.userid.key=<application defined key string>
4. For HttpSession.getValue (String key), uncomment the following pair of properties,
and define a key string for the second property:
introscope.agent.transactiontracer.userid.method=HttpServletRequest.getValue
introscope.agent.transactiontracer.userid.key=<application defined key string>
Collecting servlet request data
To record servlet request information such as HTTP request headers and parameters
1. To specify the HTTP request headers for which to collect transaction trace data,
uncomment this property, and specify the HTTP request header(s) to track, in a
comma-separated list:
#introscope.agent.transactiontracer.parameter.httprequest.headers=User-Agent
2. To specify the HTTP request parameters for which to collect transaction trace data,
uncomment this property and specify the HTTP request parameter(s) to track, in a
comma-separated list:
#introscope.agent.transactiontracer.parameter.httprequest.parameters=paramete
r1,parameter2
3. To specify the HTTP session attributes for which to trace data, uncomment this
property and specify the HTTP session attribute(s) to track, in a comma-separated
list, for example:
#introscope.agent.transactiontracer.parameter.httpsession.attributes=cartID,d
eptID
Configuring Component Stall Reporting


An Application Performance Management stall is when there is no response from
probed components for a defined length of time. By default, the APM Introscope agents
detect this condition and report Stall metrics.
Each time the agent checks for stalls, all the topmost instrumented components on the
method stack are checked. When a component is stalled, a stalled metric and a stall
snapshot are created. A stall metric is also created for each component that subscribes
to downstream monitoring.
The stall metrics are created first for the topmost instrumented components on the
method stack. These metrics are created for the components that take longer than the
introscope.agent.stalls.thresholdseconds value.
Downstream Subscriber Component Stalls
A component that subscribes to downstream monitoring is stalled when it calls a stalled
downstream component.
By default, the following components subscribe to downstream stall monitoring:
FrontendTracer probed Components
BackendTracer probed Components
WebServiceBlamepointTracer@Servicelevel probed components
WebServiceBlamepointTracer@Opertationlevel probed components
Example
A frontend component calls a Webservice which calls a backend component that is
stalled.
Frontend--> WebService-->Backend
The stall metrics are created for the backend, the Webservice, and the Frontend
components because all of them subscribe to downstream monitoring by default.


Upstream Inherited Component Stalls
The stall checker first examines the top member of the component stack to see if it is
stalled. If the top component is not stalled, the stall checker looks for stalled parent
components. When a parent is stalled, a stall metric is created for that component and
each upstream parent in the stack.
If a component M1() calls multiple components M2() that each take only a few seconds,
a stall metric could be reported for M1().
Example:
The stall threshold is set at 15 seconds. Method M1() calls method M2() in a loop 100
times. M2() never stalls because it takes only 2 seconds to respond each time. However,
M1() is eventually stalled.
Disabling the capture of stalls as Events
By default, Introscope captures transaction stalls as events in the Transaction Event
database, and generates stall metrics from the detected events. The stall metrics are
generated for the first and last method in the transaction. Users can view stall Events
and associated metrics in the Workstation Historical Event Viewer.
Note: Generated stall metrics are always available, but stall events are only visible if
Introscope Error Detector is installed. Stalls are stored as ordinary errors, and are visible
in the Errors TypeView, or in the historical query viewer by querying for
"type:errorsnapshot".
To disable the capture of stalls as events, change the stall threshold, or change the
frequency with which the agent checks for stalls. Use the following properties:
introscope.agent.stalls.thresholdseconds specifies the minimum threshold response
time at which time a transaction is considered stalled.
introscope.agent.stalls.resolutionseconds specifies the frequency that the agent
checks for stalls.


Chapter 11: Configuring the Introscope SQL
Agent

This section has instructions for configuring Introscope SQL Agent.
The SQL Agent overview (see page 175)
The SQL Agent files (see page 176)
SQL statement normalization (see page 176)
Turn Off Statement Metrics (see page 184)
SQL metrics (see page 184)
The SQL Agent overview
The SQL agent reports detailed database performance data to the Enterprise Manager.
The SQL agent provides visibility into the performance of individual SQL statements in
your application by tracking the interaction between your managed application and
your database.
In the same way that the Java agent monitors applications, the SQL agent monitors SQL
statements. The SQL agent is non-intrusive, monitoring the application or database with
very low overhead.
To provide meaningful performance measurements down to the individual SQL
statement level, the SQL agent summarizes performance data by stripping out
transaction-specific data and converting the original SQL statements into
Introscope-specific normalized statements. Since normalized statements do not include
sensitive information, such as credit card numbers, this process also protects the
security of your data.
For example, the SQL agent converts this SQL query:
SELECT * FROM BOOKS WHERE AUTHOR = 'Atwood'
to this normalized statement:
SELECT * FROM BOOKS WHERE AUTHOR = ?
The SQL Agent files


Similarly, SQL agent converts this SQL update statement:
INSERT INTO BOOKS (AUTHOR, TITLE) VALUES ('Atwood', 'The Robber Bride')
INSERT INTO BOOKS (AUTHOR, TITLE) VALUES (?, ?)
Note: Only text within quotation marks ('xyz') is normalized.
Metrics for normalized statements are aggregated and can be viewed in the JDBC node
of the Workstation Investigator.
The SQL Agent files
The SQL agent is included in all agent installations. The files providing SQL Agent
functionality are:
wily/core/ext/SQLAgent.jar
wily/core/config/sqlagent.pbd
Note: By default, agent extensions like the SQLAgent.jar file are installed in the
wily/core/ext directory. You can change the location of the agent extension directory
with the introscope.agent.extensions.directory property in the agent profile. If you
change the location of the /ext directory, be sure to move the contents of the /ext
directory as well.
SQL statement normalization
Some applications may generate an extremely large number of unique SQL statements.
If technologies like EJB 3.0 are in use, the likelihood of long unique SQL statements
increases. Long SQL statements can contribute to a metric explosion in the agent,
leading to poor performance as well as other system problems.
How poorly written SQL statements create metric explosions
In general, the number of SQL agent metrics should approximate the number of unique
SQL statements. If your SQL agent is showing a large and increasing number of unique
SQL metrics even though your application uses a small set of SQL statements, the
problem could be in how the SQL statement is written.
There are several common situations in which SQL statements can cause metric
explosions. For example, SQL statements that include embedded comments, create
temporary tables, or insert lists of values can unintentionally create unique metric
names each time they execute.


Example: Comments in SQL statements
One common reason for metric explosion is caused by how comments are used in SQL
statements. For example, if you have a SQL statement like this:
"/* John Doe, user ID=?, txn=? */ select * from table..."
the SQL agent creates a metric with the comment as part of the metric name:
"/* John Doe, user ID=?, txn=? */ select * from table..."
The comment embedded in the SQL statement is useful for the database administrator
to see who is executing what query and the database executing the query ignores them.
The SQL agent, however, does not parse the comment string when it captures the SQL
statement. Therefore, for each unique user ID, the SQL agent creates a unique metric,
potentially causing a metric explosion.
This problem can be avoided by putting the SQL comment in single quotes. For example:
"/*' John Doe, user ID=?, txn=? '*/ select * from table..."
The SQL agent then creates the following metric where the comment no longer causes a
unique metric name:
"/* ? */ select * from table..."
Example: Temporary tables or automatically generated table names
Another potential cause of metric explosion can be the result of applications that
reference temporary tables or tables that have automatically generated names in SQL
statements. For example, if you open the Investigator to view the metrics under
Backends|{backendName}|SQL|{sqlType}|sql. you might see a metric similar to this:
SELECT * FROM TMP_123981398210381920912 WHERE ROW_ID = ?
This SQL statement is accessing a temporary table that has a unique identifier appended
to the table name. The additional digits that are appended to the TMP_ table name
create a unique metric name each time the statement is executed, causing a metric
explosion.


Example: Statements that generate lists of values or insert values
Another common cause of metric explosion are SQL statements that generate lists of
values or do mass modification of values. For example, assume you have been alerted to
a potential metric explosion and your investigation brings you to a review of this SQL
statement:
#1 INSERT INTO COMMENTS (COMMENT_ID, CARD_ID, CMMT_TYPE_ID,
CMMT_STATUS_ID,CMMT_CATEGORY_ID, LOCATION_ID, CMMT_LIST_ID, COMMENTS_DSC,
USER_ID,LAST_UPDATE_TS) VALUES (?, ?, ?, ?, ?, ?, ?, "CHANGE CITY FROM CARROLTON,TO
CAROLTON, _ ", ?, CURRENT)
In studying the code, you notice that "CHANGE CITY FROM CARROLTON, TO CAROLTON,
_ " generates an array of cities.
Similarly, if you are investigating a potential metric explosion, you might review a SQL
statement similar to this:
CHANGE COUNTRY FROM US TO CA _ CHANGE EMAIL ADDRESS FROM TO BRIGGIN @ COM _ "
In studying the code, you notice CHANGE COUNTRY results in a long list of countries. In
addition, the placement of the quotes for countries results in people's e-mail addresses
getting inserted into SQL statements, creating unique metrics that could be the source
of the metric explosion.
SQL statement normalization options
To address long SQL statements, the SQL Agent includes the following normalizers for
use:
Default SQL statement normalizer (see page 178)
Custom SQL statement normalizer (see page 179)
Regular expression SQL statement normalizer (see page 180)
Command-line SQL statement normalizer (see page 184)
Default SQL statement normalizer
The standard SQL statement normalizer is on by default in the SQL agent. It normalizes
text within single quotation marks ('xyz'). For example, the SQL agent converts this SQL
query:
SELECT * FROM BOOKS WHERE AUTHOR = 'Atwood'
SELECT * FROM BOOKS WHERE AUTHOR = ?
Metrics for normalized statements are aggregated and can be viewed in the
Workstation Investigator.


Custom SQL statement normalizer
The SQL Agent lets you add extensions for performing custom normalization. To do so,
you create a JAR file containing a normalization scheme that is implemented by the SQL
agent.
To apply a SQL statement normalizer extension
1. Create an extension JAR file.
Note: The entry point class for the SQL normalizer extension file has to implement
com.wily.introscope.agent.trace.ISqlNormalizer interface.
Making a JAR extension file involves creating a manifest file that contains specific
keys for the SQL normalizer extension, which are detailed in step 2 below. However,
for your extension to work, other general keys are required. These keys are the type
you would use to construct any extension file. The extension file you create relates
to database SQL statement text normalization, for example metrics under the
Backends|{backendName}|SQL|{sqlType}|{actualSQLStatement} node. The
{actualSQLStatement} is normalized by the SQL normalizer.
2. Place the following keys in the manifest of the created extension:
com-wily-Extension-Plugins-List:testNormalizer1
Note: The value of this key can be anything. In this instance, testNormalizer1 is
used as an example. Whatever you specify as the value of this key, use it in the
following keys as well.
com-wily-Extension-Plugin-testNormalizer1-Type: sqlnormalizer
com-wily-Extension-Plugin-testNormalizer1-Version: 1
com-wily-Extension-Plugin-testNormalizer1-Name: normalizer1
Should contain the unique name of your normalizer, for example normalizer1.
com-wily-Extension-Plugin-testNormalizer1-Entry-Point-Class:
<Thefully-qualified classname of your implementation of ISQLNormalizer>
3. Place the extension file you created in the <Agent_Home>/wily/core/ext directory.
4. In the IntroscopeAgent.profile, locate and set the following property:
introscope.agent.sqlagent.normalizer.extension
Set the property to the com-wily-Extension-Plugin-{plugin}-Name from your created
extensions manifest file. The value of this property is not case sensitive. For
example:
introscope.agent.sqlagent.normalizer.extension=normalizer1
Important! This is a hot property. Changes to the extension name will result in
re-registration of the extension.


5. In the IntroscopeAgent.profile, you can optionally add the following property to set
the error throttle count:
introscope.agent.sqlagent.normalizer.extension.errorCount
For more information about errors and exceptions, see Exceptions (see page 180).
Note: If the errors thrown by the custom normalizer extension exceed the error
throttle count, the extension is disabled.
7. Restart your application.
Exceptions
If the extension you created throws an exception for one query, the default SQL
statement normalizer uses the default normalization scheme for that query. When this
happens, an ERROR message is logged, saying an exception was thrown by the
extension, and a DEBUG message is logged with stack trace information. However, after
five such exceptions are thrown, the default SQL statement normalizer disables your
created extension and stops attempting to use the created extension for future queries
until the normalizer is changed.
Null or empty strings
If the extension you created returns a null string or empty string for a query, the
StatementNormalizer uses the default normalization scheme for that query and logs an
INFO message saying the extension returned a null value. However, after five such null
or empty strings have been returned, the StatementNormalizer stops logging messages,
but will attempt to continue to use the extension.
Regular expression SQL statement normalizer
The SQL Agent ships with an extension that normalizes SQL statements based on
configurable regular expressions (regex). This file, RegexNormalizerExtension.jar, is
located in the <Agent_Home>/wily/core/ext directory.
For examples on how to use the regular expression SQL statement normalizer, see
Regular expression SQL statement normalizer examples (see page 182).
To apply the regular expressions extension
2. Locate and set the following properties:
introscope.agent.sqlagent.normalizer.extension=RegexSqlNormalizer
Specifies the name of the SQL normalizer extension that will be used to
override the preconfigured normalization scheme. When enabling the regular
expressions extension, set this property to RegexSqlNormalizer.


introscope.agent.sqlagent.normalizer.regex.keys=key1
This property specifies the regex group keys, which are evaluated in the order
they are listed. This property is required to enable the regular expressions
extension. There is no default value.
introscope.agent.sqlagent.normalizer.regex.key1.pattern=A
This property specifies the regex pattern that is used to match against the SQL
statements. All valid regular expressions allowed by the java.util.Regex
package classes can be used here. This property is required to enable the
regular expressions extension. There is no default value.
introscope.agent.sqlagent.normalizer.regex.key1.replaceFormat=B
This property specifies the replacement string format. All valid regex allowed
by the java.util.Regex package classes can be used here. This property is
required to enable the regular expressions extension. There is no default value.
introscope.agent.sqlagent.normalizer.regex.matchFallThrough=false
If this property is set to true, SQL strings are evaluated against all the regex key
groups. The implementation is chained. Hence, if the SQL strings match
multiple key groups, the normalized SQL output from group1 is fed as input to
group2, and so on.
If the property is set to false, as soon as a key group matches the SQL string,
the normalized SQL output from that group is returned. The MatchFallThrough
property does not enable or disable the extension.
For example, if you had a SQL string like: Select * from A where B, you would
set the following properties:
introscope.agent.sqlagent.normalizer.regex.keys=key1,key2
introscope.agent.sqlagent.normalizer.regex.key1.replaceFormat=X
introscope.agent.sqlagent.normalizer.regex.key2.pattern=B
introscope.agent.sqlagent.normalizer.regex.key2.replaceFormat=Y
If introscope.agent.sqlagent.normalizer.regex.matchFallThrough is false, then the
SQL is normalized against key1 regex. Output from that regex will be Select * from X
where B. This SQL will be returned.
If introscope.agent.sqlagent.normalizer.regex.matchFallThrough is true, then the
SQL is normalized against key1 regex first. The output from that regex is Select *
from X where B. This output is then fed to key2 regex. The output from key2 regex is
Select * from X where Y. This will be the SQL returned.
Note: This property is not required to enable the regular expressions extension.
introscope.agent.sqlagent.normalizer.regex.key1.caseSensitive=false
This property specifies whether the pattern match is case sensitive. The default
value is false. This property is not required to enable the regular expressions
extension.


introscope.agent.sqlagent.normalizer.regex.key1.replaceAll=false
If this property is set to false, it will replace the first occurrence of the matching
pattern in the SQL with the replacement string. If this property is set to true, it
will replace all occurrences of the matching pattern in the SQL with the
replacement string.
For example, if you have a SQL statement like Select * from A where A like Z,
you would set the properties as follows:
introscope.agent.sqlagent.normalizer.regex.key1.replaceFormat=X
If introscope.agent.sqlagent.normalizer.regex.key1.replaceAl is false, it will result in
a normalized SQL statement: Select * from X where A like Z.
If introscope.agent.sqlagent.normalizer.regex.key1.replaceAl is true, it will result in
a normalized SQL statement: Select * from X where X like Z.
The default value is false. This property is not required to enable the regular
expressions extension.
Note: If none of the regular expression patterns match the input SQL, the
RegexNormalizer will return a null string. The statement normalizer will then use
the default normalization scheme.
Important! All properties listed above are hot, meaning changes to these properties
take effect once you have saved the IntroscopeAgent.profile.
Regular expression SQL statement normalizer examples
The three examples below can help you understand how to implement the regular
expression SQL statement normalizer.
Example 1
Here is a SQL query before regular expression SQL statement normalization:
INSERT INTO COMMENTS (COMMENT_ID, CARD_ID, CMMT_TYPE_ID,CMMT_STATUS_ID,
CMMT_CATEGORY_ID, LOCATION_ID, CMMT_LIST_ID,COMMENTS_DSC, USER_ID, LAST_UPDATE_TS)
VALUES(?, ?, ?, ?, ?, ?,?, CHANGE CITY FROM CARROLTON, TO CAROLTON, _ ", ?, CURRENT)
Here is the desired normalized SQL statement:
INSERT INTO COMMENTS (COMMENT_ID, ...) VALUES (?, ?, ?, ?, ?, ?,?, CHANGE CITY FROM
( )


Here is the configuration needed to the IntroscopeAgent.profile file to result in the
normalized SQL statement shown above:
introscope.agent.sqlagent.normalizer.regex.matchFallThrough=true
introscope.agent.sqlagent.normalizer.regex.keys=key1,key2
introscope.agent.sqlagent.normalizer.regex.key1.pattern=(INSERT INTO
COMMENTS \$COMMENT_ID,)(.*)(VALUES.*)''(CHANGE CITY FROM \\().*(\$)
introscope.agent.sqlagent.normalizer.regex.key1.replaceFormat=$1 ...) $3$4 $5
introscope.agent.sqlagent.normalizer.regex.key2.pattern='[a-zA-Z1-9]+'
introscope.agent.sqlagent.normalizer.regex.key2.replaceAll=true
introscope.agent.sqlagent.normalizer.regex.key2.replaceFormat=?
Example 2
Here is a SQL query before regular expression SQL statement normalization:
SELECT * FROM TMP_123981398210381920912 WHERE ROW_ID =
Here is the desired normalized SQL statement:
SELECT * FROM TMP_ WHERE ROW_ID =
Here is the configuration needed to the IntroscopeAgent.profile file to result in the
normalized SQL statement shown above:
introscope.agent.sqlagent.normalizer.regex.key1.pattern=(TMP_)[1-9]*
introscope.agent.sqlagent.normalizer.regex.key1.replaceFormat=$1
Example 3
If you want to normalize a SQL statement like: Select .... ResID1, CustID1 where
ResID1=.. OR ResID2=.. n times OR CustID1=.. OR n times, you could set the properties
like this:
introscope.agent.sqlagent.normalizer.regex.keys=default,def
introscope.agent.sqlagent.normalizer.regex.default.pattern=(ResID)[1-9]
introscope.agent.sqlagent.normalizer.regex.default.replaceAll=true
introscope.agent.sqlagent.normalizer.regex.default.replaceFormat=$1
introscope.agent.sqlagent.normalizer.regex.default.caseSensitive=true
introscope.agent.sqlagent.normalizer.regex.def.pattern=(CustID)[1-9]
introscope.agent.sqlagent.normalizer.regex.def.replaceAll=true
introscope.agent.sqlagent.normalizer.regex.def.replaceFormat=$1
introscope.agent.sqlagent.normalizer.regex.def.caseSensitive=true
Turn Off Statement Metrics


Command-line SQL statement normalizer
If the regular expression SQL normalizer is not in use, and you have SQL statements that
enclose values in the where clause with double quotes (" "), use the following
command-line command to normalize your SQL statements:
-DSQLAgentNormalizeDoubleQuoteString=true
Important! You can use the regular expressions SQL normalizer instead of this command
to normalize SQL statements in double quotes. See Regular expression SQL statement
normalizer (see page 180) for more information.
Turn Off Statement Metrics
Some applications can generate large numbers of unique SQL statements, causing a
metric explosion in the SQL agent. You can turn off SQL statement metrics in the SQL
agent.
Note: When you turn off statement metrics, the backend or top-level JDBC metrics are
not lost.
To turn off statement metrics
1. Open the sqlagent.pbd file and locate the SQL statements. For example:
TraceOneMethodWithParametersIfFlagged: SQLAgentStatements
executeQuery(Ljava/lang/String;)Ljava/sql/ResultSet; DbCommandTracer
"Backends|{database}|SQL|{commandtype}|Query|{sql}"
2. Remove {sql} from the trace directives you want to turn off. For example:
TraceOneMethodWithParametersIfFlagged: SQLAgentStatements executeQuery(Ljava/
lang/String;)Ljava/sql/ResultSet; DbCommandTracer
"Backends|{database}|SQL|{commandtype}|Query"
3. Save the sqlagent.pbd file.
The SQL statement metrics in the SQL agent are turned off.
SQL metrics
The SQL Agent metrics appear under the Backends node in the Introscope Workstation
Investigator. SQL statement metrics can be found under the
Backends|<backendName>|SQL node.
Note: Average Response Time (ms) will only display queries that return a data reader,
i.e. queries executed via the ExecuteReader() method. This metric represents the
average time spent in the data readers Close() method.
SQL metrics


Metric types specific to SQL data include:
Connection CountThe number of live connection objects in memory.
A connection is opened when a drivers connect() method is invoked, and closed
when the connection invocation is closed via the close() method. The SQL Agent
maintains weak references to Connections in a Set. When the Connection objects
are garbage collected, the counts reflect the changes.
Average Result Processing Time (ms)The average processing time of a query.
This metric represents the average time spent processing a ResultSet from the end
of the executeQuery() call to the invocation of the ResultSet's close() method.
Note: Instrumented XADataSources may not report commit or rollback metrics. Other
instrumented DataSources may not report commit or rollback metrics unless those
metrics contain data.


Chapter 12: Enabling JMX Reporting

This section contains information about enabling the Java agent to report JMX data.
Java Agent JMX Support (see page 187)
Default JMX Metric Conversion Process (see page 188)
Using Primary Key Conversion to Streamline JMX Metrics (see page 189)
Managing Metric Volume with JMX Filters (see page 190)
Configure WebSphere and WebLogic to Use JMX Reporting (see page 191)
Enable JSR-77 Data and View JMX Metrics on WAS (see page 192)
Java Agent JMX Support
CA Introscope can collect management data that application servers or Java
applications expose as JMX-compliant MBeans. CA Introscope presents the JMX data in
the Investigator metric tree. CA Introscope supports the collection of JMX information
using the following application servers:
Glassfish - No additional configuration is required for Glassfish to report JMX
metrics.
JBoss - Instructions for configuring JBoss to report JMX metrics are in Configure
JBoss to Use the Java Agent (see page 37)
Tomcat - Instructions for configuring Tomcat to report JMX metrics are in Configure
Apache Tomcat to Use the Java Agent (see page 36)
WebLogic - You can Configure WebSphere and WebLogic to Use JMX Reporting (see
page 191)
WebSphere - You can Configure WebSphere and WebLogic to Use JMX Reporting
(see page 191)
Note: CA Introscope supports any MBean built to the Sun JMX specification. For more
information about the Sun JMX specification, see Java Management Extensions.
CA Introscope converts the JMX data to the CA Introscope metric format and displays
it in the Investigator under the following node:
<Domain>|<Host>|<Process>|AgentName|JMX|
Default JMX Metric Conversion Process


Introscope Support for WebLogic JMX Metrics
WebLogic provides the following MBeans as sources of JMX metrics:
RuntimeServiceMBean: per-server runtime metrics, including active effective
configuration
DomainRuntimeServiceMBean: domain-wide runtime metrics
EditServiceMBean: allows user to edit persistent configuration
CA Introscope polls only the RuntimeServiceMBean for the following reasons:
RuntimeServiceMBean supports local access (an efficiency issue).
RuntimeServiceMBean contains most of the data that is expected to be relevant.
Default JMX Metric Conversion Process
CA Introscope, by default, converts JMX Metrics for display in the Investigator. CA
Introscope converts an MBean if:
You use WebLogic
You have not configured primary key conversion to streamline JMX metrics (see
page 189).
Note: If you specify primary keys that no MBeans match, CA Introscope uses the
default conversion method.
In the default conversion method, CA Introscope displays both the name and the value
of the attribute, and lists the pairs alphabetically in the metric tree.
<Domain>|<Host>|<Process>|AgentName|JMX|<domain
name>|<key1>=<value1>|<key2>=<value2>:<metric>
For example, given a WebLogic MBean with these characteristics:
Domain name: WebLogic
Key/Value pairs: category=server, type=jdbc
Metric names: connections
Using Primary Key Conversion to Streamline JMX Metrics


If no primary keys are specified in the IntroscopeAgent.profile property
introscope.agent.jmx.name.primarykeys, the MBean attributes convert to the following
CA Introscope metric:
<Domain>|<Host>|<Process>|AgentName|JMX|Weblogic|category=server|type=jdbc:c
onnections
The key/value pairs display alphabetically in the CA Introscope metric.
Using Primary Key Conversion to Streamline JMX Metrics
You can optionally configure the order in which metrics appear under the JMX node.
You can define a Primary Key in the agent profile. The Primary key identifies the
ObjectName of the MBean.
If you do not configure primary key conversion, CA Introscope converts the JMX data
(see page 188). With the default conversion, metrics are listed alphabetically under the
JMX node in Investigator. This method of converting JMX data to metrics results in
streamlined metric names. You control the order of key/value pair information in the
generated metrics.
The behavior is configured in the introscope.agent.jmx.name.primarykeys property in
the agent profile. Values in the primarykeys property specify the parts of an MBeans
JMX ObjectName that uniquely identify an MBean. For example, an ObjectName for a
WebLogic MBean contains the following keys:
A Type key that specifies the kind of MBean.
A Name key that specifies the name of the resource the MBean represents.
The key/value pairs in an ObjectName can vary for different types of MBeans.
CA Introscope converts and presents the MBeans that the
introscope.agent.jmx.name.primarykeys property value identifies according to these
rules:
Only the key value information is displayed, not the key name.
Values are ordered in the sequence defined in the primarykeys property.
Values are case-sensitive.
For example, given a WebLogic MBean with these characteristics:
Domain name: WebLogic
MBean ObjectName key/value pairs: category=server, type=jdbc
Metric names: connections
Managing Metric Volume with JMX Filters


If you configure:
introscope.agent.jmx.name.primarykeys=type,category the connections attribute
appears in the Investigator tree in this structure:
<IntroscopeDomain>|<Host>|<Process>|AgentName|JMX|Weblogic|jdbc|server:connection
s
Note: WebLogic 9.0 does not have universally available primary keys. Use the key/value
pair metric naming convention found in the Default Conversion Method. As a result, the
JMX Metric tree for WebLogic 9.0 has a different structure.
More information:
Default JMX Metric Conversion Process (see page 188)

Managing Metric Volume with JMX Filters
Defining JMX filters determines what JMX MBean information is collected and displayed
in CA Introscope. If no filters are set, the agent reports all JMX MBean information to
the Enterprise Manager, increasing system overhead.
Filters are set in the introscope.agent.jmx.name.filter property in the agent profile,
IntroscopeAgent.profile. Filters are keywords, entered as comma-separated strings in
the property. CA Introscope supports filter strings that contain the asterisk (*) and
question mark (?) wildcard characters.
CA Introscope matches the filter strings to JMX-generated metrics. If it finds a match,
the metrics that match are reported to CA Introscope.
To limit the volume of metrics that are returned, define filter strings as narrowly as
possible. For example, define a filter string that matches an MBean attribute and which
exists on multiple MBeans. Metrics from each of those MBeans are reported. If you are
only interested in an attribute on selected MBeans, you can qualify the attribute name
with the MBean name in your filter string.
For example, assume you want to capture the MessagesCurrentCount attribute value
for the JMSDestinationRuntime MBean.
If the fully qualified metric name for MessagesCurrentCount is:
*SuperDomain*|host-name|Process|Agent-name|JMX|comp-1|
JMSDestinationRuntime|comp-2:MessagesCurrentCount
define introscope.agent.jmx.name.filter in the IntroscopeAgent.profile as:
JMX|comp-1|JMSDestinationRuntime|comp-2:MessagesCurrentCount
Configure WebSphere and WebLogic to Use JMX Reporting


JMX filters for WebLogic
In the IntroscopeAgent.profile file for WebLogic, the following keywords are already
defined:
ActiveConnectionsCurrentCount
WaitingForConnectionCurrentCount
PendingRequestCurrentCount
ExecuteThreadCurrentIdleCount
OpenSessionsCurrentCount
Configure WebSphere and WebLogic to Use JMX Reporting
How you configure CA Introscope to support JMX depends upon the application server
you use. This information describes how you can configure CA Introscope to collect and
present JMX data from WebLogic Server and WebSphere.
Follow these steps:
1. Shut down the managed application.
2. For WebSphere agents only, in IntroscopeAgent.profile set
introscope.agent.jmx.enable to true.
Note: The default value is false in the WebSphere agent profile.
3. In IntroscopeAgent.profile, configure primary keys.
For WebLogic 9.0, comment out:
introscope.agent.jmx.name.primarykeys
For other WebLogic versions, uncomment:
Note: If you modify the value of the property, the values must be case-sensitive and
commas must separate multiple keys.
4. In IntroscopeAgent.profile, verify that the introscope.agent.jmx.name.filter
property is uncommented.
Note: Filters must include an MBean attribute, such as a version number. For
example, given the complete metric name:
*SuperDomain*|MyServer01|WebSphere|LODVMAPM032Node02Cell/server1|JMX|WebSpher
e|cell=LODVMAPM032Node02Cell|mbeanIdentifier=default_host/hello|name=hello-2_
1_1_2_war#hello-2.1.1.2.war|node=LODVMAPM032Node02|platform=dynamicproxy|proc
ess=server1|spec=1.0|type=SessionManager|version=6.1.0.0|StatsImpl|LiveCount:
HighWaterMark
Enable JSR-77 Data and View JMX Metrics on WAS


A filtered version containing the version number attribute would be:
PlantsByWebSphere|name=PlantsByWebSphere#PlantsByWebSphere.war|node=LODVMAPM0
33Node01|platform=dynamicproxy|process=server1|spec=1.0|type=SessionManager|v
ersion=6.1.0.0
Or more simply:
Plant*re|*version*
5. Enter the desired strings and separate them with commas in the property.
For CA Introscope to match filtered strings, spell and make case sensitive the
strings exactly.
6. Save changes.
7. Start the managed application.
8. Enable JMX data by configuring a CA Introscope startup class in WebLogic Server
or a custom service in WebSphere.
More information:
Create a startup class for WebLogic (see page 41)

You can configure Introscope to collect, retain, and report metrics for JSR-77 JMX
MBean objects under WebSphere. JSR-77, the J2EE Management Specification, abstracts
the manageable parts of the J2EE architecture and defines an interface for accessing
management information.
Note: JSR-77 support requires a JVM version 1.4 at a minimum. If the JVM is 1.4, the
application server must also support JSR-77.
Follow these steps:
1. Shut down the managed application.
2. Configure a custom service in WebSphere (see page 51).
3. In the IntroscopeAgent.profile, verify that the following property is true:
introscope.agent.jmx.enable=true
4. In the IntroscopeAgent.profile, enable JSR-77 by setting the following property:
introscope.agent.jmx.name.jsr77.disable=false


5. Configure the primary keys method of metric conversion by uncommenting the
following property in the IntroscopeAgent.profile:
introscope.agent.jmx.name.primaryKeys=J2EEServer,Application,
j2eeType,JDBCProvider,name,mbeanIdentifier
Note: Only the IntroscopeAgent.profile provided with Introscope for WebSphere
contains this property definition.
6. Uncomment and set the following property to identify the metrics you want the
JSR-77 Metrics to report:
introscope.agent.jmx.name.filter
Filtering is not required, but we highly recommend it.
7. Uncomment the following property and update as desired to specify Mbean
attributes that you want to exclude in JSR-77 metrics:
introscope.agent.jmx.ignore.attributes=server
Note: For more information, see KB article TEC534087: WebSphere Performance Viewer
Data vs. Introscope PMI Data at http://ca.com/support.
More information:
Using Primary Key Conversion to Streamline JMX Metrics (see page 189)
Managing Metric Volume with JMX Filters (see page 190)


Chapter 13: Configuring Platform
Monitoring

This section has instructions for configuring Introscope Platform Monitors.
Platform Monitors (see page 195)
Enabling platform monitors on Windows Server 2003 (see page 196)
Enable Platform Monitors on AIX (see page 196)
Disable Platform Monitors (see page 197)
Configure Permissions to Access Platform Monitors on HP-UX (see page 197)
Troubleshoot Platform Monitoring (see page 197)
Platform Monitors
Platform monitors enable the Java agent to report system metrics, including CPU
statistics, to the Enterprise Manager. Platform monitors are included with the
Introscope agent installers.
Consider the following information:
Platform monitors on all operating systems except Windows Server 2003 and AIX
are automatically enabled upon Java agent installation. Windows Server 2003 and
AIX platform monitors require minimal configuration to work.
Platform monitor binaries are independent of application server and operating
system bit modes. Also, platform monitor binaries are purely dependent on JVM
architecture.
The Java agent platform monitor for HP-UX reports the percentage use of CPUs
when more than one process is present. For example, if you have 4 processes, the
maximum use of the CPU would be 400 percent (4 processes using 100 percent of
the CPU). If one process takes 110 percent, it is using 1.1 CPUs.

Note: For system requirements, see the Compatibility Guide.
The Java agent generates the following platform metrics:
ProcessID
Processor CountIndicates the number of CPUs
Utilization % (process)For the Java Agent process, indicates the percentage of
total capacity of all processors this process is using. Regardless of how many
processors there are, this metric generates only one number.
Enabling platform monitors on Windows Server 2003


Utilization % (aggregate)For this processor, indicates its total use (as a
percentage) by all processes in the system. Each processor is shown as a Resource
in the Investigator tree.
To run platform monitors on Windows Server 2003, you must have admin privileges.
Note that system objects must be enabled for platform monitoring to work on
Windows.
To determine whether system objects are enabled
1. Go to Start > Run.
2. Type perfmon and click Run.
3. In the dialog box click Add.
4. In the Add dialog, if "Process" and "Processor" performance objects are present in
the drop down list box, then system objects are enabled.
To enable system objects
1. Go to Start > Accessories > Right click on command prompt > Run as >
Administrator.
2. Run the command: lodctr /r
"Process" and "Processor" objects will be enabled.
Enable Platform Monitors on AIX
You can enable platform monitors on AIX.
Follow these steps:
1. After Java agent installation, verify that the following files are installed in the
wily/core/ext directory:
introscopeAIXPSeries32Stats.jar
introscopeAIXPSeries64Stats.jar
libIntroscopeAIXPSeries32Stats.so
libIntroscopeAIXPSeries64Stats.so
Disable Platform Monitors


2. Install the Perfstat Library. Install the following packages from the IBM FTP site:
bos.perf.libperfstat
bos.perf.perfstat
3. Restart your computer.
The patches take effect and the platform monitors are enabled.
Disable Platform Monitors
You can disable a platform monitor by moving its corresponding .jar file to a different
directory.
Follow these steps:
1. Go to the /wily/core/ext directory.
2. Select the introscope<platform>.jar file that corresponds to your platform, for
example, introscopeSolarisAmd32Stats.jar.
3. Move the .jar file from the /wily/core/ext directory to another directory.
Configure Permissions to Access Platform Monitors on HP-UX
Installing the platform monitor on HP-UX using .zip or .tar installers requires
administrators to change read-write permissions on the following files to 777:
wily/ext/introscopeHpuxItaniumStats.jar
wily/ext/introscopeHpuxItaniumStats.so
wily/ext/introscopeHpuxParisc32Stats.jar
wily/ext/introscopeHpuxParisc32Stats.so
Example (root user):
chmod 777 /<Agent_Home>/wily/ext/introscopeHpuxItaniumStats.jar
Make these changes after installing the agent and before starting it.
If you are using the first two files named above, you also must install gcc before starting
the agent. You can download gcc, a compiler for C and C++, by searching the HP support
website.
Troubleshoot Platform Monitoring
In most cases, the platform monitor detects the operating system and runs if the
operating system is supported. In rare cases where detection does not occur, you can
explicitly specify your operating system in the Java agent profile to help ensure that the
platform monitor runs.
Troubleshoot Platform Monitoring


Follow these steps:
1. Open IntroscopeAgent.profile.
2. Under the Platform Monitor Configuration heading, locate the exact matching value
for your operating system and uncomment the property. For example, for a Sun
Solaris 64-bit operating system with Sun SPARC hardware, you locate and
uncomment the following value:
#introscope.agent.platform.monitor.system=SolarisSparc64
Troubleshooting platform monitoring on Windows
On Windows platforms, the following error can appear for the Java agent:
11/28/06 08:29:55 AM PST [ERROR] [IntroscopeAgent] An error occurred polling for
platform data
If the error is infrequent, it is likely caused by a transient error originating from
Windows itself, and is harmless. On platforms other than Windows, or in the case that
the error happens all the time, this error indicates something more serious and should
be reported to CA Support for Introscope.
SAP Netweaver
Platform monitors may not function correctly on SAP Netweaver due to the lack of
permissions for SAP user accounts.
By default, SAP user accounts are not registered under Performance Monitor Users,
which is located by navigating to Computer Management > System Tools > Local Users
and Groups > Groups > Performance Monitor Users . Users registered under
Performance Monitor Users have access to Perfmon related data
(HKEY_PERFORMANCE_DATA).
If you are experiencing problems with SAP Netweaver and Introscope performance
monitoring, this issue can be resolved by adding SAP user account to the Performance
Monitor Users group, then restarting the Windows machine.


Chapter 14: Integrating CA APM with CA
LISA

How to Integrate CA APM with CA LISA (see page 199)
How to Integrate CA APM with CA LISA
You can integrate CA LISA with CA APM to monitor, detect, triage, and diagnose
application performance problems using synthetic transactions that are generated from
load and regression tests from CA LISA in pre-production environments.
Synthetic transactions represent real transaction performance that can be used to:
Monitor data in a test environment before the application has been released in a
production environment. Monitoring synthetic data ensures that the application
performs within the limits that are defined and certifies expected service level
agreements, resource consumption, and baseline performance characteristics.
Detect and identify the root cause of application performance problems or
bottlenecks early in the software development lifecycle, so that customers can
deliver higher-quality applications to the production environment.
Create reports of application performance and resource usage levels in
pre-production environments to use as guidance for production capacity planning.


Ensure metrics that are gathered provide visibility to the significant facets of the
production applications monitored.
The following illustration describes the tasks that you perform as an administrator to
integrate CA LISA with CA APM:

1. Install CA LISA (see page 203)
2. Configure CA LISA Tracers (see page 204).
3. Verify the CA APM and CA LISA Integration (see page 204).
Install CA LISA
You can install CA LISA using the following methods:
Manually Download and Extract CA LISA Files (see page 201)
Use the Interactive Installer to Install CA LISA (see page 203)


Manually Download and Extract CA LISA Files
Download and extract CA LISA files to integrate with CA APM.
Important! Verify JRE is installed before installing the CA LISA integration.
Follow these steps:
1. Download CALISAIntegrationNoInstaller<VersionNumber>.windows.zip or
CALISAIntegrationNoInstaller<VersionNumber>.unix.tar from the CA APM software
download area on CA Support.
2. Extract the distribution file contents to any directory that does not contain other
agent distributions.
The files are extracted to the directory specified.
Instrument CA APM to View Test Event Metrics and Data from CA LISA
Instrument CA APM to view test event metrics and data from CA LISA for the following
CA LISA processes:
CA LISA Coordinator
CA LISA Test Runner
CA LISA Workstation
You can instrument additional CA LISA processes. However, only minimal metrics that
are related to CPU, memory usage, and agent and JVM identity are reported.
Follow these steps:
1. Create a corresponding .vmoptions file with the same name as the available
executable or shell script files for Windows or Linux processes and save it to the
<LISA_Home>\bin directory. For example, to instrument the CA LISA Workstation,
create a LISAWorkstation.vmoptions file in the <LISA_Home>\bin directory.
The following list shows the available executable or shell script file names and the
processes that can be instrumented:
Coordinator processes--Create a corresponding file with the .vmoptions
extension for CoordinatorServer.exe or CoordinatorServer.sh.
Registry processes--Create a corresponding file with the .vmoptions extension
for the Registry.exe, Registry.sh, TestRegistry.exe, or TestRegistry.sh files.
Simulator processes--Create a corresponding file with the .vmoptions extension
for the Detached_Simulator.exe, Detached_Simulator.sh, Simulator.exe, or
Simulator.sh files.
TestRunner processes--Create a corresponding file with the .vmoptions
extension for the TestRunner.exe or TestRunner.sh files.


Virtual Service Environment processes--Create a corresponding file with the
.vmoptions extension for the VirtualServiceEnvironmentService.exe or
VirtualServiceEnvironmentService.sh files.
Workstation processes--Create a corresponding file with the .vmoptions
extension for the LISAWorkstation.exe or LISAWorkstation.sh files.
2. (Optional) Create a corresponding .vmoptions file with the same name as the
available executable or shell script files for Windows services and save it to the
<LISA_Home>\bin directory.
The following list shows the executable file names and the services that can be
instrumented:
Coordinator services--Create a corresponding file with the .vmoptions
extension for the CoordinatorService.exe.
Simulator services--Create a corresponding file with the .vmoptions extension
for the SimulatorService.exe file.
Registry services--Create a corresponding file with the .vmoptions extension for
the TestRegistryService.exe file.
Virtual Service Environment services--Create a corresponding file with the
.vmoptions for the extensionVirtualServiceEnvironmentService.exe file.
3. Change the agent node name by adding a line that defines the Java
com.wily.introscope.agent.agentName system property. For example, to name the
node CA LISA Workstation Agent, add the following line:
-Dcom.wily.introscope.agent.agentName=CA LISA Workstation Agent
4. Add the following lines to the .vmoptions file:
-javaagent:<Agent_HOME>/Agent.jar
-Dcom.wily.introscope.agentProfile=<Agent_HOME>/core/config/IntroscopeAgent.p
rofile
Where <Agent_Home> is the path to a CA LISA specific agent installation. Typically
the <Agent_Home> path is an absolute path but it can be a path relative to the
current directory in which the CA LISA process runs.
5. (Optional) Enter a separate line for each JVM command line option or system
property you want to add.
6. If the CA LISA process being instrumented are run under Java 1.7, add
-XX:-UseSplitVerifier.
The instrumentation settings are applied.


Using the Installer to Install CA LISA
You can integrate CA LISA with CA APM using the installer.
Follow these steps:
1. Select the installer for your operating system:
CALISAIntegrationInstaller<VersionNumber>.unix.tar
CALISAIntegrationInstaller<VersionNumber>.windows.zip
2. Execute the selected installer on the CA LISA server.
a. During the installation, specify the home directory of the CA LISA installation
and the directory to be used for the install.
b. Select the CA LISA processes to instrument.
c. Specify the EM and port.
The installer will create the necessary vmoptions file(s) in the <LISA_Home>\bin
directory as described in Instrument CA APM to View Test Event Metrics and
Data from CA LISA (see page 201).
Note: If CA LISA applications being instrumented are run under Java 1.7, you
must manually edit the vmoptions file after the install and add
-XX:-UseSplitVerifier.
3. Select the CA APM Integration for CA LISA as a monitoring option during the EM
installation.
The management module and typeviews display CA LISA data.
Note: If you do not select the CA APM Integration for CA LISA monitoring option
when running the installer, you can manually perform this step by copying the files
from the <EM_Home>\examples\CAAPMIntegrationForCALISA folder located in the
EM install directory.


Configure CA LISA Tracers
Configure CA LISA tracers to customize the CA LISA system components you want to
monitor. CA LISA tracers let you limit the number of metrics reported to CA APM by
controlling the lowest level at which metrics are reported to these nodes:
Test Case
Simulator
Test Steps
The minimum level at which metrics are reported is at the Test Case node.
Follow these steps:
1. Go to <LISA_Home>\core\config and open the lisa.pbd configuration file to make
the necessary configurations to meet your environment needs and save the file.
2. Go to <LISA_Home>\core\config and open the IntroscopeAgent.profile
configuration file to make the necessary configurations to meet your environment
needs and save the file.
The configuration settings are applied.
Verify the CA APM and CA LISA Integration
You can verify the CA APM and CA LISA Integration by invoking the CA LISA processes
and verifying the instrumented nodes exist in the Investigator tree.
Note: Metrics invoked using CA LISA tests and that appear under the <CA LISA> | Test
Case node in the Investigator tree are reported to the Enterprise Manager only after a
CA LISA test has been run from the CA LISA Coordinator, CA LISA Test Runner, or CA LISA
Workstation.
Follow these steps:
1. Run a test using the CA LISA Workstation. For more information about using CA
LISA, see the CA LISA documentation set.
2. Start the Enterprise Manager if it is not already running.
3. Start the CA LISA process if it is not already running.
Restarting the CA LISA process starts the agent.
4. Launch the Workstation and open the Investigator.


5. Locate data under the following node:
SuperDomain | <Host_Name> | CA LISA | <CA LISA Workstation> | CA LISA | Test Case
| <Test Case Name>
<Host_Name> is the node that houses the computer where the CA LISA agent
installed.
<CA LISA Workstation> is the node where the agent name specified in
LISAWorkstation.vmoptions file. For more information about configuring this
node, see Instrument CA APM to View Test Event Metrics and Data from CA
LISA (see page 201).
<Test Case Name> is the name of the test case run in the first step of
validation.
6. Expand folders.
Active metrics display under the configured nodes.


Chapter 15: Integrating CA APM Cloud
Monitor with CA APM

CA APM Cloud Monitor enables you to perform the following tasks:
Understand complete user experience from 60+ monitoring stations in 40+
countries.
Monitor real browsers to accurately measure user experience.
Monitor applications delivered by SaaS vendors and MSPs to keep them
accountable to SLAs.
Test application response time from outside the firewall (using synthetic
transactions) to understand global end-user experience and monitor performance
even at times when there is no real user traffic.
Replicate real-user transactions to monitor performance throughout the application
infrastructure to identify, diagnose, and resolve problems quickly.
How to Integrate CA APM Cloud Monitor In Your CA APM Deployment


How to Integrate CA APM Cloud Monitor In Your CA APM
Deployment
As an APM administrator, you can integrate CA APM Cloud Monitor in your CA APM
deployment to add additional monitoring and triage capability.
The following flowchart shows how to integrate CA APM Cloud Monitor in an
on-premise CA APM deployment.

1. Download and install the CA APM Cloud Monitor agent. (see page 208)
2. Validate the agent connection. (see page 209)
Download and Install the CA APM Cloud Monitor Agent
In this task, you download and install the CA APM Cloud Monitor agent, which is
required to integrate CA APM Cloud Monitor and CA APM.
Note: You can install the CA APM Cloud Monitor agent on any computer on your
network. Its function is to receive CA APM Cloud Monitor data over the internet and
forward the data to the Enterprise Manager, so it can be any computer, though you
should choose a computer with server-level CPU/RAM.
How to Integrate CA APM Cloud Monitor In Your CA APM Deployment


To download and install the CA APM Cloud Monitor agent:
1. Download the CA APM Cloud Monitor agent archive file. Save the file to your
computer.
Installers for Windows, Linux and Solaris are located on the CA APM software
download website.
2. Install the agent.
a. Launch the CA APM Cloud Monitor installer file.
b. Accept license terms.
c. Specify the <Agent_Home> directory.
d. Specify the Enterprise Manager host and port the agent will connect to.
e. In the Cloud Monitor Account Settings screen, enter your CA APM Cloud
Monitor userid and password. These are the same credentials you use to log
into the CA APM Cloud Monitor website.
The CA APM Cloud Monitor agent installation is complete.
Note: For instructions on starting the agent and validating connections, see Validate the
CA APM Cloud Monitor Agent Connection (see page 209).
Validate the CA APM Cloud Monitor Agent Connection
To validate that the CA APM Cloud Monitor agent is correctly installed, you start the
agent, and check for data using CA APM WebView or the CA Introscope Workstation.
Validate the connection by verifying incoming data from CA APM Cloud Monitor is
current.
Note: To see CA APM Cloud Monitor data in WebView or the Workstation, you must
have set up folders and monitors in CA APM Cloud Monitor. If you have not performed
this step, see the section on using CA APM Cloud Monitor in the CA APM Workstation
User Guide.
Follow these steps:
1. Navigate to the directory where the CA APM Cloud Monitor agent was installed,
and run one of these files to launch the agent.
On Windows, double-click APMCloudMonitor.bat.
On Linux or Solaris, run APMCloudMonitor.sh.
The agent launches in a console window.
2. Launch CA APM WebView or CA Introscope Workstation.
How to Limit Data


3. Locate data from CA APM Cloud Monitor.
a. If an Investigator window is not open, select File > New Investigator.
b. Browse to the Metric Browser tab.
c. Expand the following nodes:
SuperDomain | <Host_Name> | APMCloudMonitor | APMCloudMonitorAgent
| APM Cloud Monitor
<Host_Name> is commonly the computer where you installed the CA APM
Cloud Monitor agent, but what appears in the Metric Browser tree is the value
of the property apmcm.agent.hostName in the APMCloudMonitor.properties
file.
d. Expand folders to see individual metrics and confirm that the metrics are
current.
Troubleshooting
If you failed to select "Cloud Monitor" as a monitoring option during the Enterprise
Manager installation process, WebView/Workstation will not allow the full display of CA
APM Cloud Monitor data.
To solve this problem:
Copy contents of the <EM_Home>\examples\CAAPMIntegrationForCloudMonitor\
directory to <EM_Home>/config/modules/ and <EM_Home>/ext/xmltv.
This will make available management modules and other files necessary to properly
display Cloud Monitor metric data.
How to Limit Data
To improve performance, you can limit the amount of data sent by the CA APM Cloud
Monitor agent to the Enterprise Manager.
Limit Data by Configuring CA APM Cloud Monitor Properties
You can use properties in the file APMCloudMonitor.properties to filter data the CA
APM Cloud Monitor agent sends to the Enterprise Manager.
To filter data by configuring CA APM Cloud Monitor agent properties, edit the Metric
filters section in
<CloudMonitor_Agent_Home>/CloudMon/conf/APMCloudMonitor.properties.
For information about these settings, see the properties reference for
APMCloudMonitor.properties.
How to Limit Data


Limit Data by Removing Checkpoints
CA APM Cloud Monitor has access to over sixty checkpoint stations on five continents. It
randomly selects from among these stations and checks availability and performance
from a station to your website or application. Over time, all enabled stations will
perform this check, resulting in data being logged from each of over sixty sites.
You can remove some of the available checkpoint stations to limit the amount of data
CA APM Cloud Monitor sends to CA APM.
Follow these steps:
1. Log in to the CA APM Cloud Monitor website, cloudmonitor.ca.com.
2. Select Subscriptions > Preferences.
All stations are selected by default.
3. Change default selection:
Clear the check box for individual stations, or:
Click Clear to clear all, and select from among the groups at the top of the list.
For example, to select only stations in North America:
a. Click Clear.
b. Click North America.
4. At the bottom of the page, click Change.
Limit Data by Adjusting Schedules
By default, monitors check availability and performance on a regular basis every hour of
every day. Over time, this may result in more data being returned to CA APM than you
want.
Follow these steps:
1. Log in to the Cloud Monitor website, cloudmonitor.ca.com.
2. Select Settings > Monitors.
3. Select an individual monitor, and select More Options.
4. Reset the following settings:
Delay between checks
Check period
Check on these days only
Maintenance schedule


Appendix A: Java Agent Properties

Configuring the IntroscopeAgent.profile location
The agent refers to properties in the IntroscopeAgent.profile for its basic connection
and naming properties. When you install an agent, the agent profile is installed in the
<Agent_Home>/wily/core/config directory.
Introscope looks for the agent profile in these locations, in this sequence:
location defined in the system property com.wily.introscope.agentProfile
location defined in com.wily.introscope.agentResource
<working directory>/wily/core/config directory
Note: When adding a path on a Windows computer, escape the backslash (\) with
another backslash as follows: C:\\Introscope\\lib\\Agent.jar.
To change the location of the IntroscopeAgent.profile
1. Define the new location using one of these methods:
define a system property on the Java command line with the -D option to
specify the full path to the location of the IntroscopeAgent.profile file:
-D com.wily.introscope.agentProfile.
Make the IntroscopeAgent.profile available in a resource on the classpath. Set
com.wily.introscope.agentResource to specify the path to the resource
containing the agent profile.
Note: If you change the location of the IntroscopeAgent.profile, the AutoProbe
log location will also have to be changed. For more information, see Managing
ProbeBuilder Logs (see page 140).
2. Move your ProbeBuilder directives (PBD and PBL files) to the same location as the
agent profilethey are referenced relative to the profile location.
Command-line property overrides


If you use Sun ONE, then add the new location of the agent profile to the Sun ONE
server.xml file
To change the location of the IntroscopeAgent.profile for Sun ONE
1. To add Introscope information to the startup scripts for Sun ONE 7.0, log in as
Administrator or Root.
2. Open the server.xml file, in <SunOne_Home>/domains/domain1/server1/config/
3. Add a line to the jvm-options stanza in server.xml:
<jvm-options>
-Dcom.wily.introscope.agentProfile=SunOneHome/wily/core/config/IntroscopeAgen
t.profile
</jvm-options>
Command-line property overrides
In Introscope, you can override specific properties of the Enterprise Manager, agents,
Workstation, and WebView using the command line. With regard to the Java agent, this
is useful when you have a clustered environment with multiple copies of an agent being
shared and you want to tailor some of the agent settings for each application being
monitored.
These steps assume you have installed and configured an agent on the application
server to be monitored, and that the agent successfully connects to the Enterprise
Manager.
To override agent properties using the command line
1. Open the file where you modified the Java command to start the agent.
The location of this file varies depending on the application server you use in your
environment.
2. Add a -D command to override a property. For example, you can add the following
command to make the agent also use the weblogic-full.pbl file:
-Dintroscope.autoprobe.directivesFile=weblogic-full.pbl
Place this command next to other -D commands in the open file.
Note: When you use this command to override hot deployable properties, the
property is no longer hot deployable. Also, if you modify the property at a later time
in the configuration file, you will receive a warning message in the Workstation
stating you modified an overridden property and your change will have no effect.
To avoid this, remove the override command before modifying the property in a
configuration file.
Agent failover


3. Save the file.
4. Restart the agent.
In the example used above, you would now see the additional WebLogic metrics in
the agent node in the Workstation.
Important! System properties become part of the property space of Introscope
properties, allowing things like java.io.tmpdir to be visible to anything using
IndexedProperties.
Agent failover
If the Java agent loses connection with its primary Enterprise Manager, the Agent
failover properties specify which Enterprise Manager the agent will failover to, and how
often it will try to reconnect to its primary Enterprise Manager.
introscope.agent.enterprisemanager.connectionorder
Specifies the connection order of backup Enterprise Managers the agent uses if it is
disconnected from its default Enterprise Manager.
Property settings
Names of other Enterprise Managers the agent can connect to.
Default
The Enterprise Manager defined by the DEFAULT host name, port number, and socket
factory properties.
Example
introscope.agent.enterprisemanager.connectionorder=DEFAULT
Notes
Use a comma separated list.
effect.
Agent failover


introscope.agent.enterprisemanager.failbackRetryIntervalInSeconds
Specifies the number of seconds between attempts by a denied agent to reconnect to
these Enterprise Managers:
Enterprise Managers based on order configured in the agent profile
introscope.agent.enterprisemanager.connectionorder property value.
Any Enterprise Manager allowed based on loadbalancing.xml configuration.
If an agent cannot connect to an Enterprise Manager, it handles connections in these
ways:
Tries to connect to the next allowed Enterprise Manager.
Does not connect to any Enterprise Manager on which it is disallowed.
Note: For information about configuring loadbalancing.xml and agent to Enterprise
Manager connections, see the CA APM Configuration and Administration Guide.
Default
The default interval is 120 seconds.
Example
introscope.agent.enterprisemanager.failbackRetryIntervalInSeconds=120
Notes
Restart the managed application so that changes to this property take effect.
This property is commented out by default.
This property is useful for environments where an agent is allowed to connect across
the following CA APM components:
Clusters.
Collectors and Standalone Enterprise Managers.
Any combination of clusters, Collectors, and Standalone Enterprise Managers.
If an agent can connect to Enterprise Managers in different clusters and this property is
not set, then the following example shows what occurs:
1. An agent connected to an Enterprise Manager in Cluster disconnects.
2. The agent connects to an Enterprise Manager in Cluster 2 in disallowed mode.
Agent HTTP tunneling


3. The agent does not know when the allowed Enterprise Manager in Cluster 1
becomes available.
This property enforces the agent to keep trying to connect to its allowed Enterprise
Managers until an Enterprise Manager is available for connection.
You can configure agents to send information using tunneling technology, enabling
agents to connect to an Enterprise Manager remotely. To do this, the agent must be
configured to connect to the Enterprise Managers embedded Web server, where the
HTTP tunneling Web service is hosted.
To configure HTTP tunneled communication in IntroscopeAgent.profile as a new agent
connection, specify:
The host name of machine running the Enterprise Manager. For more information,
see introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT (see
page 219).
The connection port to the Enterprise Manager Web server. This is the value for the
introscope.enterprisemanager.webserver.port property specified in the
IntroscopeEnterpriseManager.properties for the Enterprise Manager to which the
agent will connect.
The HTTP tunneling socket factory. Specify this client socket factory:
com.wily.isengard.postofficehub.link.net.HttpTunnelingSocketFactory
Agent HTTP tunneling for proxy servers
The following properties only apply to agents configured to tunnel over HTTP and must
connect to an Enterprise Manager using a proxy server:
introscope.agent.enterprisemanager.transport.http.proxy.host (see page 218)
introscope.agent.enterprisemanager.transport.http.proxy.port (see page 218)
introscope.agent.enterprisemanager.transport.http.proxy.username (see page 218)
introscope.agent.enterprisemanager.transport.http.proxy.password (see page 219)
For more information, see Configuring a proxy server for HTTP tunneling (see page 61).


introscope.agent.enterprisemanager.transport.http.proxy.host
Specifies the proxy server host name.
Default
Commented out; not specified.
Notes
You must restart the managed application before changes to this property take effect.
introscope.agent.enterprisemanager.transport.http.proxy.port
Specifies the proxy server port number.
Default
Notes
introscope.agent.enterprisemanager.transport.http.proxy.username
If the proxy server requires the agent to authenticate it, specifies the user name for
authentication.
Default
Notes


introscope.agent.enterprisemanager.transport.http.proxy.password
If the proxy server requires the agent to authenticate it, specifies the password for
authentication.
Default
Notes
Agent HTTPS tunneling
You can configure agents to send information using HTTPS, enabling agents to connect
to an Enterprise Manager remotely:
introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT (see page 219)
introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT (see page 220)
introscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT (see
page 220)
introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT
Specifies the host name of the computer running the Enterprise Manager that the agent
connects to by default.
Default
localhost
Example
introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=localhost
Notes
Agent memory overhead


introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT
Specifies the port number on the computer that hosts the Enterprise Manager. If you
are using HTTPS tunneling, the default port that listens for connections from the agent is
8444. This property is commented out by default.
Default
8444
Example
Notes
introscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT
Specifies the client socket factory to use for connections from the agent to the
Enterprise Manager when using HTTPS.
Default
com.wily.isengard.postofficehub.link.net.HttpsTunnelingSocketFactory
Example
introscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT=com.wily.i
sengard.postofficehub.link.net.HttpsTunnelingSocketFactory
Notes
Agent memory overhead
Significant agent memory overhead only occurs in certain extreme cases. The typical
trade-off for lowering memory consumption is a potential increase in response time.
However, each application is unique and the trade-off between memory usage and
response time can vary depending on the application itself.
Agent metric aging


introscope.agent.reduceAgentMemoryOverhead
This property specifies the agent configuration to use. Uncomment if you want to
attempt to reduce the agent memory overhead. This property is set to true and
commented out by default.
Property settings
True or False
Default
True
Example
introscope.agent.reduceAgentMemoryOverhead=true
Notes
Restart the managed application for the changes to this property take effect.
Agent metric aging
Agent metric aging periodically removes dead metrics from the agent memory cache. A
dead metric is a metric that has no new data reported in a configured amount of time.
Removing old metrics helps to improve agent performance and avoid potential metric
explosions.
Note: A metric explosion happens when an agent is inadvertently set up to report more
metrics than the system can handle. When too many metrics are reported, the agent
can affect the performance of the application server, or in extreme cases, prevent the
server from functioning at all.
Metrics that are in a group are removed only if all metrics in the group are considered
candidates for removal. Currently, only BlamePointTracer and
MetricRecordingAdministrator metrics are removed as a group. Other metrics are
removed individually.
The MetricRecordingAdministrator has the following interfaces for creating, retrieving,
or removing a metric group:
getAgent().IAgent_getMetricRecordingAdministrator.addMetricGroup
String component, collection metrics. The component name is the metric resource
name of the metric group. The metrics must be under the same metric node to
qualify as a group. The metrics are a collection of
com.wily.introscope.spec.metric.AgentMetric data structures. You can only add
AgentMetric data structures to this collection.
Agent metric aging


getAgent().IAgent_getMetricRecordingAdministrator.getMetricGroup
String component. Based on the component name which is the metric resource
name, you can get the Collection of metrics.
getAgent().IAgent_getMetricRecordingAdministrator.removeMetricGroup
String component. The metric group is removed based on the component which is
the metric resource name.
getAgent().IAgent_getDataAccumulatorFactory.isRemoved
Checks if the metric is removed. You use this interface if you keep an instance of an
accumulator in your extension. If the accumulator is removed because of metric
aging, you use this interface to prevent holding onto a dead reference.
Important! If you create an extension that uses a MetricRecordingAdministrator
interface (for example, for use with other CA Technologies products), be sure to delete
your own instance of an accumulator. If a metric is aged out because it has not been
invoked, and data later become available for that metric, the old accumulator instance
will not create new metric data points. To avoid this situation, do not delete your own
instance of an accumulator and use instead the getDataAccumulatorFactory interface.
Configuring agent metric aging
Agent metric aging is on by default. You can choose to turn off this capability using the
property introscope.agent.metricAging.turnOn (see page 223). If you remove this
property from the IntroscopeAgent.profile, agent metric aging is turned off by default.
Agent metric aging runs on a heartbeat in the agent. The heartbeat is configured using
the property introscope.agent.metricAging.heartbeatInterval (see page 223). Be sure to
keep the frequency of the heartbeat low. A higher heartbeat will impact the
performance of the agent and CA Introscope.
During each heartbeat, a certain set of metrics are checked. This is configurable using
the property introscope.agent.metricAging.dataChunk (see page 224). It is also
important to keep this value low, as a higher value will impact performance. The default
value is 500 metrics to be checked per heartbeat. Each of the 500 metrics is checked to
see if it is a candidate for removal. For example, if you set this property to check chunks
of 500 metrics per heartbeat, and you have a total of 10,000 metrics in the agent
memory, then it will take longer with lower impact on performance to check all 10,000
metrics. However, if you set this property to a higher number, you would check all
10,000 metrics faster, but with possibly high overhead.
Agent metric aging


A metric is a candidate for removal if the metric has not received new data after certain
period of time. You can configure this period of time using the property
introscope.agent.metricAging.numberTimeslices (see page 224). This property is set to
3000 by default. If a metric meets the condition for removal, then a check is performed
to see if all the metrics in its group are candidates for metric removal. If this
requirement has also been met then the metric is removed.
introscope.agent.metricAging.turnOn
Turns on or off agent metric aging.
Property settings
True or False
Default
True
Example
introscope.agent.metricAging.turnOn=true
Notes
Changes to this property take effect immediately and do not require the managed
application to be restarted.
introscope.agent.metricAging.heartbeatInterval
Specifies the time interval when metrics are checked for removal, in seconds.
Default
1800
Example
introscope.agent.metricAging.heartbeatInterval=1800
Notes
Agent metric aging


introscope.agent.metricAging.dataChunk
Specifies the number of metrics that are checked during each interval.
Default
500
Example
introscope.agent.metricAging.dataChunk=500
Notes
introscope.agent.metricAging.numberTimeslices
This property specifies the number of intervals to check without any new data before
making it a candidate for removal.
Default
3000
Example
introscope.agent.metricAging.numberTimeslices=3000
Notes
Changes to this property take effect immediately and do not require restarting the
managed application.
Agent metric clamp


introscope.agent.metricAging.metricExclude.ignore.0
Excludes specified metrics from being removed. To exclude one or more metrics from
aging, add the metric name or a metric filter to the list.
Property settings
Comma separated list of metrics. You can use an asterisk (*) as a wildcard in metric
names.
Default
The default is metric names beginning with Threads (Threads*).
Example
introscope.agent.metricAging.metricExclude.ignore.0=Threads*
Notes
Agent metric clamp
You can configure the agent to approximately clamp the number of metrics sent to the
Enterprise Manager. If the number of metrics generated exceeds the value of the
property, the agent stops collecting and sending new metrics.
introscope.agent.metricClamp
Configures the agent to approximately clamp the number of metrics sent to the
Enterprise Manager.
Default
5000
Example
introscope.agent.metricClamp=5000
Notes
If the property is not set, then no metric clamping occurs. Old metrics will still
report values.
Agent naming


This clamp property works with the
introscope.enterprisemanager.agent.metrics.limit property located in the
apm-events-thresholds-config.xml file.
Note: For information about the introscope.enterprisemanager.agent.metrics.limit
property, see the CA APM Configuration and Administration Guide.
If introscope.enterprisemanager.agent.metrics.limit clamp value is triggered before
the introscope.agent.metricClamp value, then the Enterprise Manager reads agent
metrics but does not report them in the Investigator metric browser tree.
If the introscope.agent.metricClamp clamp value is triggered before the
introscope.enterprisemanager.agent.metrics.limit clamp value, the agent stops
sending metrics to the Enterprise Manager
Agent naming
You can configure properties to obtain the Java agent name for application servers and
much more.
More information:
Understanding the Java Agent name (see page 121)

Agent naming


introscope.agent.agentAutoNamingEnabled
Specifies whether agent autonaming is used to obtain the Java agent name for
supported application servers.
Property settings
True or False
Default
Varies by application server, see Notes below.
Example
introscope.agent.agentAutoNamingEnabled=false
Notes
Agent autonaming is enabled in the following application servers: WebLogic,
WebSphere, and JBoss
This property requires the Startup Class to be specified for WebLogic, and requires
the Custom Service to be specified for WebSphere.
Agent auto naming is disabled in the following application servers: Oracle
application server, Interstage, Sun ONE, and Tomcat.
effect.
Important! For WebLogic, WebSphere, and JBoss, the property
introscope.agent.agentAutoNamingEnabled is set to TRUE by default.
introscope.agent.agentAutoNamingMaximumConnectionDelayInSeconds
Specifies the amount of time in seconds the agent waits for naming information before
connecting to the Enterprise Manager.
Default
120
Example
introscope.agent.agentAutoNamingMaximumConnectionDelayInSeconds=120
Notes
Agent naming


introscope.agent.agentAutoRenamingIntervalInMinutes
Specifies the time interval in minutes during which the agent will check to see if it has
been renamed.
Default
10
Example
introscope.agent.agentAutoRenamingIntervalInMinutes=10
Notes
introscope.agent.agentName
Uncomment this property to provide a default agent name if other agent naming
methods fail.
Property settings
For any installation, if the value of this property is invalid or if this property is deleted
from the profile, the agent name will be UnnamedAgent.
Example
#introscope.agent.agentName=AgentName
Notes
effect.
In the agent profile provided with application server-specific agent installers, the
default reflects the application server, for instance WebLogic Agent.
In the agent profile provided with the default agent installer, the property value is
AgentName, and the line is commented out.
Agent naming


Use this property if you want to specify the agent name using the value of a java system
property.
Default
Not specified.
Example
Notes
introscope.agent.disableLogFileAutoNaming
Specifies whether to disable automatic naming of agent log files when using
AutoNaming options.
Setting this property to true disables the auto-naming of log files for the agent,
AutoProbe and LeakHunter with the agent name or a timestamp.
Property settings
True or False
Default
False
Example
introscope.agent.disableLogFileAutoNaming=false
Notes
effect.
Log file auto-naming only takes effect when the agent name can be determined
using a Java system property or an application server custom service.
Agent naming


introscope.agent.clonedAgent
Enables you to run identical copies of an application on the same machine. Set this
property to true if you have identical copies of an application running on the same
machine.
Property settings
True or False
Default
False
Example
introscope.agent.clonedAgent=false
Notes
introscope.agent.customProcessName
Specify the process name as it should appear in the Introscope Enterprise Manager and
Workstation.
Default
Varies by application server.
Example
introscope.agent.customProcessName=CustomProcessName
Notes
effect.
In the agent profile provided with application server-specific agent installers, the
default reflects the application server, for instance "WebLogic."
In the agent profile provided with default agent installer, the property is
commented out.
Agent recording (business recording)


introscope.agent.defaultProcessName
If no custom process name is provided and the agent is unable to determine the name
of the main application class, this value is used for the process name.
Default
UnknownProcess
Example
introscope.agent.defaultProcessName=UnknownProcess
Notes
introscope.agent.display.hostName.as.fqdn
This property specifies whether the agent name is displayed as a fully qualified domain
name (fqdn). To enable the fully-qualified domain name, set this property value to
'true.' By default, the agent displays the host name.
Note: For the Catalyst integration, set this property to 'true.'
Property settings
True or False
Default
False
Example
introscope.agent.display.hostName.as.fqdn=false
Notes
Agent recording (business recording)
You can control how the agent handles business transaction recording.
Note: For more information on agent business recording, see the CA APM Transaction
Definition Guide.
Agent thread priority


introscope.agent.bizRecording.enabled
Enables or disables business transaction recording for the agent.
Property settings
True or False
Default
True
Example
introscope.agent.bizRecording.enabled=true
Notes
To further configure business transaction recording for the agent, see the additional
properties for the application triage map.
More information:
Application triage map (see page 235)
Application triage map business transaction POST parameters (see page 240)
Application triage map managed socket configuration (see page 242)

Agent thread priority
The following property controls the priority of agent threads:
introscope.agent.thread.all.priority (see page 233)
Agent to Enterprise Manager connection


introscope.agent.thread.all.priority
Controls the priority of agent threads.
Property settings
You can set this from 1 (low) to 10 (high).
Default
Commented out;5.
Example
introscope.agent.thread.all.priority=5
Notes
You can control how the agent connects to the Enterprise Manager.
Default
localhost
Example
Notes


Specifies the port number on the computer that hosts the Enterprise Manager that
listens for connections from the agent.
Default
The default port depends on the type of communication channel you want to configure.
For direct communication between the agent and Enterprise Manager, the default port
is 5001.
Example
Notes
effect.
If you want to connect to the Enterprise Manager using HTTPS (HTTP over SSL), the
default port is 8444. If you want to connect to the Enterprise Manager using SSL,
the default port is 5443. However, these default settings are commented out by
default.
Specifies the default client socket factory to use for connections from the agent to the
Enterprise Manager.
Default
The default socket factory depends on the type of communication channel you want to
configure. For direct communication between the agent and Enterprise Manager, the
default socket factory is as follows:
com.wily.isengard.postofficehub.link.net.DefaultSocketFactory
Example
introscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT=com.wily.i
sengard.postofficehub.link.net.DefaultSocketFactory
Notes
Application triage map


introscope.agent.enterprisemanager.transport.tcp.local.ipaddress.DEFAULT
Specifies the IP address of the computer running Enterprise Manager that the agent
Default
This property is not defined by default.
Example
introscope.agent.enterprisemanager.transport.tcp.local.ipaddress.DEFAULT=<address
>
Notes
introscope.agent.enterprisemanager.transport.tcp.local.port.DEFAULT
Specifies the local port of the computer running Enterprise Manager that the agent
Default
This property is not defined by default.
Example
introscope.agent.enterprisemanager.transport.tcp.local.port.DEFAULT=CA Portal
Notes
You can configure application triage map data.
Note: For information on how to use the application triage map, see the CA APM


introscope.agent.appmap.enabled
Enables or disables the tracking of monitored code for the application triage map.
Property settings
True or False
Default
True
Example
introscope.agent.appmap.enabled=true
Notes
Enabled by default.
introscope.agent.appmap.metrics.enabled
Enables or disables the tracking of metrics for application triage map nodes.
Property settings
True or False
Default
False
Example
introscope.agent.appmap.metrics.enabled=false
Notes


introscope.agent.appmap.queue.size
Sets the buffer size for the application triage map.
Property settings
Positive integers.
Default
1000
Example
introscope.agent.appmap.queue.size=1000
Notes
The value must be a positive integer.
If the value is set to 0, the buffer is unbounded.
introscope.agent.appmap.queue.period
Sets the frequency in milliseconds for sending application triage map data to the
Enterprise Manager.
Property settings
Positive integers.
Default
1000
Example
introscope.agent.appmap.queue.period=1000
Notes
Must be a positive integer.
If the value is set to 0, the default value is used.
Application Triage Map and Catalyst Integration


introscope.agent.appmap.intermediateNodes.enabled
Enables or disables the ability to include intermediate nodes between the application
frontend and backend nodes.
Property settings
True or False
Default
False
Example
#introscope.agent.appmap.intermediateNodes.enabled=true
Notes
Changes to this property takes effect immediately and do not require the managed
If you set this property to true, you may experience a slowdown of agent
performance.
You can configure application triage map data for the Catalyst integration.
Note: For information on how to use the application triage map, see the CA APM
Configure the Ability to Send Information
This property enables or disables the ability to send additional information from the
agent for integration with Catalyst.
Follow these steps:
1. Open the default IntroscopeAgent.profile file in a text editor.


Locate the line:
introscope.agent.appmap.catalystIntegration.enabled=<false|true>, and set a value
as follows:
true
Enables the ability to send additional information from the agent for
integration with Catalyst.
false
Disables the configuration.
The following example shows the format:
introscope.agent.appmap.catalystIntegration.enabled=false
Note: This property is commented out by default.
The agent is set up to use the configuration.
Configure a List of Available Networks
The introscope.agent.primary.net.interface.name property specifies the primary
network interface name of the host computer used by the agent for the Catalyst
integration. You can change the configuration of this property and the change is applied
automatically.
Note: When the agent logging level is set to DEBUG, information about network
interface names available for configuration appears in the log file. Alternatively, you can
use the Network Interface utility to determine the primary network interface name for
this property.
Follow these steps:
1. Open the default IntroscopeAgent.profile file in a text editor.
2. Locate the line: introscope.agent.primary.net.interface.name=<false|true>, and
specify the name value.
The following example shows the name format:
introscope.agent.primary.net.interface.name=eth4
Note: The default value is undefined. When this property is not set, the agent
assigns the first available network interface as the primary interface. You can use
the Network Interface utility to determine the name value for this property.
Application triage map business transaction POST parameters


3. (Optional) Allow for multiple network addresses by specifying the subinterface
number (starts at 0).
The following example shows the subinterface number format:
introscope.agent.primary.net.interface.name=eth4.1
The profile is set up to use the configuration.
More information:
Using the Network Interface Utility (see page 347)

You can configure your Local Product Shorts to perform more complex monitoring by
matching POST parameters.
introscope.agent.bizdef.matchPost
This property determines when POST parameters are matched.
Property settings
The valid settings for this property are never, before, or after.
Set the property to never for full agent functionality and better performance. This
setting allows your application to identify all business transactions using URLs,
cookies or header parameters, but will fail to match any business transactions that
are identified solely through POST parameters.
Set the property to before to get full agent performance. This setting allows
applications to use POST parameters to identify some or all business transactions,
but never access the servlet stream directly for HTTP form requests. New
applications deployed must also conform to standard API when this property is set
to before.
Important! Setting this property to before could have potentially hazardous
repercussions for your applications. Review this property setting with a CA
Technologies representative before implementation.


Set the property to after to safely match business transactions with POST
parameters, but with limited agent functionality. When this property is set to after,
the agent will not be able to map business transactions that are identified by POST
parameters across processes, or produce full sets of metrics for them. This setting
also consumes slightly more CPU time compared to the other options, but is
considered the safest setting if one needs the POST parameter functionality. It
allows applications to use POST parameters to identify some or all business
transactions and cannot guarantee that the servlet stream is never accessed
directly.
Example
introscope.agent.bizdef.matchPost=after
Notes
never - never attempt to match POST parameters. This is the fastest option but may
result in inaccurate business transaction component matching.
before - matches POST parameters before the servlets execute.
after - matches POST parameter patterns after the servlet has executed. Cross
process mapping and some metrics will not be available. Default setting for the
parameter.
Known limitations
The metrics defined using agent recording are displayed in the application triage map in
the Investigator. When configuring agent recording, there are some known limitations
when using regular expressions. Most of the limitations have to do with POST
parameters.
The known limitations are:
Line terminators (.) are not supported for POST parameter values.
If POST parameter definitions are dependent on the business transaction definition,
only three metrics will be provided for the business transaction component. These
metrics are:
Average Response Time
Responses Per Interval
Errors Per Interval
Application triage map managed socket configuration


If POST parameter definitions are dependent on the business transaction definition,
then the business component name in the transaction trace component will have a
generic name and not the specific name of the business service, business
transaction, and business transaction component. This also applies to business
transaction definitions that are dependent on POST parameter definitions that do
not match.
Some versions of JBoss and Tomcat might save header keys as lower case values
which makes the caseSensitiveName attribute not work properly for
HEADER_TYPEs.
Note: For more information on agent recording, see the CA APM Transaction Definition
Guide.
The following properties allow you to enable or disable the appearance of socket
metrics in the application triage map:
introscope.agent.sockets.managed.reportToAppmap (see page 243)
introscope.agent.sockets.managed.reportClassAppEdge (see page 243)
introscope.agent.sockets.managed.reportMethodAppEdge (see page 244)
introscope.agent.sockets.managed.reportClassBTEdge (see page 244)
introscope.agent.sockets.managed.reportMethodBTEdge (see page 245)
See the CA APM Workstation User Guide for more information on how to use the
application triage map.


introscope.agent.sockets.managed.reportToAppmap
Enables managed sockets to appear in application triage map.
Property settings
True or False
Default
True
Example
introscope.agent.sockets.managed.reportToAppmap=true
Notes
introscope.agent.sockets.managed.reportClassAppEdge
Enables managed sockets to report class level application edges to the application triage
map.
Property settings
True or False
Default
False
Example
introscope.agent.sockets.managed.reportClassAppEdge=false
Notes


introscope.agent.sockets.managed.reportMethodAppEdge
Enables managed sockets to report method level application edges to the application
triage map.
Property settings
True or False
Default
True
Example
introscope.agent.sockets.managed.reportMethodAppEdge=true
Notes
introscope.agent.sockets.managed.reportClassBTEdge
Enables managed sockets to report class level business transaction edges to the
Property settings
True or False
Default
False
Example
introscope.agent.sockets.managed.reportClassBTEdge=false
Notes
AutoProbe


introscope.agent.sockets.managed.reportMethodBTEdge
Enables managed sockets to report method level business transaction edges to the
Property settings
True or False
Default
True
Example
introscope.agent.sockets.managed.reportMethodBTEdge=true
Notes
AutoProbe
The following properties configure AutoProbe:
introscope.autoprobe.directivesFile (see page 245)
introscope.autoprobe.enable (see page 246)
introscope.autoprobe.directivesFile
Specifies ProbeBuilder directives files for AutoProbe.
Default
Varies by installer.
Notes
effect.
If this property includes one or more directories, and dynamic instrumentation is
enabled, the agent will load directives files from the specified directories without an
application restart.
AutoProbe


introscope.autoprobe.enable
This property setting enables or disables inserting probes into bytecode automatically.
Property settings
true or false
Default
true
Example
introscope.autoprobe.enable=true
Note: Setting this property to false turns off automatic insertion of probes into
application bytecode, but does not turn off the agent or agent reporting. Restart of the
JVM is required for any change to take effect.
introscope.autoprobe.logfile
Introscope AutoProbe always attempts to log the changes it makes. Set this property to
move the location of the log file to something other than the default.
Property settings
Absolute file paths, or non-absolute paths. Non-absolute names are resolved relative to
the location of this properties file.
Default
../../logs/AutoProbe.log
Example
introscope.autoprobe.logfile=../../logs/AutoProbe.log
To disable logging, comment out the log file as follows:
introscope.autoprobe.logfile=logs/AutoProbe.log
Notes
effect.
Bootstrap Classes Instrumentation Manager


Bootstrap Classes Instrumentation Manager
The following properties configure the Bootstrap Classes Instrumentation Manager:
introscope.bootstrapClassesManager.enabled (see page 247)
introscope.bootstrapClassesManager.waitAtStartup (see page 247)
The Bootstrap Classes Instrumentation Manager instruments a set of classes after the
agent bootstrap, allowing easy implementation of tracers for Java NIO and Secure
Sockets Layer (SSL), as well as improving agent performance. You can disable this
property by commenting it out in the IntroscopeAgent.profile.
introscope.bootstrapClassesManager.enabled
Enables or disables the bootstrap manager.
Property settings
True or False
Default
True
Example
introscope.bootstrapClassesManager.enabled=true
Notes
This property only functions with JVMs running Java 1.5 at a minimum.
If set to false, no system classes are instrumented.
If the property is not set, the default value is false.
effect.
introscope.bootstrapClassesManager.waitAtStartup
Sets the time, in seconds, for how long the agent waits after startup to instrument
bootstrap classes.
Property settings
Time, in seconds
CA CEM Agent Profile Properties


Default
240 seconds when used with HP-UX, Interstage, WebLogic, or WebSphere
Application Server.
5 seconds when used with JBoss, Oracle, Sun, or Tomcat.
Example
introscope.bootstrapClassesManager.waitAtStartup=5
Notes
This property only functions with JVMs running Java 1.5 at a minimum.
When this property is active, it can overrule classes that have been designated as
skipped. If skipped classes are being instrumented, contact your CA Technologies
representative, or CA Support.
You can configure CA CEM-related IntroscopeAgent.profile properties. The CA
Introscope agent profile file is found in the <Agent_Home>\wily directory.
All the CA CEM-related properties are preconfigured to the required options for the
integration with CA CEM and CA Introscope to work.
introscope.autoprobe.directivesFile
You need to enable ServletHeaderDecorator / HTTPHeaderDecorator and CEMTracer
through the directivesFile property configuration.
The directives file property specifies where to find the directive files (PBD) or directive
lists (PBL) for AutoProbe.
AutoProbe uses directives to Introscope-enable your applications, and to determine
which metrics the agents report to the Enterprise Manager.
Settings
Depends on the agent application server installed, with the format of
<appserver>-full.pbl or <appserver>-typical.pbl.
Default
default-typical.pbl
Example
introscope.autoprobe.directivesFile=weblogic-typical.pbl


Notes
Although you can simply add "ServletHeaderDecorator.pbd" or
"httpheaderdecorator.pbd" to the end of this property list, it is better practice to:
1. Locate the PBL file specified in the property (in the example above,
weblogic-typical.pbl).
2. Open the PBL file in a text editor.
3. For a Java agent, uncomment to enable the ServletHeaderDecorator.pbd line.
4. For a .NET agent, uncomment to enable the httpheaderdecorator.pbd line.
5. Save your changes to the PBL file.
introscope.agent.remoteagentconfiguration.allowedFiles
This property identifies the files that are allowed to be copied remotely from any
machine to the agent directory.
The Enterprise Manager uses the file name in this property to identify the valid CA CEM
domain configuration file to send to the agents. The domain configuration file contains
the CA CEM business service and transactions definitions.
Settings
Use valid file name.
Default
domainconfig.xml
Example
introscope.agent.remoteagentconfiguration.allowedFiles=domainconfig.xml
Notes
This property also applies to the Introscope Command-Line Workstation (CLW) Send
Config File command.
For more information, see Using the Command-Line Workstation.
This property is valid for CA CEM releases.


introscope.agent.remoteagentconfiguration.enabled
If this Boolean value is set to true, it allows remote file copies to the agent from another
computer.
The Enterprise Manager requires you to set this property to true to send the CA CEM
domain configuration file to the agents. The domain configuration file contains the CA
CEM business service and transactions definitions.
Settings
true or false
Default
true for Java Agents
false for .NET Agents
Example
introscope.agent.remoteagentconfiguration.enabled=true
Notes
A remote user can also use the CA Introscope Command-Line Workstation (CLW) Send
Config File command to copy the file or files specified in the
introscope.agent.remoteagentconfiguration.allowedFiles property into the agent
directory.
This property is valid for CA CEM 4.0 and 4.1 releases. This property is also valid with CA
CEM 4.2 / 4.5 only when you have selected the CEMTracer 4.0 / 4.1 Support option on
the Introscope Settings page.
The CEMTracer 4.0 / 4.1 Support option lets you stagger your agent migration from 4.0
or 4.1 to 4.2 / 4.5 over time. Use this option only if it is required.
For noncompatible agents (that is, an unsupported .NET agent, an EPA agent, or other
non-Java agent), set the introscope.agent.remoteagentconfiguration.enabled property
to false.
introscope.agent.decorator.enabled
If this Boolean value is set to true, it configures the agent to add additional performance
monitoring information to HTTP response headers. ServletHeaderDecorator /
HTTPHeaderDecorator attaches the GUID to each transaction and inserts the GUID into
an HTTP header, x-apm-info.


This enables the correlation of transactions between CA CEM and CA Introscope.
Settings
True or false
Default
false for Java agents
true for .NET agents
Example
introscope.agent.decorator.enabled=false
introscope.agent.decorator.security
This property determines the format of decorated HTTP response headers that are sent
to CA CEM.
Settings
clearclear text encoding
encryptedheader data is encrypted
Default
clear
Example
introscope.agent.decorator.security=clear
Notes
The default setting of clear is appropriate for initial testing. However, this setting can
reveal information in the transaction header that you do not want known beyond the
firewall. Set the property to encrypted for a more secure production environment.
To set this property to encrypted, use a supported JVM.
Note: For JVM support information, see the Compatibility Guide.
introscope.agent.cemtracer.domainconfigfile
The name of the CA CEM domain configuration file that specifies the CA CEM business
service and transaction hierarchy. CEMTracer looks for a file with this name in its
installation directory.


Each time a CA CEM administrator clicks Synchronize All Monitors in CA CEM, the
domain configuration file gets pushed to the Enterprise Manager, which in turn pushes
the file to each connected agent.
See Notes (below) for CA CEM release-specific information.
Settings
Can be any valid file name.
Default
domainconfig.xml
Example
introscope.agent.cemtracer.domainconfigfile=domainconfig.xml
Notes
This property is valid for CA CEM 4.0 and 4.1 releases. It is also valid with CA CEM 4.2 /
4.5 only when you have selected the CEMTracer 4.0 / 4.1 Support option on the
Introscope Settings page.
The CEMTracer 4.0 / 4.1 Support option allows you to stagger your agent migration from
4.0 or 4.1 to 4.2 / 4.5 over time; use only if required.
If an agent is not connected to the Enterprise Manager, the domain configuration
file does not get sent.
If the agent directory is read-only, the domain configuration file cannot be written.
If CEMTracer 4.0 / 4.1 is not enabled on the agent, then no action will be taken on
the domain configuration file once it is sent.
introscope.agent.cemtracer.domainconfigfile.reloadfrequencyinminutes
The frequency, in minutes, that the agent reloads the domain configuration file. (The 4.0
/ 4.1 agent does not automatically reload the domain configuration file every time the
Enterprise Manager sends it. If it has not changed, the agent will not reload it.)
See Notes (below) for CA CEM release-specific information.
Settings
numeric value
Default
1


Example
introscope.agent.cemtracer.domainconfigfile.reloadfrequencyinminutes=1
Notes
This property is valid for CA CEM 4.0 and 4.1 releases. It is also valid with CA CEM 4.2 /
4.5 only when you have selected the CEMTracer 4.0 / 4.1 Support option on the
Introscope Settings page.
The CEMTracer 4.0 / 4.1 Support option allows you to stagger your agent migration from
4.0 or 4.1 to 4.2 over time; use only if required.
introscope.agent.distribution.statistics.components.pattern
To collect response time distribution information from BlamePointTracers, uncomment
and edit this property. You can use this response time information to create average
response time metrics.
For more information:
How to Configure an Agent to Collect Distribution Statistic Metrics (see page 72)

Configure the Session ID Collection
The introscope.agent.transactiontracer.parameter.capture.sessionid property enables
or disables the collection of the Session ID in the TransactionTracer Data. By default, this
property is enabled and recorded in the TransactionTracer Data. When you disable this
property, data is not available for use in filters.
Follow these steps:
1. Open the IntroscopeAgent.profile file in a text editor.
Locate the following lines:
# Uncomment the following property to disable sessionid capture in
TransactionTracer data.
# By default, it is enabled and recorded in the TT Data.

# introscope.agent.transactiontracer.parameter.capture.sessionid=true
2. Follow the instructions to enable or disable the property by commenting or
uncommenting the line:
# introscope.agent.transactiontracer.parameter.capture.sessionid=true
ChangeDetector configuration


3. Save and close the file, and restart the agent.
The agent configuration is set to use the value you specified for collecting the
session ID.
You can control how the local agent works with ChangeDetector.
Note: For more information about using ChangeDetector, see the CA APM
ChangeDetector User Guide.
introscope.changeDetector.enable
Specifies whether ChangeDetector is enabled or disabled. Set the property to true to
enable ChangeDetector. It is commented out and set to false by default. If you enable
ChangeDetector, you should also set the additional ChangeDetector-related properties.
Property settings
True or False
Default
False
Example
introscope.changeDetector.enable=false
Notes
introscope.changeDetector.agentID
Specifies the text string used by ChangeDetector to identify the local agent. This
property is commented out by default. If you enable ChangeDetector, you should
uncomment this property and set it to an appropriate value.
Default
The default value is SampleApplicationName.
Example
introscope.changeDetector.agentID=SampleApplicationName


introscope.changeDetector.rootDir
Specifies the root directory for ChangeDetector files. The root directory is the folder
where ChangeDetector creates its local cache files.
Property settings
Full path to the root directory for ChangeDetector files as a text string.
Default
The default path is c://sw//AppServer//wily//change_detector.
Example
introscope.changeDetector.rootDir=c://sw//AppServer//wily//change_detector
Notes
Use a backslash to escape the backslash character, as in the example.
introscope.changeDetector.isengardStartupWaitTimeInSec
Specifies the number of seconds to wait after the agent starts before ChangeDetector
tries to connect to the Enterprise Manager. This property is commented out by default.
Default
The default is 15 seconds.
Example
introscope.changeDetector.isengardStartupWaitTimeInSec=15
introscope.changeDetector.waitTimeBetweenReconnectInSec
Specifies the number of seconds ChangeDetector waits before retrying a connection to
the Enterprise Manager. This property is commented out by default.
Default
Example
introscope.changeDetector.waitTimeBetweenReconnectInSec=10


introscope.changeDetector.profile
Specifies the absolute or relative path to the ChangeDetector datasources configuration
file. This property is commented out by default.
Default
The default is ChangeDetector-config.xml.
Example
introscope.changeDetector.profile=CDConfig\\ChangeDetector-config.xml
Notes
Use a backslash to escape the backslash character, as in the example.
introscope.changeDetector.profileDir
Specifies the absolute or relative path to the directory that contains datasource
configuration files. If this property is set, all of the datasource configuration files in this
directory are used in addition to any file specified by the
introscope.changeDetector.profile property. This property is commented out by default.
Default
The default is changeDetector_profiles.
Example
introscope.changeDetector.profileDir=c:\\CDconfig\\changeDetector_profiles
Notes
Use a backslash to escape the backslash character.
Cross-process tracing in WebLogic Server


introscope.changeDetector.compressEntries.enable
Specifies whether to allow compression on the ChangeDetector data buffer. You can set
this property to true if you experience memory consumption at start-up to improve
performance.
Property settings
True or False
Default
The default value is false if the property is not set in the agent profile or if commented
out.
Example
introscope.changeDetector.compressEntries.enable=true
Notes
introscope.changeDetector.compressEntries.batchSize
This property defines the batch size for the compression job, set in
introscope.changeDetector.compressEntries.enable above.
Default
1000
Example
introscope.changeDetector.compressEntries.batchSize=1000
Notes
Cross-process tracing in WebLogic Server
The following property configures cross-process tracing in WebLogic Server:
introscope.agent.weblogic.crossjvm (see page 258)
Cross-process transaction trace


introscope.agent.weblogic.crossjvm
Property settings
True or False
Default
Commented out; True
Example
introscope.agent.weblogic.crossjvm=true
Cross-process transaction trace
Uncomment the following property to enable automatic collection of downstream
traces due to tail filter.
introscope.agent.transactiontracer.tailfilterPropagate.enable (see page 258)
Enabling this property and running long periods of Transaction Trace session with tail
filters can cause large numbers of unwanted traces to be sent to the Enterprise
Manager.
introscope.agent.transactiontracer.tailfilterPropagate.enable
Controls whether the presence of a tail filter triggers automatic collection of traces from
downstream agents or not. This property does not affect collection of automatic
downstream traces due to passing of head filters.
Property settings
True or False
Default
False; commented out.
Example
introscope.agent.transactiontracer.tailfilterPropagate.enable=false
Notes
Dynamic instrumentation


You can enable classes and methods to be instrumented dynamically without writing
custom PBDs, restarting the application server, or restarting the agent.
Note: For more information about using Transaction Tracing and dynamic
instrumentation from the Introscope Workstation, see the CA APM Workstation User
Guide.

introscope.autoprobe.dynamicinstrument.enabled
Valid for: Agents that run under JDK 1.5 and use AutoProbe
This property enables dynamic ProbeBuilding for agents.
Property settings
True or False
Default
False
Example
introscope.autoprobe.dynamicinstrument.enabled=false
More information:
Dynamic ProbeBuilding (see page 77)



autoprobe.dynamicinstrument.pollIntervalMinutes
Valid for: Agents that run under JDK 1.5 using AutoProbe and dynamic ProbeBuilding
This property determines the frequency with which the agent polls for new and changed
PBDs.
Default
1
Example
autoprobe.dynamicinstrument.pollIntervalMinutes=1
introscope.autoprobe.dynamicinstrument.classFileSizeLimitInMegs
Some classloader implementations have been observed to return huge class files.This is
to prevent memory errors.
Default
1
Example
introscope.autoprobe.dynamicinstrument.classFileSizeLimitInMegs=1
Notes
introscope.autoprobe.dynamic.limitRedefinedClassesPerBatchTo
Re-defining too many classes at a time might be very CPU intensive. In cases where the
changes in PBDs trigger a re-definition of a large number of classes, this batches the
process at a comfortable rate.
Default
10
Example
introscope.autoprobe.dynamic.limitRedefinedClassesPerBatchTo=10
ErrorDetector


introscope.agent.remoteagentdynamicinstrumentation.enabled
Enables or disables remote management of dynamic instrumentation.
Property settings
True or False
Default
True
Example
introscope.agent.remoteagentdynamicinstrumentation.enabled=true
Notes
You must restart managed applications before changes to this property take effect.
Dynamic instrumentation is a CPU-intensive operation. Use configurations that
minimize the classes that are being instrumented.
introscope.autoprobe.dynamicinstrument.pollIntervalMinutes
Defines the polling interval in minutes to poll for PBD changes.
Default
1
Example
introscope.autoprobe.dynamicinstrument.pollIntervalMinutes=1
Notes
ErrorDetector
You can control how the agent interacts with ErrorDetector.
ErrorDetector


introscope.agent.errorsnapshots.enable
Enables the agent to capture transaction details about serious errors. Introscope
ErrorDetector is installed by default with the agent. This property must be set to true for
error snapshots to be available for viewing.
Property settings
True or False
Default
True
Notes
This property is dynamic. You can change the configuration of this property during run
time and the change will be picked up automatically.
introscope.agent.errorsnapshots.throttle
Specifies the maximum number of error snapshots that the agent can send in a
15-second period.
Default
10
Example
introscope.agent.errorsnapshots.throttle=10
Notes
time and the change will be picked up automatically.
introscope.agent.errorsnapshots.ignore.<index>
Specifies one or more error message filters. You can specify as many filters as you need
using the index identifier that is appended to the property name (for example, .0, .1, .2
...). You can use wildcards (*) Error messages matching the criteria that you specify are
ignored. No error snapshots are generated for errors matching the filters that you
define and no error events are sent to the Enterprise Manager for them.
Important! This property cannot be used to filter SOAP error messages.
Extensions


Default
Example definitions are provided, and commented out, as shown below.
Example
introscope.agent.errorsnapshots.ignore.0=*com.company.HarmlessException*
introscope.agent.errorsnapshots.ignore.1=*HTTP Error Code: 404*
Notes
time and the change are picked up automatically.
Extensions
You can configure the location of agent extensions.
introscope.agent.extensions.directory
Specifies the location of all extensions to be loaded by the agent. You can specify an
absolute or relative path to the directory. If you do not specify an absolute path, the
value you specify is resolved relative to the location of the IntroscopeAgent.profiles file.
Default
The default location is ext directory in the <Agent_Home>/ext directory.
Example
introscope.agent.extensions.directory=../ext
Notes
introscope.agent.common.directory
Configures the location of Agent Extension related files.
Default
The default location is common folder in the <Agent_Home>/common directory.
Example
introscope.agent.common.directory=../../common
GC Monitor


GC Monitor
The metrics under the GC Monitor node report information on Garbage Collectors and
Memory Pools. These metrics help you detect memory-related issues that are adversely
affecting performance. You must manually enable the collection of these metrics in the
agent profile.
introscope.agent.gcmonitor.enable
This property enables or disables the metrics for Garbage Collectors and Memory Pools.
Property settings
True or False
Default
The default value is true.
Example
introscope.agent.gcmonitor.enable=true
Notes
time and the change is picked up automatically.
You can only report GC Monitor metrics for agents that monitor Sun or IBM JVMs.
Java NIO
The Java Agent supports the Java New I/O (Java NIO, or NIO) capabilities. Java NIO is a
collection of APIs designed to provide access to the low-level I/O operations of modern
operating systems. Java NIO metrics capture information about how instrumented
applications use Java NIO.
Note: CA Introscope Java NIO metrics and metric collection of Java NIO information is
only available on Java 1.5 JVMs at a minimum.
The CA Introscope Java Agent collects metrics for NIO channels.
You can restrict the generation of certain NIO metrics.
Java NIO tracer groups have been enabled by default. You can turn off these tracer
groups to restrict metric generation further.
Java NIO


More information:
Channels (see page 265)
Restricting Java NIO metrics (see page 266)
Default Tracer Groups and Toggles Files (see page 90)

Channels
Java NIO channels provide bulk data transfers to and from NIO buffers, as well as
external systems. This is a low-level data transfer mechanism that was specifically
designed to address performance and scalability issues within standard Java I/O.
Channels provide mechanisms for moving bytes between buffers and external systems.
Introscope channel metrics characterize data flow rates through channels. Collected NIO
channel metrics correspond to metrics currently created for file and socket I/O using
standard Java I/O techniques. Metrics for the following channel types are collected
separately and displayed in the Workstation Investigator:
Datagram Channels
Socket Channels
NIODatagramTracing metrics
Although UDP is a connection-less protocol, the term "connection" is used in the
description of NIODatagramTracing to describe how the Java Agent collects datagram
metrics. The Java Agent collects metrics separately for each remote endpoint datagrams
that are sent to or received from. Connections are classified as client or server
depending on the direction of the first datagram observed from each to or from
endpoint.
For example, if the first datagram is from the endpoint, the Java Agent classifies all
further datagrams to or from that endpoint under
NIO|Channels|Datagrams|Server|Port {PORT}, where {PORT} is the local port.
If the first datagram is to the endpoint, all further datagrams to or from that endpoint
are classified under NIO|Channels|Datagrams|Client|{HOST}|Port {PORT}, where
{HOST} and {PORT} are the remote endpoint.
The Java Agent also generates backend metrics for client "connections", except in the
case of datagrams read by the 'receive' method. Datagram channels created using
DatagramChannel connect method are considered client connections irrespective of
direction of first datagram observed.
Note: Datagram channels created using a UDP connection (with a connect method) are
considered client connections.
Java NIO


Restricting Java NIO metrics
You can configure properties to control how Java NIO instrumentation works. You can
also restrict generation of datagram and socket metrics. These properties only affect the
detail metrics generated by NIOSocketTracing and NIODatagramTracing tracer groups.
Note: For more information about tracer groups, see Default tracer groups and toggles
files (see page 90).
introscope.agent.nio.datagram.client.hosts
Restricts metric reporting to 'client' UDP "connections" with specified host(s).
Property settings
Comma separated list of hosts.
Default
undefined (no value)
Example
introscope.agent.nio.datagram.client.hosts=hostA,hostB
Notes
If the list is left empty, no host restriction will apply.
Hosts may be specified by name or textual representation of IP address (in either
IPv4 or IPv6 forms).
Invalid host names will be reported in the agent log and ignored.
This property is dynamic. You can change the configuration of this property during
run time and the change will be picked up automatically.
Duplicate host names are discarded. In cases where multiple host names map to a
single IP, only one of the names is retained. The property will, however, match
client connections with any of the set of synonymous names.
Java NIO


introscope.agent.nio.datagram.client.ports
Lists the ports that will report NIO metrics. Only 'client' datagram metrics for specified
ports will be generated.
Property settings
Comma separated list of port numbers. Port is the remote port to/from which
datagrams are sent/received.
Default
Example
introscope.agent.nio.datagram.client.ports=123,456,789
Notes
If the list is empty, no port restriction will apply.
Invalid port numbers will be reported in the agent log and ignored.
Duplicate ports are discarded. In cases where multiple ports map to a single IP, only
one of the ports is retained.
Java NIO


introscope.agent.nio.datagram.server.ports
Lists the ports that will report NIO metrics. Only 'server' datagram metrics for specified
Property settings
Comma separated list of port numbers. Port is the local port through which datagrams
are sent/received.
Default
Example
introscope.agent.nio.datagram.server.ports=123,456,789
Notes
If the list is empty, no port restriction will apply.
introscope.agent.nio.socket.client.hosts
Restricts metric reporting to 'client' TCP "connections" with specified host(s).
Property settings
Comma separated list of hosts.
Default
Example
introscope.agent.nio.socket.client.hosts=hostA, hostB
Notes
If the list is empty, no host restriction will apply.
Hosts may be specified by name or textual representation of IP address (in either
IPv4 or IPv6 forms).
Invalid host names will be reported in the agent log and ignored.
Java NIO


introscope.agent.nio.socket.client.ports
Lists the ports that will report NIO metrics. Only 'client' socket metrics for specified
Property settings
Comma separated list of port numbers. Port is the remote port to/from which
datagrams are sent/received.
Default
Example
introscope.agent.nio.socket.client.ports=123,456,789
Notes
If list is empty, no port restriction will apply.
introscope.agent.nio.socket.server.ports
Lists the ports that will report NIO metrics. Only 'server' socket metrics for specified
Property settings
Comma separated list of port numbers. Port is the local port through which datagrams
are sent/received.
Default
Example
introscope.agent.nio.socket.client.ports=123,456,789
Notes
If list is empty, no port restriction will apply.
JMX


Java NIO metrics appear under the NIO node under the agent's top level node in the
Workstation Investigator. Additional NIO metrics for any 'client' connections will appear
under the Backends node.
Individual NIO metrics may be suppressed by commenting out the
TraceOneMethodIfFlagged or TraceOneMethodWithParametersIfFlagged directives for
the metric(s) to be suppressed. However, no tracers whose name ends with
BackendTracer or MappingTracer should be commented out.
For example, to suppress the Concurrent Readers metric for Datagrams, comment out:
TraceOneMethodWithParametersIfFlagged: NIODatagramTracing read
NIODatagramConcurrentInvocationCounter "Concurrent Readers"
JMX
The following properties configure JMX metrics:
introscope.agent.jmx.enable (see page 270)
introscope.agent.jmx.ignore.attributes (see page 271)
introscope.agent.jmx.name.filter (see page 271)
introscope.agent.jmx.name.jsr77.disable (see page 272)
introscope.agent.jmx.name.primarykeys (see page 273)
introscope.agent.jmx.excludeStringMetrics (see page 274)
introscope.agent.jmx.enable
Enables collection of JMX Metrics.
Property settings
True or False.
Default
Varies by agent version.
Example
introscope.agent.jmx.enable=false
Notes
JMX


introscope.agent.jmx.ignore.attributes
Controls which (if any) JMX MBean attributes are to be ignored.
Property settings
A comma-separated list of keywords.
Default
Commented out; server.
Example
introscope.agent.jmx.ignore.attributes=server
Notes
If an MBean attribute name matches one on the list, the attribute will be ignored.
Leave the list empty to include all MBean attributes.
effect.
introscope.agent.jmx.name.filter
Specifies a comma-separated list of filter strings to determine what JMX data Introscope
collects and displays.
Introscope reports JMX-generated metrics that match a filter string. Filter strings can
contain the asterisk (*) and question mark (?) wildcard characters:
* matches zero or more characters
? matches a single character.
To match a literal * or ?, escape the character with \\.
Examples:
ab\\*c matches a metric name that contains ab*c
ab*c matches a metric name that contains abc, abxc, abxxc etc.
ab?c matches a metric name that contains abxc
ab\\?c matches a metric names that contains ab?c
JMX


Default
Commented out.
For WebLogic:
ActiveConnectionsCurrentCount,WaitingForConnectionCurrentCount,PendingRequestCurr
entCount,ExecuteThreadCurrentIdleCount,OpenSessionsCurrentCount,j2eeType
Example
#introscope.agent.jmx.name.filter=ActiveConnectionsCurrentCount,WaitingForConnect
ionCurrentCount,PendingRequestCurrentCount,ExecuteThreadCurrentIdleCount,OpenSess
ionsCurrentCount,j2eeType
Notes
Leave empty to include all MBean data available in the system.
effect.
introscope.agent.jmx.name.jsr77.disable
This property controls whether or not Introscope collects and reports full JSR77 data,
including complex JMX data.
This property is only available for use in the WebLogic and WebSphere
IntroscopeAgent.profile files.
Property settings
True or False
Default
True
Notes
JSR-77 Management support must be provided by the application server in order
for this property to have any effect.
effect.
JMX


User-defined order of MBean information, and simplifies name conversion.
Property settings
A comma-separated, ordered list of keys which should uniquely identify a particular
MBean.
Default
Commented out in default IntroscopeAgent.profile file.
Example
introscope.agent.jmx.name.primarykeys=J2EEServer
Notes
Property settings for WebLogic:
Type
Name
Comment out this property if using WebLogic Server 9.0.
Property settings for WebSphere:
J2EEServer
Application
j2eeType
JDBCProvider
name
mbeanIdentifier
effect.
LeakHunter


introscope.agent.jmx.excludeStringMetrics
Controls whether or not to include string-valued metrics. To enable string-valued
metrics, set this property value to false.
Property settings
True or False
Default
True
Example
introscope.agent.jmx.excludeStringMetrics=true
Notes
Excluding string-valued metrics reduces the overall metric count, improving agent
and EM performance.
effect.
LeakHunter
The following properties configure agent interaction with LeakHunter:
introscope.agent.leakhunter.collectAllocationStackTraces (see page 275)
introscope.agent.leakhunter.enable (see page 276)
introscope.agent.leakhunter.leakSensitivity (see page 276)
introscope.agent.leakhunter.logfile.append (see page 277)
introscope.agent.leakhunter.logfile.location (see page 277)
introscope.agent.leakhunter.timeoutInMinutes (see page 278)
LeakHunter


introscope.agent.leakhunter.collectAllocationStackTraces
Controls whether LeakHunter generates allocation stack traces for potential leaks.
Setting this property to true gives you more precise data about the potential leak's
allocation, but requires additional memory and CPU overhead. For this reason, the
default is false.
Property settings
True or False
Default
False
Example
introscope.agent.leakhunter.collectAllocationStackTraces=
false
Notes
Setting this property to true has the potential to create higher system overhead, in
CPU usage and memory.
LeakHunter


introscope.agent.leakhunter.enable
Controls whether the LeakHunter feature is enabled. Set the value to true to enable
LeakHunter.
Property settings
True or False
Default
False
Example
introscope.agent.leakhunter.enable=false
Notes
Turning on this option has the potential to create higher CPU and memory usage.
You should only enable this feature when other metrics suggest there is a memory
leak.
effect.
introscope.agent.leakhunter.leakSensitivity
Controls the sensitivity level of the LeakHunter leak detection algorithm. A higher
sensitivity setting will result in more potential leaks reported and a lower sensitivity will
result in fewer potential leaks reported.
Property settings
The leak sensitivity range must be positive integer value from 1 (low) to10 (high).
Default
5
Example
introscope.agent.leakhunter.leakSensitivity=5
Notes
LeakHunter


introscope.agent.leakhunter.logfile.append
Specifies whether to replace the log file or add information to an existing log file on
application restart.
Property settings
True or False
False replaces the log file.
True adds information to an existing log file.
Default
False
Example
introscope.agent.leakhunter.logfile.append=false
Notes
introscope.agent.leakhunter.logfile.location
Controls the location for the LeakHunter.log file. You can specify an absolute or relative
path to the file name. Relative paths resolve relative to the <Agent_Home> directory.
Leave the value blank or commented out if you do not want LeakHunter to record data
in a log file.
Default
The default path is ../../logs/LeakHunter.log. This locates the log file in the
<Agent_Home>logs directory.
Example
introscope.agent.leakhunter.logfile.location=../../logs/LeakHunter.log
Notes
LeakHunter


Controls the length of time (in minutes) that LeakHunter spends looking for new
potential leaks. After the time specified, LeakHunter stops looking for new potential
leaks. It continues tracking the previously identified potential leaks.
Property settings
Must be a positive integer (no negative numbers).
Default
The default is 120 minutes.
Example
introscope.agent.leakhunter.timeoutInMinutes=120
Notes
Set the value to zero if you want LeakHunter to always look for new potential leaks.
effect.
introscope.agent.leakhunter.ignore.<number>
Use this to ignore any class matching any supplied patterns. Ten classes have been
supplied by default - comment out any you want to use.
Property settings
A comma-separated list of class matching patterns.
Default
None
Logging


Example
introscope.agent.leakhunter.ignore.4=java.util.SubList
introscope.agent.leakhunter.ignore.5=com.sun.faces.context.BaseContextMap$EntrySe
t
introscope.agent.leakhunter.ignore.6=com.sun.faces.context.BaseContextMap$Key
Notes
Some collections cannot be used with LeakHunter. In order for a collection to be
LeakHunter-safe, it must be safe to call size() at any time, from any thread.
You can used the "*" wildcard.
Logging
The following properties configure agent logging options:
log4j.logger.IntroscopeAgent (see page 280)
log4j.appender.logfile.File (see page 281)
log4j.logger.IntroscopeAgent.inheritance (see page 281)
log4j.appender.pbdlog.File (see page 282)
log4j.appender.pbdlog (see page 282)
log4j.appender.pbdlog.layout (see page 283)
log4j.appender.pbdlog.layout.ConversionPattern (see page 283)
log4j.additivity.IntroscopeAgent.inheritance (see page 284)
Logging


log4j.logger.IntroscopeAgent
This property controls both the logging level and the output location for log information.
Property settings
Level of detail value can be:
INFO
VERBOSE#com.wily.util.feedback.Log4JSeverityLevel
Destination value can be:
console
logfile
both console and logfile
Default
INFO, console, logfile
Example
log4j.logger.IntroscopeAgent=INFO,console,logfile
To disable agent logging, remove the options from this property as follows:
Before:
log4j.logger.IntroscopeAgent=INFO, console, logfile
After:
log4j.logger.IntroscopeAgent=
Notes
Changes to this property take effect immediately and do not require a restart of the

Logging


log4j.appender.logfile.File
Specifies the name and location of IntroscopeAgent.log file if logfile is specified in
log4j.logger.IntroscopeAgent. The filename is relative to the directory that contains the
agent profile.
Default
IntroscopeAgent.log
Example
log4j.appender.logfile.File=../../logs/IntroscopeAgent.log
Notes
System properties (Java command line -D options) are expanded as part of the file
name. For example, if Java is started with -Dmy.property=Server1, then
log4j.appender.logfile.File=../../logs/Introscope-${my.property}.log is expanded to:
log4j.appender.logfile.File=../../logs/Introscope-Server1.log.
log4j.logger.IntroscopeAgent.inheritance
Controls log level and destination for log messages about classes that require
instrumentation.
Property settings
To configure logging of classes that have not been instrumented because they extend a
supertype or interface, set this property to: INFO, pbdlog
For information about inheritance class logging see Controlling directive logging (see
page 84).
Default
None
Example
log4j.logger.IntroscopeAgent.inheritance=INFO,pbdlog
Logging


log4j.appender.pbdlog.File
Identifies a log file for messages about classes that require instrumentation.
Property settings
supertype or interface set to: pbdupdate.log
Default
None
Example
log4j.appender.pbdlog.File=../../pbdupdate.log
log4j.appender.pbdlog
Specifies a package for logging messages about classes that require instrumentation.
Property settings
supertype or interface, set this property to:
com.wily.introscope.agent.AutoNamingRollingFileAppender
Default
None
Example
log4j.appender.pbdlog=com.wily.introscope.agent.AutoNamingRollingFileAppender
Logging


log4j.appender.pbdlog.layout
Specifies rules for logging messages about classes that require instrumentation.
Property settings
supertype or interface, set this property to: com.wily.org.apache.log4j.PatternLayout
Default
None
Example
log4j.appender.pbdlog.layout=com.wily.org.apache.log4j.PatternLayout
log4j.appender.pbdlog.layout.ConversionPattern
Specifies rules for logging messages about classes that require instrumentation.
Property settings
supertype or interface, set this property to:
%d{M/dd/yy hh:mm:ss a z} [%-3p] [%c] %m%n
Default
None
Example
log4j.appender.pbdlog.layout.ConversionPattern=%d{M/dd/yy hh:mm:ss a z} [%-3p] [%c]
%m%n
Metric count


log4j.additivity.IntroscopeAgent.inheritance
Causes the directives for multiple level inheritance to be logged only in the
pbdupdate.log file.
Property settings
True or False
Default
True
Example
log4j.additivity.IntroscopeAgent.inheritance=true
Notes
To configure the logging of multiple level inheritance directives in the pbdupdate.log
only, add this property to the agent profile and set to false.
Metric count
The following property affects where you will see the Metric Count metric in the
Investigator:
introscope.ext.agent.metric.count (see page 285)
Multiple inheritance


introscope.ext.agent.metric.count
Controls where you will see the Metric Count metric in the Investigator. By default, the
Metric Count is displayed as under the Custom Metric Agent node. If you want to see the
Metric Count metric under the Agent Stats node, add this property to the
Property settings
True or False
Default
Not present in the IntroscopeAgent.profile; False
Example
introscope.ext.agent.metric.count=true
Notes
Add this property to the IntroscopeAgent.profile and set it to true to see the Metric
Count metric under the Agent Stats node.
For directives based on interfaces or super classes, the agent is unable to detect
multiple inheritance and hence those classes are not instrumented. Enable the following
properties to determine those cases after the application server or the agent process
starts up. These properties log classes which need to be instrumented but have not
been and relies on dynamic instrumentation to affect the changes.
introscope.autoprobe.hierarchysupport.enabled (see page 286)
introscope.autoprobe.hierarchysupport.runOnceOnly (see page 286)
introscope.autoprobe.hierarchysupport.pollIntervalMinutes (see page 287)
introscope.autoprobe.hierarchysupport.executionCount (see page 287)
introscope.autoprobe.hierarchysupport.disableLogging (see page 288)
introscope.autoprobe.hierarchysupport.disableDirectivesChange (see page 288)


introscope.autoprobe.hierarchysupport.enabled
For agents that run under JDK 1.5 using AutoProbe and dynamic instrumentation, you
can use this property to enable instrumentation of classes that extend a supertype or
interface.
Property settings
True or False
Default
True
Example
introscope.autoprobe.hierarchysupport.enabled=true
Notes
introscope.autoprobe.hierarchysupport.runOnceOnly
If you have enabled instrumentation of classes that extend a supertype or interface, you
can use this property to control whether the utility that enables the feature runs only
once, or at a specified interval.
Change this property to true only if you need detection on a periodic basis.
Property settings
True or False
Default
False
Example
introscope.autoprobe.hierarchysupport.enabled=false
Notes
Logging properties related to dynamic instrumentation are defined in Logging.
effect.


introscope.autoprobe.hierarchysupport.pollIntervalMinutes
The polling interval to check for classes which could not be instrumented due to
multiple inheritance. In most cases this happens only once; however, a conservative
value is recommended to account for application server initialization.
Default
5
Example
introscope.autoprobe.hierarchysupport.pollIntervalMinutes=5
Notes
introscope.autoprobe.hierarchysupport.executionCount
If you need the polling interval to run a finite times instead of running it only once or
running it periodically, use this property to specify the exact number of times the polling
interval is run. Always use this property to specify the exact number of times it should
run.
Using this property overrides the run once only setting.
Property settings
A positive integer.
Default
3
Example
introscope.autoprobe.hierarchysupport.executionCount=3
Notes
Platform monitoring


introscope.autoprobe.hierarchysupport.disableLogging
Uncomment this property if you do not need to log the classes being detected. Only
uncomment this property if dynamic instrumentation is enabled.
Property settings
True or False
Default
True
Example
#introscope.autoprobe.hierarchysupport.disableLogging=true
Notes
introscope.autoprobe.hierarchysupport.disableDirectivesChange
Uncomment this property to only log the changes and disable the triggering of dynamic
instrumentation.
Property settings
True or False
Default
True
Example
introscope.autoprobe.hierarchysupport.disableDirectivesChange=true
Notes
Platform monitoring
The following property configures platform monitoring metrics:
introscope.agent.platform.monitor.system (see page 289)
Remote configuration


introscope.agent.platform.monitor.system
Name of operating system to load a platform monitor for.
Property settings
See Troubleshooting platform monitoring (see page 197) for more information about
the options for this property.
Default
Commented out; varies by platform.
Example
introscope.agent.platform.monitor.system=Solaris
Notes
Remote configuration
The following properties allow remote configuration of the Java Agent:
introscope.agent.remoteagentconfiguration.enabled (see page 289)
introscope.agent.remoteagentconfiguration.allowedFiles (see page 290)
This property enables or disables remote configuration of the agent.
Property settings
True or False
Default
True
Example
introscope.agent.remoteagentconfiguration.enabled=true
Notes
Security


introscope.agent.remoteagentconfiguration.allowedFiles
This property lists the exact list of files that are allowed to be remotely transferred to
this agent.
Property settings
domainconfig.xml
Default
domainconfig.xml
Example
introscope.agent.remoteagentconfiguration.allowedFiles=domainconfig.xml
Notes
Security
The following property configures security of HTTP headers being sent to CA CEM:
introscope.agent.decorator.security (see page 290)
introscope.agent.decorator.security
This property determines the format of decorated HTTP response headers, which are
sent to CA CEM.
Property settings
Clear: clear text encoding
Encrypted: header data is encrypted
Default
Clear
Example
introscope.agent.decorator.security=clear
Servlet header decorator


Servlet header decorator
The following property enables correlation of transactions between CA CEM and
Introscope:
introscope.agent.decorator.enabled (see page 291)
introscope.agent.decorator.enabled
If this Boolean value is set to true, it configures the agent to add additional performance
monitoring information to HTTP response headers. ServletHeaderDecorator attaches
the GUID to each transaction and inserts the GUID into an HTTP header, for example:
x-apm-info
Property settings
True or False
Default
False
Example
introscope.agent.decorator.enabled=false
Socket metrics
Generation of I/O Socket metrics may be restricted by the following parameters:
introscope.agent.sockets.reportRateMetrics (see page 292)
introscope.agent.io.socket.client.hosts (see page 292)
introscope.agent.io.socket.client.ports (see page 293)
introscope.agent.io.socket.server.ports (see page 293)
Socket metrics


introscope.agent.sockets.reportRateMetrics
Enables reporting of individual socket's input/output (I/O) bandwidth rate metrics.
Property settings
True or False
Default
False
Example
introscope.agent.sockets.reportRateMetrics=false
Notes
Only functions if ManagedSocketTracing is enabled and SocketTracing is disabled.
See Backwards compatibility (see page 135) for more information.
effect.
introscope.agent.io.socket.client.hosts
Restrict socket client connections instrumented to those with specified remote hosts.
Property settings
A comma-separate list of values.
Example
introscope.agent.io.socket.client.hosts=
Notes
If any individual value is invalid, it will be ignored.
If any parameter is not defined, or after exclusion of any invalid values is an empty
list, no restriction will apply to that parameter.
SQL Agent


introscope.agent.io.socket.client.ports
Restrict socket client connections instrumented to those with specified remote ports
Property settings
Example
introscope.agent.io.socket.client.ports=
Notes
introscope.agent.io.socket.server.ports
Restrict socket client connections instrumented to those using specified local ports.
Property settings
Example
introscope.agent.io.socket.server.ports=
Notes
SQL Agent
You can configure aspects of the SQL Agent.
SQL Agent


More information:
introscope.agent.sqlagent.normalizer.extension (see page 294)
introscope.agent.sqlagent.normalizer.regex.matchFallThrough (see page 295)
introscope.agent.sqlagent.normalizer.regex.keys (see page 296)
introscope.agent.sqlagent.normalizer.regex.key1.pattern (see page 296)
introscope.agent.sqlagent.normalizer.regex.key1.replaceAll (see page 297)
introscope.agent.sqlagent.normalizer.regex.key1.replaceFormat (see page 297)
introscope.agent.sqlagent.normalizer.regex.key1.caseSensitive (see page 298)
introscope.agent.sqlagent.sql.artonly (see page 298)
introscope.agent.sqlagent.sql.rawsql (see page 299)
introscope.agent.sqlagent.sql.turnoffmetrics (see page 299)
introscope.agent.sqlagent.sql.turnofftrace (see page 299)

This property specifies the name of the SQL normalizer extension that is used to
override the preconfigured normalization scheme.
To make custom normalization extension work, the value of its manifest attribute
com-wily-Extension-Plugin-{pluginName}-Name must match the value given in this
property.
If you specify a comma-separated list of names, the Agent uses the default normalizer
extension.
For example, with the following setting RegexSqlNormalizer is used for normalization:
introscope.agent.sqlagent.normalizer.extension=ext1, ext2
This property limits how much of a SQL statement appears in the Investigator tree for
SQL Agent metrics, in bytes.
Property settings
The name of the SQL normalizer extension that is used to override the preconfigured
normalization scheme.
Default
RegexSqlNormalizer
Example
SQL Agent


Notes
If you use the default setting, you also must configure the regular expressions SQL
statement normalizer properties:
introscope.agent.sqlagent.normalizer.regex.matchFallThrough (see page 295)
introscope.agent.sqlagent.normalizer.regex.keys (see page 296)
introscope.agent.sqlagent.normalizer.regex.key1.pattern (see page 296)
introscope.agent.sqlagent.normalizer.regex.key1.replaceAll (see page 297)
introscope.agent.sqlagent.normalizer.regex.key1.replaceFormat (see page 297)
introscope.agent.sqlagent.normalizer.regex.key1.caseSensitive (see page 298)
Changes to this property take effect immediately and do not require that you restart the
introscope.agent.sqlagent.normalizer.regex.matchFallThrough
Use this property in conjunction with introscope.agent.sqlagent.normalizer.extension
(see page 294) to set the regular expressions SQL statement normalizer. When this
property is set to true, it will evaluate SQL strings against all regex key groups.
The implementation is chained. For example, if SQL matches multiple key groups, the
normalized SQL output from group1 is fed as input to group2, and so on.
If the property is set to false, as soon as a key group matches, the normalized SQL
output from that group is returned.
Property settings
True or False
Default
false
Example
introscope.agent.sqlagent.normalizer.regex.matchFallThrough=false
Notes
SQL Agent


introscope.agent.sqlagent.normalizer.regex.keys
(see page 294) to set the regular expressions SQL statement normalizer. This property
specifies the regex group keys. They are evaluated in order.
Default
key1
Example
Notes
introscope.agent.sqlagent.normalizer.regex.key1.pattern
specifies the regex pattern that will be used to match against the SQL.
Property settings
All valid regex entries allowed by java.util.Regex package can be used here.
Default
.*call(.*\)\.FOO(.*\)
Example
introscope.agent.sqlagent.normalizer.regex.key1.pattern=.*call(.*\)\.FOO(.*\)
Notes
SQL Agent


introscope.agent.sqlagent.normalizer.regex.key1.replaceAll
(see page 294) to set the regular expressions SQL statement normalizer. When this
property is set to false, it will replace the first occurrence of the matching pattern in the
SQL query with the replacement string. If set to true, it will replace all occurrences of
the matching pattern in the SQL query with replacement the string.
Property settings
True or False
Default
false
Example
Notes
introscope.agent.sqlagent.normalizer.regex.key1.replaceFormat
specifies the replacement string format.
Property settings
All valid regex entries allowed by the java.util.Regex package and
java.util.regex.Matcher class can be used here.
Default
$1
Example
introscope.agent.sqlagent.normalizer.regex.key1.replaceFormat=$1
Notes
SQL Agent


introscope.agent.sqlagent.normalizer.regex.key1.caseSensitive
specifies whether the pattern match is sensitive to case.
Property settings
true or false
Default
false
Example
Notes
introscope.agent.sqlagent.sql.artonly
The introscope.agent.sqlagent.sql.artonly property can be used to configure the agent
to create and send only the Average Response Time metric. All SQL agent metrics under
back-ends are affected. When the value for this property is true, performance of the
agent for SQL metrics and transaction traces can improve.
Note: Setting introscope.agent.sqlagent.sql.turnoffmetrics (see page 299)=true
overrides this property.
Important! The following tracer parameter must be set for this property setting to
work:
SetTracerParameter: StatementToConnectionMappingTracer agentcomponent "SQL Agent"
This property is turned off by default:
introscope.agent.sqlagent.sql.artonly=false
Changes to this property take effect immediately and can be made using the
Management interface.
Note: This property does not control custom metrics, such as the connection count.
SQL Agent


introscope.agent.sqlagent.sql.rawsql
The introscope.agent.sqlagent.sql.rawsql property configures the agent to add
unnormalized SQL as a parameter for SQL components in Transaction Trace. When the
value for this property is true, performance of the agent for SQL metrics and transaction
traces can improve.
introscope.agent.sqlagent.sql.rawsql=false
Changes to this property take effect after you restart the managed application.
Important! Enabling this property can result in passwords and sensitive information
being presented in Transaction Trace.
introscope.agent.sqlagent.sql.turnoffmetrics
You can turn off SQL statement metrics to send fewer metrics from the agent to the
Enterprise Manager by using the introscope.agent.sqlagent.sql.turnoffmetrics property.
When the value for this property is true, performance of the agent for SQL metrics and
transaction traces can improve.
work:
introscope.agent.sqlagent.sql.turnoffmetrics=false
This property overrides the introscope.agent.sqlagent.sql.artonly property.
Changes to this property take effect immediately and can be changed using the
Management user interface.
introscope.agent.sqlagent.sql.turnofftrace
The introscope.agent.sqlagent.sql.turnofftrace property controls whether the agent
creates and sends transaction trace components to the Enterprise Manager for SQL
statements under back-ends. When the value for this property is true, performance of
the agent for SQL metrics and transaction traces can improve.
SSL communication


work:
introscope.agent.sqlagent.sql.turnofftrace=false
Changes to this property take effect immediately and can be changed using the
Management user interface.
SSL communication
The agent can connect to the Enterprise Manager over SSL. Use the following properties
to configure that communication:
introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT (see page 219)
introscope.agent.enterprisemanager.transport.tcp.truststore.DEFAULT
introscope.agent.enterprisemanager.transport.tcp.trustpassword.DEFAULT
introscope.agent.enterprisemanager.transport.tcp.keystore.DEFAULT
introscope.agent.enterprisemanager.transport.tcp.keypassword.DEFAULT
introscope.agent.enterprisemanager.transport.tcp.ciphersuites.DEFAULT
Default
localhost
Example
Notes
SSL communication


Specifies the port number on the computer that hosts the Enterprise Manager that
listens for connections from the agent. If you are using Secure Socket Layer (SSL)
protocol, the default port that listens for connections from the agent is 5443.
Default
5443
Example
Notes
Specifies the client socket factory to use for connections from the agent to the
Enterprise Manager when using SSL.
Default
com.wily.isengard.postofficehub.link.net.SSLSocketFactory
Example
ntroscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT=com.wily.is
engard.postofficehub.link.net.SSLSocketFactory
Notes
SSL communication


introscope.agent.enterprisemanager.transport.tcp.truststore.DEFAULT
Location of a truststore containing trusted Enterprise Manager certificates. If no
truststore is specified, the agent trusts all certificates.
Property settings
Either an absolute path or a path relative to the agent's working directory.
Example
introscope.agent.enterprisemanager.transport.tcp.truststore.DEFAULT=/var/trustedc
erts
Notes
On Windows, backslashes must be escaped. For example: C:\\keystore
introscope.agent.enterprisemanager.transport.tcp.trustpassword.DEFAULT
The password for the truststore.
Example
introscope.agent.enterprisemanager.transport.tcp.trustpassword.DEFAULT=
introscope.agent.enterprisemanager.transport.tcp.keystore.DEFAULT
Location of a keystore containing the agent's certificate. A keystore is needed if the
Enterprise Manager requires client authentication.
Property settings
Either an absolute path or a path relative to the agent's working directory.
Example
introscope.agent.enterprisemanager.transport.tcp.keystore.DEFAULT=c:\\keystore
Notes
On Windows, backslashes must be escaped. For example: C:\\keystore
Stall metrics


introscope.agent.enterprisemanager.transport.tcp.keypassword.DEFAULT
The password for the keystore.
Example
introscope.agent.enterprisemanager.transport.tcp.keypassword.DEFAULT=MyPassword76
8
introscope.agent.enterprisemanager.transport.tcp.ciphersuites.DEFAULT
Set the enabled cipher suites.
Property settings
A comma-separated list of cipher suites.
Example
introscope.agent.enterprisemanager.transport.tcp.
ciphersuites.DEFAULT=SSL_DH_anon_WITH_RC4_128_MD5
Notes
If not specified, use the default enabled cipher suites.
Stall metrics
The following properties are for stall metrics:
introscope.agent.stalls.thresholdseconds (see page 303)
introscope.agent.stalls.resolutionseconds (see page 304)
For more information on stall metric properties, see Disabling the capture of stalls as
Events (see page 174).
introscope.agent.stalls.thresholdseconds
This property specifies the number of seconds that an executing process can take before
it is considered a stalled process. To ensure an accurate Stall Count metric, set the stall
threshold to 15 seconds or more. This setting allows time for the Enterprise Manager to
complete its harvest cycle.
Default
Thread dumps


Example
introscope.agent.stalls.thresholdseconds=30
Notes
introscope.agent.stalls.resolutionseconds
This property specifies the frequency that the agent checks for stalls. To ensure an
accurate Stall Count metric, do not set the stall resolution to less than 10 seconds. This
setting allows time for the Enterprise Manager to complete its harvest cycle.
Default
The default is every 10 seconds.
Example
introscope.agent.stalls.resolutionseconds=10
Notes
Thread dumps
These properties enable and configure agent aspects of CA Introscope thread dump
functionality:
introscope.agent.threaddump.enable (see page 305)
introscope.agent.threaddump.deadlockpoller.enable (see page 305)
introscope.agent.threaddump.deadlockpollerinterval (see page 306)
introscope.agent.threaddump.MaxStackElements (see page 306)
Note: For more information about configuring thread dumps, see How to enable and
configure thread dumps (see page 69).

Thread dumps


introscope.agent.threaddump.enable
Enables thread dumps to be collected on an agent JVM and allows users to view the
Thread Dumps tab.
Property settings
True or False
Default
True
Example
introscope.agent.threaddump.enable=true
Notes
Changes to this property take effect immediately and do not require you to restart
the managed application.
This property works together with the IntroscopeEnterpriseManager.properties file
introscope.enterprisemanager.threaddump.enable property, which enables the
Enterprise Manager thread dump functionality if this property is set to true.
introscope.agent.threaddump.deadlockpoller.enable
Enables the Deadlock Count metric in the metric browser tree to display the current
number of deadlocks in the agent JVM.
Property settings
True or False
Default
False
Example
introscope.agent.threaddump.deadlockpoller.enable=true
Notes
Changes to this property take effect immediately and do not require you to restart
the managed application.
Thread dumps


introscope.agent.threaddump.deadlockpollerinterval
Frequency in milliseconds at which CA Introscope polls the agent JVM for deadlocked
threads
Property settings
Integer greater than 0
Default
15000 (milliseconds)
Example
introscope.agent.threaddump.deadlockpollerinterval=15000
Notes
Restart the managed application so changes to this property can take effect.
introscope.agent.threaddump.MaxStackElements
The total number of lines in the thread stack trace determines the size of a CA
Introscope thread dump. This property sets the number of lines allowed in the thread
stack.
Property settings
Integer greater than 0 and no greater than 25,000
Default
12,000
Example
introscope.agent.threaddump.MaxStackElements=12000
Notes
Restart the managed application so that changes to this property can take effect.
Transaction tracing


Transaction tracing
The following properties are for transaction tracing and sampling:
introscope.agent.bizdef.turnOff.nonIdentifying.txn (see page 307)
introscope.agent.transactiontracer.parameter.httprequest.headers (see page 308)
introscope.agent.transactiontracer.parameter.httprequest.parameters (see
page 308)
introscope.agent.transactiontracer.parameter.httpsession.attributes (see page 309)
introscope.agent.transactiontracer.userid.key (see page 309)
introscope.agent.transactiontracer.userid.method (see page 310)
introscope.agent.transactiontrace.componentCountClamp (see page 311)
introscope.agent.crossprocess.compression (see page 312)
introscope.agent.crossprocess.compression.minlimit (see page 313)
introscope.agent.crossprocess.correlationid.maxlimit (see page 314)
introscope.agent.transactiontracer.sampling.enabled (see page 314)
introscope.agent.transactiontracer.sampling.perinterval.count (see page 315)
introscope.agent.transactiontracer.sampling.interval.seconds (see page 315)
introscope.agent.transactiontrace.headFilterClamp (see page 316)
For more information, see Configuring Transaction Trace Options (see page 165).
introscope.agent.bizdef.turnOff.nonIdentifying.txn
Enable or disable tracing of nonidentifying transactions.
Traces for nonidentifying transactions are generated when this property is set to FALSE
in introscopeAgent.profile.
By default traces for nonidentifying transactions are not generated even if the feature is
enabled in the CEM UI.
Property Settings
TRUE or FALSE
Default
TRUE
Transaction tracing


Example
introscope.agent.bizdef.turnOff.nonIdentifying.txn=FALSE
introscope.agent.transactiontracer.parameter.httprequest.headers
Specifies (in comma-separated list) HTTP request header data to capture. Use a comma
separated list.
Default
Commented out; User-Agent
Example
introscope.agent.transactiontracer.parameter.httprequest.headers=User-Agent
Notes
The IntroscopeAgent.profile contains a commented out statement that sets the value of
this property to a null value. The user may optionally uncomment the statement and
supply the desired header names.
introscope.agent.transactiontracer.parameter.httprequest.parameters
Specifies (in comma-separated list) HTTP request parameter data to capture.
Default
Commented out; generic parameters.
Example
introscope.agent.transactiontracer.parameter.httprequest.parameters=parameter1,pa
rameter2
Notes
supply the desired parameter names.
Transaction tracing


introscope.agent.transactiontracer.parameter.httpsession.attributes
Specifies (in comma-separated list) HTTP session attribute data to capture.
Default
Example
introscope.agent.transactiontracer.parameter.httpsession.attributes=attribute1,at
tribute2
Notes
supply the desired parameter names.
introscope.agent.transactiontracer.userid.key
User-defined key string.
Default
Example
#introscope.agent.transactiontracer.parameter.httpsession.attributes=attribute1,a
ttribute2
Notes
supply the correct value if, in your environment, user IDs are accessed using
HttpServletRequest.getHeader or HttpServletRequest.getValue.
For more information, see introscope.agent.transactiontracer.userid.method (see
page 310).
Transaction tracing


introscope.agent.transactiontracer.userid.method
Specifies the method that returns User IDs. The Agent profile includes a commented out
property definition for each of the three allowable values.
Uncomment the appropriate statement, based on whether user ID is accessed by
getRemoteUser, getHeader, or getValue.
Property settings
Allowable values are:
HttpServletRequest.getRemoteUser
HttpServletRequest.getHeader
HttpServletRequest.getValue
Default
Commented out; see options above.
Example
The IntroscopeAgent.profile includes a commented out property definition for each of
the three allowable values. You can uncomment the property you want to use.
introscope.agent.transactiontracer.userid.method=HttpServletRequest.getRemoteUser
#introscope.agent.transactiontracer.userid.method=HttpServletRequest.getHeader
#introscope.agent.transactiontracer.userid.method=HttpSession.getValue
Transaction tracing


introscope.agent.transactiontrace.componentCountClamp
Limits the number of components allowed in a transaction trace.
Default
5000
Important! If the clamp size is increased, the requirements on memory are higher. In
extreme cases, the maximum heap size for the JVM may need to be adjusted or the
managed application could run out of memory.
Example
introscope.agent.transactiontrace.componentCountClamp=5000
Notes
Any Transaction Trace exceeding the clamp is discarded at the agent and a warning
message is logged in the agent log file.
When the set limit is reached, warnings appear in the log, and the trace stops.
Zero is not a valid value. Do not set
introscope.agent.transactiontrace.componentCountClamp=0.
Transaction tracing


introscope.agent.crossprocess.compression
Use this property to reduce the size of cross process transaction tracing data.
Property settings
lzma, gzip, none
Default
lzma
Example
introscope.agent.crossprocess.compression=lzma
Notes
This option will increase agent CPU overhead, but reduce the size of interprocess
headers.
lzma compression is more efficient than gzip, but may use more CPU.
.NET agents do not support the gzip option, so if interoperability is required, do not
use gzip.
Transaction tracing


introscope.agent.crossprocess.compression.minlimit
Use this property to set the minimum length of cross process parameter data length for
which to apply compression.
Property settings
Can be set from 0 to twice the total maximum limit, set in the
introscope.agent.crossprocess.correlationid.maxlimit (see page 314).
If set below the default of 1500, the compression will run more frequently and consume
more CPU overhead. The default setting of 1500 usually results in no impact to CPU
overhead in normal conditions.
Default
1500
Example
introscope.agent.crossprocess.compression.minlimit=1500
Notes
Used with the introscope.agent.crossprocess.compression property above.
Transaction tracing


introscope.agent.crossprocess.correlationid.maxlimit
Maximum size of cross process parameter data allowed.
If the total size of cross process parameter data is more than this limit, even after
applying compression, some data will be dropped and some cross process correlation
functionality will not work properly.
However, this setting will protect user transactions from failing in network transmission
due to too large header size.
Default
4096
Example
introscope.agent.crossprocess.correlationid.maxlimit=4096
Notes
Used with the introscope.agent.crossprocess.compression and
introscope.agent.crossprocess.compression.minlimit properties above
introscope.agent.transactiontracer.sampling.enabled
Uncomment the following property to disable Transaction Tracer Sampling.
Property settings
True or False
Default
False
Example
introscope.agent.transactiontracer.sampling.enabled=false
Notes
Transaction tracing


introscope.agent.transactiontracer.sampling.perinterval.count
This property is normally configured in the Enterprise Manager. Configuring this
property in the agent disables the configuration in the Enterprise Manager. See the CA
APM Configuration and Administration Guide for more information.
Default
1
Example
introscope.agent.transactiontracer.sampling.perinterval.count=1
Notes
introscope.agent.transactiontracer.sampling.interval.seconds
This property is normally configured in the Enterprise Manager. Configuring this
property in the agent disables the configuration in the Enterprise Manager.
Note: For more information, see the CA APM Configuration and Administration Guide.
Default
120
Example
introscope.agent.transactiontracer.sampling.interval.seconds=120
Notes
Transaction tracing


introscope.agent.transactiontrace.headFilterClamp
Specifies the maximum depth of components allowed in head filtering. Head filtering is
the process of examining the start of a transaction for the purpose of potentially
collecting the entire transaction. Head filtering checks each component until the first
blamed component exits. For transaction with very deep call stacks, this can be a
problem if no clamping is applied. The clamp value limits the memory and CPU
utilization impact of this behavior by forcing the agent to only look up to a fixed depth.
Default
30
Warning! If the clamp size is increased, the requirement on memory is higher. Garbage
collection behavior will be affected, which will have an application-wide performance
impact.
Example
introscope.agent.transactiontrace.headFilterClamp=30
Notes
Any Transaction Trace whose depth exceeds the clamp will no longer be examined
for possible collection unless some other mechanism, such as sampling or
user-initiated transaction tracing, is active to select the transaction for collection.
URL grouping


introscope.agent.ttClamp
This property limits the number of transactions that are reported by the agent per
reporting cycle.
Property settings
Integers.
Default
50
Example
introscope.agent.ttClamp=50
Notes
effect.
If the property is not set (left blank), the value defaults to 200.
URL grouping
The following properties are for configuring URL Groups for frontend metrics:
introscope.agent.urlgroup.keys (see page 318)
introscope.agent.urlgroup.group.default.pathprefix (see page 318)
introscope.agent.urlgroup.group.default.format (see page 318)
For more information, see Using URL groups (see page 157).
URL grouping


introscope.agent.urlgroup.keys
Configuration settings for Frontend naming.
Default
Default
Example
introscope.agent.urlgroup.keys=default
Notes
If a URL address belongs to two URL Groups, the order in which you list the keys for the
URL Groups in this property is important. The URL Group defined by the narrower
pattern should precede the URL Group specified by the broader pattern.
For example, if the URL Group with key alpha contains a single address, and the URL
Group with key beta includes all addresses on the network segment that contains the
address in the first URL Group, alpha should precede beta in the keys parameter.
introscope.agent.urlgroup.group.default.pathprefix
Configuration settings for frontend naming.
Default
*
Example
introscope.agent.urlgroup.group.default.pathprefix=*
introscope.agent.urlgroup.group.default.format
Configuration settings for Frontend naming.
Default
Default
Example
introscope.agent.urlgroup.group.default.format=default
WebSphere PMI


WebSphere PMI
The following properties configure WebSphere PMI metrics:
introscope.agent.pmi.enable (see page 320)
introscope.agent.pmi.enable.alarmManagerModule (see page 320)
fintroscope.agent.pmi.enable.beanModule (see page 321)
introscope.agent.pmi.enable.cacheModule (see page 321)
introscope.agent.pmi.enable.connectionPoolModule (see page 322)
introscope.agent.pmi.enable.hamanagerModule (see page 322)
introscope.agent.pmi.enable.j2cModule (see page 323)
introscope.agent.pmi.enable.jvmpiModule (see page 323)
introscope.agent.pmi.enable.jvmRuntimeModule (see page 324)
introscope.agent.pmi.enable.objectPoolModule (see page 324)
introscope.agent.pmi.enable.orbPerfModule (see page 325)
introscope.agent.pmi.enable.schedulerModule (see page 325)
introscope.agent.pmi.enable.servletSessionsModule (see page 326)
introscope.agent.pmi.enable.systemModule (see page 326)
introscope.agent.pmi.enable.threadPoolModule (see page 327)
introscope.agent.pmi.enable.transactionModule (see page 327)
introscope.agent.pmi.enable.webAppModule (see page 328)
introscope.agent.pmi.enable.webServicesModule (see page 328)
introscope.agent.pmi.enable.wlmModule (see page 329)
introscope.agent.pmi.enable.wsgwModule (see page 329)
introscope.agent.pmi.filter.objrefModule (see page 330)
These properties are found in the IntroscopeAgent.websphere.profile file, or the default
agent profile for a WebSphere installation.
WebSphere PMI


introscope.agent.pmi.enable
Enables collection of data from WebSphere PMI.
Property settings
True or False
Default
True
Example
introscope.agent.pmi.enable=true
Notes
introscope.agent.pmi.enable.alarmManagerModule
Enables collection of PMI alarm manager data when set to true.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.alarmManagerModule=false
Notes
The alarm manager data category must be turned on in WebSphere to be visible as
Introscope data.
effect.
WebSphere PMI


introscope.agent.pmi.enable.beanModule
Enables collection of PMI bean data.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.beanModule=false
introscope.agent.pmi.enable.cacheModule
Enables collection of PMI cache data when set to true.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.cacheModule=false
Notes
The cache data category must be turned on in WebSphere to be visible as
Introscope data.
effect.
WebSphere PMI


introscope.agent.pmi.enable.connectionPoolModule
Enables collection of PMI connectionPool data.
Property settings
True or False
Default
True
Example
introscope.agent.pmi.enable.connectionPoolModule=true
introscope.agent.pmi.enable.hamanagerModule
Enables collection of PMI manager data when set to true.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.hamanagerModule=false
Notes
The manager data category must be turned on in WebSphere to be visible as
Introscope data.
effect.
WebSphere PMI


introscope.agent.pmi.enable.j2cModule
Enables collection of PMI J2C data when set to true.
Property settings
True or False
Default
True
Example
introscope.agent.pmi.enable.j2cModule=true
Notes
The J2C data category must be turned on in WebSphere to be visible as Introscope
data.
effect.
introscope.agent.pmi.enable.jvmpiModule
Enables collection of PMI JVM PI data.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.jvmpiModule=false
Notes
For data to be provided to this module, JVMPI must be turned on in WebSphere.
WebSphere PMI


introscope.agent.pmi.enable.jvmRuntimeModule
Enables collection of PMI JVM runtime data.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.jvmRuntimeModule=false
Notes
For data to be provided to this module, JVMPI must be turned on in WebSphere.
effect.
introscope.agent.pmi.enable.objectPoolModule
Enables collection of PMI object pool data when set to true.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.objectPoolModule=false
Notes
The object pool data category must be turned on in WebSphere to be visible as
Introscope data.
effect.
WebSphere PMI


introscope.agent.pmi.enable.orbPerfModule
Enables collection of PMI orbPerf data when set to true.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.orbPerfModule=false
Notes
The orbPerf data category must be turned on in WebSphere to be visible as
Introscope data.
effect.
introscope.agent.pmi.enable.schedulerModule
Enables collection of PMI scheduler data when set to true.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.schedulerModule=false
Notes
The scheduler data category must be turned on in WebSphere to be visible as
Introscope data.
effect.
WebSphere PMI


introscope.agent.pmi.enable.servletSessionsModule
Enables collection of PMI servletSessions data.
Property settings
True or False
Default
True
Example
introscope.agent.pmi.enable.servletSessionsModule=true
Notes
introscope.agent.pmi.enable.systemModule
Enables collection of PMI system data when set to true.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.systemModule=false
Notes
The system data category must be turned on in WebSphere to be visible as
Introscope data.
effect.
WebSphere PMI


introscope.agent.pmi.enable.threadPoolModule
Enables collection of PMI thread pool data when set to true.
Property settings
True or False
Default
True
Example
introscope.agent.pmi.enable.threadPoolModule=true
Notes
The thread pool data category must be turned on in WebSphere to be visible as
Introscope data.
effect.
introscope.agent.pmi.enable.transactionModule
Enables collection of PMI transaction data.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.transactionModule=false
Notes
WebSphere PMI


introscope.agent.pmi.enable.webAppModule
Enables collection of PMI webApp data.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.webAppModule=false
Notes
introscope.agent.pmi.enable.webServicesModule
Enables collection of PMI web services data when set to true.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.webServicesModule=false
Notes
The web services data category must be turned on in WebSphere to be visible as
Introscope data.
effect.
WebSphere PMI


introscope.agent.pmi.enable.wlmModule
Enables collection of PMI WLM data when set to true.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.wlmModule=false
Notes
The WLM data category must be turned on in WebSphere to be visible as
Introscope data.
effect.
introscope.agent.pmi.enable.wsgwModule
Enables collection of PMI WSGW data when set to true.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.enable.wsgwModule=false
Notes
The WSGW data category must be turned on in WebSphere to be visible as
Introscope data.
effect.
WLDF metrics


introscope.agent.pmi.filter.objrefModule
Controls for hard-coded filters.
The objref filter filters out names ending with "@xxxxx" where "xxxxx" is a numeric
string.
Property settings
True or False
Default
False
Example
introscope.agent.pmi.filter.objrefModule=false
Notes
You must restart the managed application before changes to this property take effect
WLDF metrics
The following properties configure WLDF metrics:
introscope.agent.wldf.enable (see page 330)
introscope.agent.wldf.enable
This property enables the collection of WLDF metrics.
Property settings
True or False
Default
False
Example
introscope.agent.wldf.enable=false


Appendix B: Alternative Methods for
Instrumentation

This section describes alternate methods for instrumenting application when you cannot
use JVM AutoProbe. CA Technologies recommends using JVM AutoProbe over the
alternatives described in this section in all cases. However, if you cannot use JVM
AutoProbe for a specific application server, you can use instructions in this section to
instrument your applications.
Java Agent Deployment on Other Application Servers (see page 331)
Configure Sun ONE to Use AutoProbe (see page 332)
Configure Oracle to Use AutoProbe (see page 333)
Configure WebLogic Server (see page 334)
Configuring HTTP servlet tracing (see page 335)
Create an AutoProbe Connector File (see page 335)
About running ProbeBuilder manually (see page 339)
Configure AutoProbe for WebSphere for z/OS (see page 339)
Java Agent Deployment on Other Application Servers
JVM AutoProbe is a commonly used method for instrumenting applications. CA
Technologies highly recommends using JVM AutoProbe to instrument your applications.
However, you can use Application Server AutoProbe when you are running JVMs 1.4 at a
maximum on the following application servers:
Sun ONE 7.0
Application Server AutoProbe is only supported on Sun ONE version 7 application
server.
Oracle 10g 10.0.3
Application Server AutoProbe is only supported on Oracle version 10g 10.0.3
application server.
WebSphere or WebLogic.
Configure Sun ONE to Use AutoProbe


Important! Application Server AutoProbe is not supported on:
Any JVM beyond 1.5
OS/400
Important! Use only one method to instrument your applications. If you already use
JVM AutoProbe, do not use Application Server AutoProbe.
When starting the application server, avoid using the hyphen (-) character as an
identifier for a classname. CA Introscope does not parse this character, and using it can
lead to class loading errors in the agent logs.
More information:
Configure Sun ONE to Use AutoProbe (see page 332)
Configure Oracle to Use AutoProbe (see page 333)
Configure IBM WebSphere to Use the Java Agent (see page 46)
Configure Oracle WebLogic to Use the Java Agent (see page 40)

Configure Sun ONE to Use AutoProbe
Valid for: Sun ONE 7.0
You can configure Sun ONE installations to use AutoProbe to instrument applications.
Follow these steps:
Note: The use of "..." in the following .xml examples indicates that information is not
shown in the .xml code. This information is not relevant to the examples.
1. Log in as Administrator or Root.
2. Navigate to the following location and open the server.xml file:
<Sun ONE install dir>/domains/domain1/server1/config/
Note: The item separator is a colon (:).
3. Add the full path of wily/Agent.jar to the "server-classpath" property of the
java-config element in the server.xml file. For example:
<java-config ... server-classpath="/sw/sun/sunone7/wily/Agent.jar:..." ...>
Configure Oracle to Use AutoProbe


4. Navigate to the java-config element:
Add the bytecode-preprocessors property and set it to the value
com.wily.introscope.api.sun.appserver.SunONEAutoProbe.
For example:
<java-config ...
bytecode-preprocessors="com.wily.introscope.api.sun.appserver.SunONEAutoP
robe">
Add a jvm-options element and define the location of the agent profile. Define
either com.wily.introscope.agentProfile, or
com.wily.introscope.agentResource.
An example of com.wily.introscope.agentProfile follows:
<java-config ...>
...
<jvm-options>-Dcom.wily.introscope.agentProfile=/sw/sun/sunone7/wily/core
/config/IntroscopeAgent.profile </jvm-options>
</java-config>
An example of com.wily.introscope.agentResource follows:
<java-config ...>
...
<jvm-options>-Dcom.wily.introscope.agentResource=<virtual path
to>/IntroscopeAgent.profile</jvm-options>
</java-config>
(Optional) If you configured com.wily.introscope.agentResource, add the
resource file to the server classpath.
5. Collect servlet data by configuring tracer groups.
Configure Oracle to Use AutoProbe
Valid for: Oracle 10g 10.0.3
You can configure Oracle installations to use AutoProbe to instrument applications.
Follow these steps:
1. Add Agent.jar to the application server classpath.
2. Set the system property oracle.classpreprocessor.classes with the value of
com.wily.introscope.api.oracle.OracleAutoProbe.
3. Set the system property oracle.j2ee.class.preprocessing with the value of true.
4. Run this command at the command line:
-Dcom.wily.introscope.probebuilder.oracle.enable=true
Configure WebLogic Server


5. Restart the Oracle Application Server 10g, using this command:
java
-Doracle.classpreprocessor.classes=com.wily.introscope.api.oracle.OracleAutoP
robe -Doracle.j2ee.class.preprocessing=true
-Dcom.wily.introscope.probebuilder.oracle.enable=true -classpath
oc4j.jar:<path to wily install dir>/wily/Agent.jar
com.evermind.server.OC4JServer -config <path to oracle install
dir>/config/server.xml
Important! Users running Oracle 10g Release 2 using Sun JDK 1.42 on a Windows
host must use the ^ (caret) character to escape a forward slash. For example:
-Xbootclasspath^/p:<IntroscopeAgent.jar path>
6. Collect servlet data by configuring tracer groups.
More information:

Configure WebLogic Server
You can configure WebLogic Server to use AutoProbe to instrument applications.
Follow these steps:
1. Edit the classpath in the application startup script (such as startMedRecServer.cmd)
to include the wily/Agent.jar file.
2. Set the following property in the application startup script on the Java command
line with the -D option to activate Introscope AutoProbe:
-Dweblogic.classloader.preprocessor=
com.wily.introscope.api.weblogic.PreProcessor
3. Configure Tracer Groups to collect servlet data by configuring HTTP servlet tracing
(see page 335).
Configuring HTTP servlet tracing


Configuring HTTP servlet tracing
Before you use AutoProbe with your application servers to instrument applications, you
must configure Tracer Groups in the toggles-full.pbd and toggles-typical.pbd files. This
will enable servlet data to be collected.
You will turn one Tracer Group off, and turn another Tracer Group on.
To configure HTTP servlet tracing
1. Navigate to the <your-application-server-home>/wily/core/config/toggles-full.pbd
file and open it.
2. Go to the HTTP Servlets Configuration section of the PBD.
3. Turn off the HTTPServletTracing Tracer Group by placing a pound sign at the
beginning of the line. For example:
#TurnOn: HTTPServletTracing
4. Turn on the HTTPAppServerAutoProbeServletTracing Tracer Group by removing the
pound sign from the beginning of the line. For example:
TurnOn: HTTPAppServerAutoProbeServletTracing
5. Repeat steps 2-4 for <your-app-server-home>/wily/core/config/toggles-typical.pbd
file.
Create an AutoProbe Connector File
The deprecated JVM AutoProbe method requires a connector .jar file to operate
correctly. To create the AutoProbe Connector, follow this procedure. If your JVM is 1.5,
follow the instructions in JVM AutoProbe.
Follow these steps:
1. Change the working directory to wily/connectors under the installation directory.
2. Run the Create AutoProbe Connector tool using one of these commands:
To specify the JVM using the JVM that is running the tool:
java -jar CreateAutoProbeConnector.jar -current
To specify the JVM by passing the JVM directory on the command line:
java -jar CreateAutoProbeConnector.jar -jvm <directory>
The output is a file with the form: wily/connectors/AutoProbeConnector.jar
3. (Optional) Rename the created .jar file to be more manageable and universally
accepted. For example:
wily/connectors/AutoProbeConnector131_02_Sun.jar
wily/connectors/AutoProbeConnector130_IBM.jar


More information:

Running the AutoProbe Connector for a JVM
After you create the AutoProbe Connector for the Sun or IBM JVM, you run the created
file to instrument your applications. The way that you run the Connector depends on
the application server you use. For more information, see the appropriate section for
your application server.
To run the AutoProbe Connector for SAP J2EE 6.20
1. Open the file:
<drive>:\usr\sap\<J2EE_ENGINE_ID>\j2ee\j2ee_<INSTANCE>\cluster\
server\cmdline.properties
2. Append these commands to JavaParameters section:
-Xbootclasspath/p:PathToAutoProbeConnectorJar;PathToAgentJar
-Dcom.wily.introscope.agentProfile=<path-to-IntroscopeAgent.profile>
-Dcom.wily.introscope.agent.agentName=<yourAgentName>
For example:
Xbootclasspath/p:C:/usr/sap/P602/j2ee/j2ee_00/ccms/wily/connectors/AutoProbeC
onnector.jar;C:/usr/sap/P602/j2ee/j2ee_00/ccms/wily/Agent.jar
-Dcom.wily.introscope.agentProfile=C:/usr/sap/P602/j2ee/j2ee_00/ccms/wily/cor
e/config/IntroscopeAgent.profile
To run the AutoProbe Connector for NetWeaver 04/SAP J2EE 6.40
1. Run the SAP J2EE Configtool.
2. Select the server that you want to modify.
3. Add these new java parameters in the Java Parameters field:
-Xbootclasspath/p:PathToAutoProbeConnectorJar;PathToAgentJar
-Dcom.wily.introscope.agentProfile=<path-to-IntroscopeAgent.profile>
For example:
Xbootclasspath/p:D:/usr/sap/ccms/wily/connectors/AutoProbeConnector.jar;D:/us
r/sap/ccms/wily/Agent.jar
-Dcom.wily.introscope.agentProfile=D:/usr/sap/ccms/wily/core/config/Introscop
eAgent.profile
Note: For NetWeaver 6.40 on Windows, the slashes for these java parameters must
be forward slashes.
4. Click Disk to save.


5. Repeat steps 2 - 4 for each server.
7. To verify that Configtool changes were made, open the file:
<drive>:\usr\sap\ccms\P66\JC00\j2ee\cluster\instance.properties
8. Look for a line beginning with ID<server_id>.JavaParameters, and confirm that it
contains the lines you entered.
To run the AutoProbe Connector for Sun ONE
1. Log in as Administrator or Root.
You must be logged in with Administrator or Root permissions to add Introscope
information to startup scripts for Sun ONE 7.0.
2. Open the server.xml file, which is located at:
<SunONE install dir>/domains/domain1/server1/config/
3. Add this line to the server.xml file:
<jvm-options>
-Xbootclasspath/p:PathToAutoProbeConnectorJar:PathToAgentJar
</jvm-options>
The item separator is a colon (:). For example:
<jvm-options>
-Xbootclasspath/p:/sw/sun/sunone7/wily/connectors/AutoProbeConnector.jar:/sw/
sun/sunone7/wily/Agent.jar
</jvm-options>
To run the AutoProbe Connector for Oracle 10g
To run the AutoProbe Connector, modify the bootstrap classpath:
-Xbootclasspath/p:wily/connectors/AutoProbeConnectorJar:PathToAgentJar
Users running Oracle 10g Release 2 using Sun JDK 1.42 on a Windows host must use the
^ (caret) character to escape a forward slash. For example:
-Xbootclasspath^/p:<IntroscopeAgent.jar path>
Different versions of WebLogic use different versions of Java to run. If you use Java 1.4
or earlier, use the following steps to run the AutoProbe connector. If you use Java 1.5 or
later, see JVM AutoProbe for more information.
To run the AutoProbe Connector for WebLogic
1. Edit the bootstrap classpath in the application startup script to include the
AutoProbeConnector.jar you created (such as startMedRecServer.cmd) using this
command:
-Xbootclasspath/p:PathToAutoProbeConnectorJar:PathToAgentJar


Add the -X switch to the final start command at the end of the script, after the
JAVA_VM and JAVA_OPTIONS. The following excerpt shows the correct place to
insert the switch:
"$JAVA_HOME/bin/java" ${JAVA_VM} ${MEM_ARGS} ${JAVA_OPTIONS}
-Xbootclasspath/p:${WL_HOME}/wily/connectors/AutoProbeConnector.jar:${WL_HOME
}/wily/Agent.jar
-Dweblogic.Name=${SERVER_NAME}
-Dweblogic.management.username=${WLS_USER}
-Dweblogic.management.password=${WLS_PW}
-Dweblogic.ProductionModeEnabled=${PRODUCTION_MODE}
-Djava.security.policy="${WL_HOME}/server/lib/weblogic.policy"
weblogic.Server
2. If you are using something other than the default bootstrap classpath, add the
Agent.jar and AutoProbeConnector.jar files to the beginning of your customized
bootstrap classpath.
To run the AutoProbe Connector for WebLogic with JRockit JVM
Add the following command-line options when starting the JVM:
-Xbootclasspath/a:<PathToAgentJar>
-Xmanagement:class=com.wily.introscope.api.jrockit.AutoProbeLoader
To run the AutoProbe Connector for other application servers
To run the AutoProbe Connector, add the Agent.jar and the AutoProbe Connector
to the Application Server bootstrap classpath using this command:
-Xbootclasspath/p:wily/connectors/AutoProbeConnector.jar:PathToAgentJar
Example: Use Xbootclasspath to Instrument WAS
This example demonstrates how you can use the Xbootclasspath option to instrument
a WebSphere Application Server. This option allows you to override entities (classes,
jars, directories, zips) that are loaded at bootstrap time by default by the JVM. You
create an AutoprobeConnector.jar file for the JVM that you want to instrument because
you cannot use Xbootclasspath directly on the agent.jar file.
Follow these steps:
1. Locate the Java executable that WebSphere Application Server uses, for example,
under AppServer/java/jre/bin.
2. Open a command prompt and enter the following commands:
cd <agent_install_dir>/wily/connectors
<path_to_WAS>/AppServer/java/jre/bin/java jar CreateAutoProbeConnector.jar
current
The AutoprobeConnector.jar is created.
About running ProbeBuilder manually


3. Enter the following command:
-Xbootclasspath/p:<path_to_created_Autoprobe_jar>/AutoprobeConnector.jar:<pat
h_to_agent>/Agent[NoRedef].jar
Dcom.wily.introscope.agentProfile=<path_to_agent>/IntroscopeAgent[NoRedef].p
rofile
-Xbootclasspath/p:
Specifies a colon-separated path of directories, JAR archives, and ZIP archives
to prepend the default bootstrap class path. Overrides entities that are loaded
at bootstrap time by default by the JVM.
Note: On UNIX systems, use a colon (:) in Xbootclasspath, and on Windows
systems, use a semicolon (;).
About running ProbeBuilder manually
Running ProbeBuilder manually is a non-dynamic method of instrumenting applications.
When you run ProbeBuilder manually, it instruments classes on disk before the
application server is run. You use manual ProbeBuilding when your environment does
not support AutoProbe, or you prefer not to use AutoProbe.
Manual ProbeBuilding should not be used with other methods of instrumentation, and
should be used as a last resort.
The instructions for manual ProbeBuilding assume you have performed the following
installation and configuration tasks:
1. Installed the Java agent. See Installing the Java Agent for more information.
2. Configured Java agent connection properties. See Connecting to the Enterprise
Manager (see page 59) for more information.
3. Configured the Java agent name. See Java Agent Naming (see page 121) for more
information.
4. Configured options for ProbeBuilder. See AutoProbe and ProbeBuilding Options
(see page 75) for more information.
Configure AutoProbe for WebSphere for z/OS
Valid for: WebSphere 6.1 and 7.0 for z/OS
You can configure WebSphere on z/OS installations to use AutoProbe to instrument
applications. For more information about AutoProbe, see AutoProbe and ProbeBuilding
Options (see page 75).


Note: Instrumenting WebSphere 7.0 for z/OS using the following procedure does not
provide as detailed metrics as does the JVM 1.5 AutoProbe for WAS 7 on z/OS method.
For example, the thread metric levels are not instrumented.
Important! If you are using the Java Agent 9.0 at a minimum to monitor WebSphere 7.0
on z/OS, the application server process can restart repeatedly. To avoid this problem,
upgrade to WAS 7.0 build level 7.0.0.8 at a minimum.
Follow these steps:
1. In WebSphere, start the Administrator Console.
2. Select Application Servers > <your server> > Process Definition.
Control and Servant items are listed.
3. Click Servant and then JavaVirtualMachine.
4. Set the Generic JVM Argument field to specify the classloader plug-in, and the
location of the IntroscopeAgent.profile file. Set one of the following:
com.wily.introscope.agentProfile
Or
com.wily.introscope.agentResource
The argument has the following value (several properties are set in one argument):
-Dcom.ibm.websphere.classloader.plugin=com.wily.introscope.api.websphere.WASA
utoProbe
-Dcom.wily.introscope.agentProfile=<path to IntroscopeAgent.profile>
Or
-Dcom.ibm.websphere.classloader.plugin=com.wily.introscope.api.websphere.WASA
utoProbe
-Dcom.wily.introscope.agentResource=<path to Resource containing
IntroscopeAgent.profile>
5. Place the Agent.jar file in the <WebSphere Instance dir>/lib/ext directory.
Note: Do not place the Agent.jar file in the WebSphere installation directory.
The following shows examples of the incorrect and correct directory:
Incorrect: /usr/lpp/zWebSphere/V5R0M0/lib/ext
Correct: /WebSphere/V5R0M0/AppServer/lib/ext
6. Confirm that all newly created CA Introscope files and directories within the ./wily
directory are read-accessible by the WebSphere process.
7. Confirm all *.log files have write access to the WebSphere process. The Java Agent
and ProbeBuilder write these files in the ./wily folder. These files include:
All the CA Introscope files and directories
The CA Introscope files inside <WAS instance dir>/lib/ext
8. Restart WebSphere application server.


9. When WebSphere says "open for e-business," open the Administrator Console.
Metrics start reporting.
10. For AutoProbe to run correctly in WebSphere environments with Java2 Security
enabled, add permissions to your Java2 security policy (see page 51).
11. Collect servlet data by configuring HTTP servlet tracing (see page 335).
More information:
AutoProbe and ProbeBuilding Options (see page 75)


Appendix C: Using the PBD Generator

You can use the PBD Generator tool to instrument custom Java class files for use by
agents.
About the CA PBD Generator (see page 343)
Configuring the PBD Generator (see page 344)
Using the PBD Generator (see page 344)
About the CA PBD Generator
The PBD Generator utility can create a PBD file from Javadoc tags with which you have
annotated your Java code, to facilitate the instrumentation of custom Java class files for
use by the Java agent.
The Generator examines a set of Java source files, and instruments the methods in the
classes that contain the Javadoc tag @instrument.
Using the PBD Generator tool, you can:
automate building of PBD files, to eliminate potential for errors that might be
introduced by creating PBD files manually.
integrate PBD generation into your build systems to create and update PBD files
automatically and incorporate any changes to the Java source.
You configure the PBD Generator by integrating it into an Apache Ant target using the
PBDGenerator.jar file, then running it as an Ant Javadoc task.
Configuring the PBD Generator


Configuring the PBD Generator
This tool is intended to be incorporated into Ant-based build systems, as a Javadoc task
in an Ant target.
This sample Javadoc task illustrates the use of this tool in Ant:
<javadoc sourcepath="/src/engineering/products/introscope/source"
destdir="/src/engineering/products/introscope/source/generatedpbd"
maxmemory="512m"
packagenames="com.wily.introscope.console.thornhill.ui.util"
verbose="false"
private="true">
<doclet name="com.wily.util.build.javadoc.PBDInstrumentDoclet"
path="/Wily/tools/WilyPBDGenerator.jar">
<param name="-d"
value="/src/engineering/products/introscope/source/generatedpbd"/>
</doclet>
</javadoc>
Required PBD Generator parameters
These key PBD Generator parameters are required:
sourcepath
the root directory of the Java source tree
destdir
the directory path of the PBD file that will be output from the tool
packagenames
a comma-separated list of the Java packages to be examined for instrumentation
doclet path
the path to find the PBD Generator jar file, which contains this tool
param name="-d"
this must contain the same value as destdir
Using the PBD Generator
Before you can use the PBD Generator, you insert special Javadoc tags into the Java
source files to be instrumented.
Using the PBD Generator


The syntax for the JavaDoc tag is:
@instrument <valid metric prefix> <optional tracer name>
where:
<valid metric prefix> is any valid Introscope metric prefixa string without a colon
character (:). Pipe characters (|) are acceptable.
<optional tracer name> can be BlamePointTracer, FrontendMarker or BackendMarker.
The default is BlamePointTracer if the tracer name is missing.


Appendix D: Using the Network Interface
Utility

You use the Network Interface utility to determine the network interface name value of
the host computer used by the agent for the Catalyst integration.
Determine Network Interface Names (see page 347)
Determine Network Interface Names
The Network Interface utility provides the name and subinterface values for the
introscope.agent.primary.net.interface.name property. Run this utility on the same JVM
and application server used by the agent.
Follow these steps:
1. From the command line, navigate to the following directory:
<Agent_Home>/wily/tools
2. Run this command to invoke the utility:
java -jar NetInterface.jar
Your browser displays the list of network interface names supported by Java on the
Network Interfaces tab.
More information:
Configure a List of Available Networks (see page 239)

Index 349

Index

A
About Automatically Generated Response Files 27
About Introscope and Java Agents 19
About running ProbeBuilder manually 339
About servlet request data 171
About the CA PBD Generator 343
About the Java Agent Directory Structure 30
About the sample response file 27
About User ID data 170
About WebLogic Diagnostic Framework (WLDF) 43
Adding classes to a tracer group 100
Advanced automatic agent naming options 127
Advanced error data capture 154
Advanced naming techniques for URL groups
(optional) 159
Advanced single-metric tracers 111
Agent failover 215
Agent Heap Sizing 169
Agent HTTP tunneling 217
Agent HTTP tunneling for proxy servers 217
Agent HTTPS tunneling 219
Agent initialization 115
Agent log files and automatic agent naming 138
Agent memory overhead 220
Agent metric aging 221
Agent metric clamp 225
Agent naming 226
Agent naming considerations for clustered
applications 124
Agent recording (business recording) 231
Agent Rename Check Interval 128
Agent thread priority 232
Agent to Enterprise Manager connection 233
Alternative Methods for Instrumentation 331
Application servers that support agent naming 125
Application triage map 235
238
Application triage map and the agent name 130
Application triage map business transaction POST
parameters 240
Application triage map managed socket
configuration 242
Applying ProbeBuilder Directives 103
Automatic agent naming 126
Automatic agent naming and renamed agents 127
AutoProbe 245
AutoProbe and ProbeBuilding Options 75
AutoProbe and ProbeBuilding Overview 75
AutoProbe log name and location 140
autoprobe.dynamicinstrument.pollIntervalMinutes
260
Average tracer example 110
B
Backwards Compatibility 135
Before You Install the Agent 23
Blame Tracers 117
Blame Tracers in standard PBDs 119
Bootstrap Classes Instrumentation Manager 247
Boundary Blame and Oracle Backends 119
C
CA CEM Agent Profile Properties 248
CA Technologies Product References 3
Change the Component Name in the Application
Triage Map 134
ChangeDetector configuration 254
Changing the name or location of the agent logfile
138
Channels 265
Cloned agent naming scenario 129
Collect Input and Output Bandwidth Metrics 136
Collect Socket Metrics 135
Collecting servlet request data 172
Collecting user id data 172
Combined counter tracers example 111
Combining custom tracers 115
Command-line ProbeBuilder and ProbeBuilder
Wizard log name and location 140
Command-line property overrides 214
Command-line SQL statement normalizer 184
Commonly used tracer names and examples 109
Components traced by the default PBDs 86
Configure a Custom Service in WebSphere 51
Configure a List of Available Networks 239
Configure a proxy server for HTTP tunneling 61
Configure Agent Load Balancing 63
Configure Apache Tomcat to use the Java agent 36


Configure AutoProbe for WebSphere for z/OS 339
Configure CA LISA Tracers 204
Configure Dynamic ProbeBuilding 79
Configure GlassFish to Use the Java Agent 57
Configure IBM WebSphere to Use the Java Agent
46
Configure JBoss 7 for Autonaming 40
Configure JBoss to Use the Java Agent 37
Configure Oracle Application Server to use the Java
agent 57
Configure Oracle to Use AutoProbe 333
Configure Oracle WebLogic to Use the Java Agent
40
Configure periodic polling for uninstrumented
subclasses 83
Configure Permissions to Access Platform Monitors
on HP-UX 197
Configure SAP Netweaver to Use the Java Agent 58
Configure Sun ONE to Use AutoProbe 332
Configure the Ability to Send Information 238
Configure the Reporting of Resource Metric Map
Data for WebSphere 55
Configure the reporting of WebSphere PMI metrics
in Introscope 54
Configure the response file properties and install the
agent 27
Configure the Session ID Collection 253
Configure WebLogic Server 334
Configure WebLogic with JRockit JVM to Use the
Java Agent 45
Configure WebLogic with JRockit JVM to View Socket
Metrics 45
Configure WebSphere and WebLogic to Use JMX
Reporting 191
Configure WebSphere Application Server 6.1 on
UNIX, Windows, OS/400, z/OS, IBM JVM 1.5 47
UNIX, Windows, Sun, HP, Other JVM 1.5 49
UNIX, Windows, OS/400, JVM 1.5 49
Configure WebSphere Application Server 7.0 on z/OS
- JVM 1.5 50
Configuring agent connection metrics 131
Configuring agent metric aging 222
Configuring Agent Properties 67
Configuring Agent to collect additional transaction
trace data 171
Configuring Boundary Blame 157
Configuring Component Stall Reporting 173
Configuring Detection of Synthetic Transactions 32
Configuring ErrorDetector options 153
Configuring HTTP servlet tracing 335
Configuring LeakHunter and ErrorDetector 141
Configuring LeakHunter properties 144
Configuring logging options 136
Configuring Platform Monitoring 195
Configuring ProbeBuilding 76
Configuring the Agent to Use Legacy Mode
Transaction Tracing 166
Configuring the Application Server to Start the Java
Agent 36
Configuring the connection to the Enterprise
Manager 59
Configuring the Introscope SQL Agent 175
Configuring the IntroscopeAgent.profile location
213
Configuring the PBD Generator 344
Configuring Transaction Trace Options 165
Configuring unique names for application instances
129
Connect to the Enterprise Manager over SSL 62
Connect to the Enterprise Manager Using a Direct
Socket Connection 59
Connect to the Enterprise Manager with HTTP
tunneling 60
Connect to the Enterprise Manager with HTTPS
tunneling 61
Contact CA Technologies 5
168
Controlling directive logging 84
Counter tracer example 111
Counting object instances 114
Create a startup class for WebLogic 41
Create an AutoProbe Connector File 335
Creating custom tracers 106
Cross-process tracing in WebLogic Server 257
Cross-process transaction trace 258
Cross-Process Transaction Tracing 170
Custom FrontendMarker Directive 118
Custom SQL statement normalizer 179
Customize Data Collection 37
D
Default JMX Metric Conversion Process 188
Default PBD Files 87
Default PBD Files From Previous Releases 89

Index 351

Default PBL files 90
Default SQL statement normalizer 178
Default Tracer Groups and Toggles Files 90
Define a Baseline Agent Profile with Appropriate
Configuration Properties 21
Define the name for a URL group 159
Defining keys for URL groups 158
Defining membership of each URL group 158
Defining new error types 154
Deploying the Java Agent 22
Determine Configuration Requirements 21
Determine Network Interface Names 347
Directive names and arguments used in tracer syntax
107
Disable directive updates 83
Disable Platform Monitors 197
Disabling the capture of stalls as Events 174
Download and Install the CA APM Cloud Monitor
Agent 208
Downstream Subscriber Component Stalls 173
Dynamic instrumentation 259
Dynamic Instrumentation Impacts Performance for
IBM JDK 80
Dynamic ProbeBuilding 77
Dynamic ProbeBuilding Versus Dynamic
Instrumentation 80
E
EJB 3.0 annotations 101
EJB Naming 102
EJB subclass tracing 100
EJB Support for the Application Triage Map 101
Eliminate Startup Timing Issues With Logging
Facilities 56
Enable cross-process tracing in WebLogic Server 41
Enable cross-process tracing in WebSphere 55
Enable instrumentation of multiple levels of
subclasses 81
192
Enable Platform Monitors on AIX 196
Enable the Collection of WebSphere PMI Metrics
53
Enable WLDF reporting 45
Enabling and disabling LeakHunter 144
Enabling cloned agent naming in clustered
environments 129
Enabling ErrorDetector in the Java Agent 152
Enabling JMX Reporting 187
196
ErrorDetector 151, 261
Evaluate Agent Performance Overhead 22
Example
Comments in SQL statements 177
Creating URL groups by application path 159
Mapping a group key to a URL path prefix 159
Statements that generate lists of values or insert
values 178
Temporary tables or automatically generated
table names 177
Use Xbootclasspath to Instrument WAS 338
Example 1 182
Example 2 183
Example 3 183
Example of Servlet and JSP Distribution Metrics 74
Example of Servlet and JSP Distribution Metrics for
Individual and Summary Level 73
Examples of Distribution Statistics Metrics 73
ExceptionErrorReporter 154
Exceptions 115, 180
Extending transaction trace data collection 170
Extensions 263
F
Fine-tuning socket and SSL metric collection 133
Full or typical tracing options 76
G
GC Monitor 264
H
High agent CPU overhead from deep nested
frontend transactions 118
How ErrorDetector works 152
How Introscope resolves agent naming conflicts
123
How LeakHunter works 142
How poorly written SQL statements create metric
explosions 176
How the Agent Determines its Name 122
How to Configure an Agent to Collect Distribution
Statistic Metrics 72
How to configure backup Enterprise Managers and
failover properties 67
How to enable and configure thread dumps 69


How to enable and use additional GC metrics 69
How to instrument applications 35
How to Integrate CA APM Cloud Monitor In Your CA
APM Deployment 208
How to Integrate CA APM with CA LISA 199
How to Limit Data 210
How to modify communication with Enterprise
Manager 67
How to Set Up Resource Metrics in Weblogic 43
How WLDF data is converted into Introscope metrics
44
HTTPErrorCodeReporter 156
I
150
Identified potential leak stops leaking log entry 149
Identifying potential leaks with collection IDs 147
Ignoring collections that cause poor performance
146
Initial Enterprise Manager Connection Delay 128
Install and Evaluate the Default Functionality 20
Install CA LISA 200
Install Manually Using Installation Archives 30
Install the Java Agent Interactively 24
Install the Java Agent Silently 26
Installing and Configuring the Java Agent 23
Instrument CA APM to View Test Event Metrics and
Data from CA LISA 201
Instrumenting and Inheritance 115
Instrumenting with new and changed PBDs 104
Integrating CA APM Cloud Monitor with CA APM
207
Integrating CA APM with CA LISA 199
Introduction to the Java Agent 19
Introscope Support for WebLogic JMX Metrics 188
introscope.agent.agentAutoNamingEnabled 227
introscope.agent.agentAutoNamingMaximumConne
ctionDelayInSeconds 227
introscope.agent.agentAutoRenamingIntervalInMinu
tes 228
introscope.agent.agentName 228
229
introscope.agent.appmap.enabled 236
introscope.agent.appmap.intermediateNodes.enabl
ed 238
introscope.agent.appmap.metrics.enabled 236
introscope.agent.appmap.queue.period 237
introscope.agent.appmap.queue.size 237
introscope.agent.bizdef.matchPost 240
introscope.agent.bizdef.turnOff.nonIdentifying.txn
307
introscope.agent.bizRecording.enabled 232
introscope.agent.cemtracer.domainconfigfile 251
introscope.agent.cemtracer.domainconfigfile.reloadf
requencyinminutes 252
introscope.agent.clonedAgent 230
introscope.agent.common.directory 263
introscope.agent.crossprocess.compression 312
introscope.agent.crossprocess.compression.minlimit
313
introscope.agent.crossprocess.correlationid.maxlimit
314
introscope.agent.customProcessName 230
introscope.agent.decorator.enabled 250, 291
introscope.agent.decorator.security 251, 290
introscope.agent.defaultProcessName 231
introscope.agent.disableLogFileAutoNaming 229
introscope.agent.display.hostName.as.fqdn 231
introscope.agent.distribution.statistics.components.
pattern 253
introscope.agent.enterprisemanager.connectionord
er 215
introscope.agent.enterprisemanager.failbackRetryIn
tervalInSeconds 216
introscope.agent.enterprisemanager.transport.http.
proxy.host 218
proxy.password 219
proxy.port 218
proxy.username 218
introscope.agent.enterprisemanager.transport.tcp.ci
phersuites.DEFAULT 303
introscope.agent.enterprisemanager.transport.tcp.h
ost.DEFAULT 219, 233, 300
introscope.agent.enterprisemanager.transport.tcp.k
eypassword.DEFAULT 303
introscope.agent.enterprisemanager.transport.tcp.k
eystore.DEFAULT 302
introscope.agent.enterprisemanager.transport.tcp.lo
cal.ipaddress.DEFAULT 235
introscope.agent.enterprisemanager.transport.tcp.lo
cal.port.DEFAULT 235

Index 353

introscope.agent.enterprisemanager.transport.tcp.p
ort.DEFAULT 220, 234, 301
introscope.agent.enterprisemanager.transport.tcp.s
ocketfactory.DEFAULT 220, 234, 301
introscope.agent.enterprisemanager.transport.tcp.tr
ustpassword.DEFAULT 302
introscope.agent.enterprisemanager.transport.tcp.tr
uststore.DEFAULT 302
introscope.agent.errorsnapshots.enable 262
introscope.agent.errorsnapshots.ignore.<index>
262
introscope.agent.errorsnapshots.throttle 262
introscope.agent.extensions.directory 263
introscope.agent.gcmonitor.enable 264
introscope.agent.io.socket.client.hosts 292
introscope.agent.io.socket.client.ports 293
introscope.agent.io.socket.server.ports 293
introscope.agent.jmx.enable 270
introscope.agent.jmx.excludeStringMetrics 274
introscope.agent.jmx.ignore.attributes 271
introscope.agent.jmx.name.filter 271
introscope.agent.jmx.name.jsr77.disable 272
introscope.agent.jmx.name.primarykeys 273
introscope.agent.leakhunter.collectAllocationStackTr
aces 275
introscope.agent.leakhunter.enable 276
introscope.agent.leakhunter.ignore.<number> 278
introscope.agent.leakhunter.leakSensitivity 276
introscope.agent.leakhunter.logfile.append 277
introscope.agent.leakhunter.logfile.location 277
278
introscope.agent.metricAging.dataChunk 224
introscope.agent.metricAging.heartbeatInterval
223
introscope.agent.metricAging.metricExclude.ignore.
0 225
introscope.agent.metricAging.numberTimeslices
224
introscope.agent.metricAging.turnOn 223
introscope.agent.metricClamp 225
introscope.agent.nio.datagram.client.hosts 266
introscope.agent.nio.datagram.client.ports 267
introscope.agent.nio.datagram.server.ports 268
introscope.agent.nio.socket.client.hosts 268
introscope.agent.nio.socket.client.ports 269
introscope.agent.nio.socket.server.ports 269
introscope.agent.platform.monitor.system 289
introscope.agent.pmi.enable 320
introscope.agent.pmi.enable.alarmManagerModule
320
introscope.agent.pmi.enable.beanModule 321
introscope.agent.pmi.enable.cacheModule 321
introscope.agent.pmi.enable.connectionPoolModule
322
introscope.agent.pmi.enable.hamanagerModule
322
introscope.agent.pmi.enable.j2cModule 323
introscope.agent.pmi.enable.jvmpiModule 323
introscope.agent.pmi.enable.jvmRuntimeModule
324
introscope.agent.pmi.enable.objectPoolModule
324
introscope.agent.pmi.enable.orbPerfModule 325
introscope.agent.pmi.enable.schedulerModule 325
introscope.agent.pmi.enable.servletSessionsModule
326
introscope.agent.pmi.enable.systemModule 326
introscope.agent.pmi.enable.threadPoolModule
327
introscope.agent.pmi.enable.transactionModule
327
introscope.agent.pmi.enable.webAppModule 328
introscope.agent.pmi.enable.webServicesModule
328
introscope.agent.pmi.enable.wlmModule 329
introscope.agent.pmi.enable.wsgwModule 329
introscope.agent.pmi.filter.objrefModule 330
introscope.agent.reduceAgentMemoryOverhead
221
introscope.agent.remoteagentconfiguration.allowed
Files 249, 290
250, 289
introscope.agent.remoteagentdynamicinstrumentati
on.enabled 261
introscope.agent.sockets.managed.reportClassAppE
dge 243
introscope.agent.sockets.managed.reportClassBTEdg
e 244
introscope.agent.sockets.managed.reportMethodAp
pEdge 244
introscope.agent.sockets.managed.reportMethodBT
Edge 245
introscope.agent.sockets.managed.reportToAppmap
243
introscope.agent.sockets.reportRateMetrics 292


294
introscope.agent.sqlagent.normalizer.regex.key1.cas
eSensitive 298
introscope.agent.sqlagent.normalizer.regex.key1.pat
tern 296
introscope.agent.sqlagent.normalizer.regex.key1.rep
laceAll 297
introscope.agent.sqlagent.normalizer.regex.key1.rep
laceFormat 297
introscope.agent.sqlagent.normalizer.regex.keys
296
introscope.agent.sqlagent.normalizer.regex.matchFa
llThrough 295
introscope.agent.sqlagent.sql.artonly 298
introscope.agent.sqlagent.sql.rawsql 299
introscope.agent.sqlagent.sql.turnoffmetrics 299
introscope.agent.sqlagent.sql.turnofftrace 299
introscope.agent.stalls.resolutionseconds 304
introscope.agent.stalls.thresholdseconds 303
introscope.agent.thread.all.priority 233
introscope.agent.threaddump.deadlockpoller.enable
305
introscope.agent.threaddump.deadlockpollerinterva
l 306
introscope.agent.threaddump.enable 305
introscope.agent.threaddump.MaxStackElements
306
introscope.agent.transactiontrace.componentCount
Clamp 311
introscope.agent.transactiontrace.headFilterClamp
316
introscope.agent.transactiontracer.parameter.httpre
quest.headers 308
introscope.agent.transactiontracer.parameter.httpre
quest.parameters 308
introscope.agent.transactiontracer.parameter.https
ession.attributes 309
introscope.agent.transactiontracer.sampling.enable
d 314
introscope.agent.transactiontracer.sampling.interval
.seconds 315
introscope.agent.transactiontracer.sampling.perinte
rval.count 315
introscope.agent.transactiontracer.tailfilterPropagat
e.enable 258
introscope.agent.transactiontracer.userid.key 309
introscope.agent.transactiontracer.userid.method
310
introscope.agent.ttClamp 317
introscope.agent.urlgroup.group.default.format
318
introscope.agent.urlgroup.group.default.pathprefix
318
introscope.agent.urlgroup.keys 318
introscope.agent.weblogic.crossjvm 258
introscope.agent.wldf.enable 330
introscope.autoprobe.directivesFile 245, 248
introscope.autoprobe.dynamic.limitRedefinedClasse
sPerBatchTo 260
introscope.autoprobe.dynamicinstrument.classFileSi
zeLimitInMegs 260
introscope.autoprobe.dynamicinstrument.enabled
259
introscope.autoprobe.dynamicinstrument.pollInterv
alMinutes 261
introscope.autoprobe.enable 246
introscope.autoprobe.hierarchysupport.disableDirec
tivesChange 288
introscope.autoprobe.hierarchysupport.disableLoggi
ng 288
introscope.autoprobe.hierarchysupport.enabled
286
introscope.autoprobe.hierarchysupport.executionCo
unt 287
introscope.autoprobe.hierarchysupport.pollInterval
Minutes 287
introscope.autoprobe.hierarchysupport.runOnceOnl
y 286
introscope.autoprobe.logfile 246
introscope.bootstrapClassesManager.enabled 247
introscope.bootstrapClassesManager.waitAtStartup
247
introscope.changeDetector.agentID 254
introscope.changeDetector.compressEntries.batchSi
ze 257
introscope.changeDetector.compressEntries.enable
257
introscope.changeDetector.enable 254
introscope.changeDetector.isengardStartupWaitTim
eInSec 255
introscope.changeDetector.profile 256
introscope.changeDetector.profileDir 256
introscope.changeDetector.rootDir 255
introscope.changeDetector.waitTimeBetweenRecon
nectInSec 255
introscope.ext.agent.metric.count 285

Index 355

J
Java 7 Autoprobe 35
Java Agent Deployment on Other Application Servers
331
Java Agent JMX Support 187
Java Agent Monitoring and Logging 131
Java Agent Naming 121
Java Agent Properties 213
Java Agent Support for JMX Metrics 42
Java Annotations 116
Java NIO 264
JBoss Application Server Logging Considerations 39
JBoss-specific PBDs and PBLs 40
JMX 270
JMX filters for WebLogic 191
JVM 1.5 systems using JVM AutoProbe via -javaagent
104
K
Keyword-based substitution
Example 1 112
Example 2 113
Known limitations 241
L
LeakHunter 141, 274
LeakHunter log file 148
LeakHunter timeout log entry 150
log4j.additivity.IntroscopeAgent.inheritance 284
log4j.appender.logfile.File 281
log4j.appender.pbdlog 282
log4j.appender.pbdlog.File 282
log4j.appender.pbdlog.layout 283
log4j.appender.pbdlog.layout.ConversionPattern
283
log4j.logger.IntroscopeAgent 280
log4j.logger.IntroscopeAgent.inheritance 281
Logging 279
Logging considerations on WebSphere for z/OS 56
M
Managing Metric Volume with JMX Filters 190
Managing ProbeBuilder Logs 140
Manually Download and Extract CA LISA Files 201
Method 1
Agent name specified in a Java system property
122
Method 2
Agent name specified in a system property key in
the IntroscopeAgent.profile 122
Method 3
Agent name obtained automatically from the
application server 123
Method 4
Agent name specified explicitly in the agent
profile 123
Method 5
Agent name determined to be 123
MethodCalledErrorReporter 155
Metric count 284
Metric name keyword-based substitution 112
Metric-name-based example 114
Metric-name-based parameters 113
Modifying Java2 Security Policy 51
Multiple inheritance 285
N
New and Changed ProbeBuilder Files 105
New Mode of Transaction Tracing 165
NIODatagramTracing metrics 265
Null or empty strings 180
O
Obtaining an agent name from the application server
125
P
Per interval counter tracer example 111
Planning a Java Agent Deployment 20
Platform monitoring 288
Platform Monitors 195
Potential leak first identified log entry 148
ProbeBuilder Directives 85
ProbeBuilder Directives Overview 85
ProbeBuilding Class Hierarchies 81
R
Rate tracer example 111
Redirecting agent output to a file 137
Regular expression SQL statement normalizer 180
Regular expression SQL statement normalizer
examples 182
Remote configuration 289
Removing line numbers in bytecode 84
Required PBD Generator parameters 344


Restricting Java NIO metrics 266
Restricting socket and SSL metric collection 132
Rolling up logs by date or size 139
Running LeakHunter 147
Running the agent in verbose mode 136
Running the AutoProbe Connector for a JVM 336
Running the URLGrouper 163
S
SAP Netweaver 198
Security 290
Select the method for installing the Java agent 24
Servlet header decorator 291
Setting toggles to gather additional metric
information 99
Signature differentiation 112
Skip directives 114
Socket metrics 132, 291
Specifying an agent name using a Java system
property 124
Specifying an agent name using a system property
key 125
SQL Agent 293
SQL metrics 184
SQL statement normalization 176
SQL statement normalization options 178
SSL communication 300
SSL, NIO, and Socket Tracing in the Application
Triage Map 133
Stall metrics 303
Support for Multiple Inheritance, Interfaces, and
Abstract Methods 82
System and version requirements 143
T
Tagging log output as EBCDIC 56
The SQL Agent files 176
The SQL Agent overview 175
ThisErrorReporter 155
Thread dumps 304
Transaction Trace component clamp 168
Transaction trace sampling 169
Transaction tracing 307
Troubleshoot Platform Monitoring 197
Troubleshooting platform monitoring on Windows
198
Turn Off Statement Metrics 184
Turning Off Agent Log File Automatic Naming 128
Turning off socket and SSL metric collection 135
Turning on InstrumentPoint directives 115
Turning tracer groups on or off 99
Types of errors 151
U
Understanding Boundary Blame 157
Understanding the Java Agent name 121
Uninstalling the Java agent 64
Uninstalling the Java agent from z/OS 65
Unsupported Instrumentation Method 76
Upgrade Multiple Agent Types 63
Upstream Inherited Component Stalls 174
URL grouping 317
Using a custom BlamePointTracer tracer for common
metrics 106
Using a delimited portion of the request path as a
URL group name 162
Using a substring of the request path as a URL group
name 161
Using Blame tracers 163
Using Blame Tracers to mark blame points 117
Using error tracer directives with caution 156
Using ErrorDetector 156
Using host as a URL group name 160
Using JVM AutoProbe 103
Using LeakHunter 150
Using multiple naming methods for URL groups
162
Using parameter as a URL group name 161
Using port as a URL group name 160
Using Primary Key Conversion to Streamline JMX
Metrics 189
Using protocol as a URL group name 160
Using quotes in custom tracer definitions 110
Using the command-line ProbeBuilder 105
Using the Installer to Install CA LISA 203
Using the IntroscopeAgent.profile, PBLs, and PBDs
together 103
Using the Network Interface Utility 347
Using the PBD Generator 343, 344
Using the ProbeBuilder Wizard 105
Using the ProbeBuilder Wizard or command-line
ProbeBuilder 104
Using the TagScript Utility 34
Using URL groups 157

Index 357

V
Validate and Deploy the Agent Configuration 22
Validate the CA APM Cloud Monitor Agent
Connection 209
Verify the CA APM and CA LISA Integration 204
W
WebSphere PMI 319
What LeakHunter does not track 143
What LeakHunter tracks in Java 142
WLDF metrics 330

APM - 9.5 - Java Agent Implementation Guide

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

APM - 9.5 - Java Agent Implementation Guide

Hochgeladen von

Copyright:

Verfügbare Formate

Java Agent Implementation Guide

Das könnte Ihnen auch gefallen