Welcome to:

Troubleshooting
Unit Objectives
After completing this unit, you should be able to:
•Discuss problem determination issues
•Identify resources for problem determination
–Log files
–Trace output
–Messages
•Describe tools for troubleshooting
–Tracing
–Log Analyzer
–JNDI Namespace Dump Tool
–Collector Tool
–First Failure Data Capture tool
–Application Server Toolkit
Problem Determination – Divide and Conquer
Check activity.log,
SystemOut.log,
Client SystemErr.log

Application Server
Use browser to access
Ping the IP http://localhost (if local)
address, ping the or http://<ip_address>
cluster address Web EJB
(if remote)
Container Container

Use browser to Use Java client to

Load Balancer IBM HTTP Server access port 9080 access EJB

Web Server Plug-

in

Check Use database

http_plugin.log file client with SQL to Database
check tables
Problem Determination
•Navigate through the problem:
–What kind of problem do I have?
–What component is causing the problem?
•Resources for identifying problems.
•WebSphere Application Server topology
Resources for Identifying a Problem
•Console messages provide configuration problem
information, starting server and loading of module information
•Java exceptions provide stack trace.
•Log files provide information about servers as they initialize
and run.
•Log Analyzer diagnoses problems from event history and
symptom database.
•Traces provide detailed information about WebSphere
Application Server components.
•JNDI Dump Name Space displays global names for objects
and references.
Console Messages (1 of 4)
•Console messages appear in the
WebSphere Status pane
–Displays
•Configuration Problem
messages
•Runtime messages
–Available on every
page of the Administrative
Console.
–Automatically
refreshes every
60 seconds
(configurable)
Console Messages (2 of 4)
•Runtime events grouped by severity
–Error
–Warning
–Information
•Clicking on
error link
shows details
Console Messages (3 of 4)
•Runtime event details
Console Messages (4 of 4)
•Configuration problems
–Changing level of validation will show more or less severe
problems
Log Files in logs Directory
•SystemOut.log - standard JVM output log
–Indicates if code running in the server started and stopped
successfully
–Can be used for application messages
–Destination and name are configurable
•SystemErr.log - standard JVM error log
–Exceptions thrown by code running in the server
–Destination and name are configurable
•startServer.log and stopServer.log
–startup and shutdown of the Application Servers
•activity.log - events that show a history of activities
–use Log Analyzer to read output from this file
•http_plugin.log - plug-in log
–Errors and trace data from the Web server plug-in
•trace.log – output from diagnostic trace
–Destination and name are configurable
Log Formatting

•The JVM log and the Trace

logs can be formatted in:
– Basic format
– Advanced format
•In addition the Trace log can
also be produced in activity log
format. In this format the Log
Analyzer tool can be used to
view.
•Formatting controlled through
Administrative Console
Log and Trace File Format
[31/01/03 9:51:15:081 EST] 3c07adad SystemOut O Looking up Account

Thread Component
Timestamp Msg type Message
ID

Entry to a method (debug)

< Exit a method (debug)
A Audit
W Warning
X Error
E Event (debug)
D Debug (debug)
T Terminate (exits process)
F Fatal (exits process)
I Information
O Program output
Log Analyzer Tool
•The Log Analyzer looks at error conditions in the log entries
to diagnose problems and suggest possible solutions.
•Update your Symptom Database by downloading from IBM.

IBM
Support

Symptom
Activity Log
Database
Log Analyzer
XML file

Problem
Diagnosis
Log Analyzer (1 of 2)

Record Pane

{
(Display Detail)

MaxRetries

{
set to 3 in
admin.config.
Error repeats
three times

{
Log Pane
(Navigation)
Analysis Pane
Log Analyzer (2 of 2)
•Log Pane - Navigation
–Raw logging data is formatted into Units of Work (UoW)
–UoWs contain individual lines detailing activity
•Record Pane - Details
–Process, Thread and Source IDs
–Function name
–Extended Message
•Analysis Pane - More (external) information
–Looks up (on command) symptoms in database:
•<AS_root>/properties/logbr/symptoms/adv/symptomdb.xml
Traces
•Trace can be started
–While server is running using Runtime Diagnostic Trace
–When server is started using Configuration Diagnostic
Trace
•Trace output can be directed to:
–Memory ring buffer - Dumped after trace stops
–File
•Trace has a significant impact on performance
–Enable temporarily for problem determination
–Trace to file is slower than trace to memory ring buffer

Java Ring
Trace Dump Dump file
components buffer
Trace a Running Application Server

Current Trace String

Click to modify
trace
specifications

Select Trace to
Memory Buffer or File

View Trace File on Console

Trace Application Server from Startup
Trace Administration

! Select Groups or
Components.
! Click, select trace type.

Gray None
Blue Entry/Exit
Yellow Event
Red Debug

! Click Apply.
! Click Close.
Components to Trace
•A different way to select what
components to trace
•WebSphere used packages
are instrumented for tracing
•User created applications can
be instrumented too, and be
included in the trace output.
Reading a Trace File
•Timestamps give good clues:
–Time stamps are real machine time values
–Good when comparing traces from different processes.
•Look for exceptions (search for "exception" from top)
–Events prior to exception are probable causes.
–Events after exception are recovery attempts.
•Often useful to follow a single thread.
Dumping the JNDI Name Space
•DumpNameSpace utility shows JNDI directory content.
•Useful to ensure correct association of named objects:
–Data sources
–EJBs
–JMS resources
–Other resources
•Syntax and some of the options:
<as_root>\bin\DumpNameSpace
[-host bootstrap_host_name (defaults to localhost)]
[-port bootstrap_port_number (defaults to 2809)]
[-startAt subcontext/in/the/tree]

•Output can be redirected to a file and inspected.

Dumping the JNDI Name Space - Example

==============================================================================
Beginning of Name Space Dump
==============================================================================

1 (top)
2 (top)/domain javax.naming.Context
2 Linked to context: was5host
3 (top)/cells javax.naming.Context
4 (top)/clusters javax.naming.Context
5 (top)/cellname java.lang.String
6 (top)/cell javax.naming.Context
6 Linked to context: was5host
7 (top)/nodes javax.naming.Context
8 (top)/nodes/was5host javax.naming.Context
9 (top)/nodes/was5host/persistent javax.naming.Context
10 (top)/nodes/was5host/nodename java.lang.String
11 (top)/nodes/was5host/domain javax.naming.Context
11 Linked to context: was5host
12 (top)/nodes/was5host/cell javax.naming.Context
12 Linked to context: was5host
13 (top)/nodes/was5host/servers javax.naming.Context
14 (top)/nodes/was5host/servers/server1 javax.naming.Context
15 (top)/nodes/was5host/servers/server1/jms javax.naming.Context
16 (top)/nodes/was5host/servers/server1/jms/StockQueue
.
.
.
Application Server Toolkit
•Based on Eclipse tooling
•Similar to WebSphere Studio Application Developer, limited
function
–Remote Debugger
–Log Analyzer
–Profiler
–Java perspectives
•Installed from separate CD
First Failure Data Capture tool (FFDC)
•Preserves the information generated from a processing
failure
•Captured data is saved in log files for use in analysis
•This tool is meant to be used by IBM support
•Capturing FFDC data does not affect performance
•FFDC data is collected on the <as_root>\logs\ffdc directory
•There are no administrative tasks to manage FFDC
Collector Tool
•Gathers information about the WebSphere Application Server
installation
•Packages information in JAR file
•JAR file is sent to IBM Support
•Run collector from the <as_root>\bin directory
•IBM Support will ask you to run it to collect information to
solve a problem.
Information Resources
•WebSphere V5 Systems Management and Configuration (Redbook)
– http://www.redbooks.ibm.com/redpieces/pdfs/sg246195.pdf
•WebSphere Application Server Library
– http://www.ibm.com/software/webservers/appserv/library.html
•WebSphere InfoCenter
– http://www.ibm.com/software/info/websphere/library/index.html
•WebSphere Developer Domain
– http://www7b.boulder.ibm.com/wsdd/
•WebSphere Administration: Lessons from the Experts
– http://www7b.boulder.ibm.com/wsdd/library/presents/wsexperts.html
•WebSphere Advisor Magazine
– http://Advisor.com/www/WebSphereAdvisor
•WebSphere Application Server Support
– http://www.ibm.com/software/webservers/appserv/support.html
Problem Determination Demonstration

Problem Determination
Demonstration
Unit Summary
•Discussed problem determination issues.
•Described resources and tools for problem determination.
•Described how to view messages in the Administrative
Console.
•Described how to enable tracing for WebSphere components.
•Described how to use the Log Analyzer to view events in the
activity.log file.
•Described how to dump the JNDI namespace.
•Touched on other problem determination tools (FFDC,
Collector)
Additional Material
HTTP Plug-in Log
•The plug-in runs in the Web server process and routes
requests from the Web server to the application server.
•The plug-in log can be found in:
<as_root>\logs\http_plugin.log
•Location of log and LogLevel is configured in file:
<as_root>\config\cells\plugin-cfg.xml

in element:

<Log LogLevel="Error"
Name=“C:\Program Files\WebSphere\AppServer\logs\http_plugin.log"
/>
•Level of detail (LogLevel) of the log messages :
–Trace
–Warn
–Error
HTTP Server Logging
•HTTP Server provides logging of errors and HTTP requests
–Configured via logging directives in httpd.conf file
–Error and access logging configured by default
ErrorLog logs/error.log
LogLevel warn
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
CustomLog logs/access.log common
•Change log level for problem determination
•Change log format if analyzing access logs
•Configure separate logs for virtual hosts
•Fast Response Cache Accelerator (FRCA) logging
•Enable logging by adding directives to httpd.conf file
AfpaLogging on
AfpaLogFile "C:/Program Files/IBMHTTPServer/logs/afpalog" V-ECLF
HTTP Server SNMP Support
•HTTP Server includes a Simple Network Management
Protocol agent
•Provided as a plug-in module
•Supports the following Management Information Bases (MIB)
•SNMP MIB
•WWW MIB
•Apache MIB
•Configured via SNMP directives in httpd.conf file
•SNMP module is not configured by default
LoadModule snmp_agt_module modules/IBMModuleSNMP.dll
SNMPenable
<SNMP 161>
SNMPcommunity public
sysContact myadmins@myco.com
sysLocation 3rd floor computer room, London office
</SNMP>