D83175GC10 Ag PDF

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY.
COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
D85190
Edition 1.0
January 2014
D83175GC10
Activity Guide
Troubleshooting Workshop
Oracle WebLogic Server 12c:
Oracle University and In Motion Servicios S.A. use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
Disclaimer
This document contains proprietary information and is protected by copyright and other intellectual property laws. You may copy and
print this document solely for your own use in an Oracle training course. The document may not be modified or altered in any way.
Except where your use constitutes "fair use" under copyright law, you may not use, share, download, upload, copy, print, display,
perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express authorization
of Oracle.
The information contained in this document is subject to change without notice. If you find any problems in the document, please
report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not
warranted to be error-free.
Restricted Rights Notice

If this documentation is delivered to the United States Government or anyone using the documentation on behalf of the United
States Government, the following notice is applicable:
U.S. GOVERNMENT RIGHTS

The U.S. Governments rights to use, modify, reproduce, release, perform, display, or disclose these training materials are restricted
by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract.
Trademark Notice
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective
owners.
Author
Bill Bell
Technical Contributors and Reviewers

Jose Alvarez, Maria Billings, Mark Lindros, Takyiu Liu, Will Lyons, Serge Moiseev,
TJ Palazzolo, Madhavi Siddireddy, Matthew Slingsby, Kevin Tate
Editors
Raj Kumar, Anwesha Ray
Graphic Designer
Rajiv Chandrabhanu
Publishers
Nita Brozowski,Sujatha Nagendra, Michael Sebastian
This book was published using: Oracle Tutor

Table of Contents
Practices for Lesson 1: Course Overview .....................................................................................................1-1
Practices for Lesson 1: Overview ...................................................................................................................1-2
Practices for Lesson 2: Troubleshooting: Overview ....................................................................................2-1
Practice 2-1: Accessing and Setting Up the Practice Environment ................................................................2-3
Practice Solution: Accessing and Setting Up the Practice Environment ........................................................2-7
Practice 2-2: <OPTIONAL> Running RDA .....................................................................................................2-8
Practice Solution: <OPTIONAL> Running RDA .............................................................................................2-13

Practices for Lesson 3: WebLogic Server Diagnostic Framework: Overview ............................................3-1
Practice 3-1: Harvesting Diagnostic Metrics ...................................................................................................3-3
Practice Solution: Harvesting Diagnostic Metrics ...........................................................................................3-10
Practice 3-2: Monitoring Diagnostic Metrics ...................................................................................................3-11
Practice Solution: Monitoring Diagnostic Metrics ...........................................................................................3-13
Practices for Lesson 4: Diagnostic Instrumentation ....................................................................................4-1
Practice 4-1: Configuring and Monitoring Diagnostic Events .........................................................................4-3
Practice Solution: Configuring and Monitoring Diagnostic Events ..................................................................4-8
Practice 4-2: Tracing a Client Request ...........................................................................................................4-9
Practice Solution: Tracing a Client Request ...................................................................................................4-13
Practices for Lesson 5: JVM Troubleshooting ..............................................................................................5-1
Practice 5-1: Troubleshooting a JVM .............................................................................................................5-3
Practice Solution: Troubleshooting a JVM .....................................................................................................5-10
Practices for Lesson 6: Troubleshooting Servers ........................................................................................6-1
Practice 6-1: Investigating Server Problems ..................................................................................................6-3
Practice Solution: Investigating Server Problems...........................................................................................6-10
Practices for Lesson 7: Troubleshooting Applications ................................................................................7-1
Practice 7-1: Investigating Application Problems ...........................................................................................7-3
Practice Solution: Investigating Application Problems....................................................................................7-7
Practices for Lesson 8: Troubleshooting JDBC ............................................................................................8-1
Practice 8-1: Investigating JDBC Problems ...................................................................................................8-3
Practice Solution: Investigating JDBC Problems ............................................................................................8-8
Practices for Lesson 9: Troubleshooting Node Manager .............................................................................9-1
Practice 9-1: Investigating Node Manager Problems .....................................................................................9-3
Practice Solution: Investigating Node Manager Problems .............................................................................9-7
Practices for Lesson 10: Troubleshooting Clusters .....................................................................................10-1
Practices for Lesson 10: Overview .................................................................................................................10-2
Practice 10-1: Investigating Proxy Problems ..................................................................................................10-3
Practice Solution: Investigating Proxy Problems ............................................................................................10-8
Practice 10-2: Investigating Cluster Replication Problems .............................................................................10-9
Oracle WebLogic Server 12c: Troubleshooting Workshop Table of Contents

i
Practice Solution: Investigating Cluster Replication Problems .......................................................................10-13

Practices for Lesson 11: Troubleshooting Security .....................................................................................11-1
Practices for Lesson 11: Overview .................................................................................................................11-2
Practice 11-1: Troubleshooting Changes by Using Auditing ..........................................................................11-3
Practice Solution: Troubleshooting Changes by Using Auditing ....................................................................11-7
Practice 11-2: <OPTIONAL> Recovering the Lost Admin Password .............................................................11-8
Practice Solution: <OPTIONAL> Recovering the Lost Admin Password .......................................................11-11
Oracle WebLogic Server 12c: Troubleshooting Workshop Table of Contents

ii
Practices for Lesson 1: Course Overview

Chapter 1 - Page 1
Chapter 1

Course Overview
Practices for Lesson 1:

Practices for Lesson 1: Overview

Practices Overview
There are no practices for this lesson.
Practices for Lesson 1: Course Overview

Chapter 1 - Page 2

Troubleshooting: Overview
Chapter 2
Practices for Lesson 2: Troubleshooting: Overview

Chapter 2 - Page 1

Practices Overview
In these practices, you access your practice environment and run some scripts to set up the
WebLogic Server domain. There is also an optional practice to run the Oracle Remote
Diagnostic Agent.
The practice environment consists of your main computer, which is referred to as the gateway
machine, and two Linux hosts, known as host01 and host02. You access the Linux hosts
from the gateway machine by using VNC Viewer.
Here is a diagram of the setup, and what will be running on each host:

Chapter 2 - Page 2
Practice 2-1: Accessing and Setting Up the Practice Environment

Overview
In this practice you access your practice environment for the first time. You also run some
scripts to set up the WebLogic Server domain. Finally, you use the administration console to
investigate the domain.
Here is a diagram of the domain the scripts create:

Assumptions
You are sitting at your classroom machine or have been assigned a remote machine.
Tasks
1. Access host01.
a. From your gateway machine, double-click the VNC Viewer - host01 icon on the
desktop. You connect to host01 as the oracle user. The Username field is not
displayed. Enter oracle in the Password field (it is displayed as ******) and press
the Enter key.

Chapter 2 - Page 3
b. Unless stated otherwise, you will be working within the VNC session of host01 for the
remainder of this practice.
2. Run the scripts to set up your domain.
a. Open a Terminal window.
Note: There is a launcher for a Terminal window in the panel at the top of the desktop.

b. Navigate to /practices/tshoot/practice02-01.
$> cd /practices/tshoot/practice02-01
Note:
The files you need for all the practices are found under /practices/tshoot.
Remember that you can use the Tab key to auto-complete directory and file names
as you type them
c. Run the main script, setup.sh, which calls the other scripts that set up the WebLogic
Server domain.
$> ./setup.sh
Important Note: These scripts will take a few minutes to complete. As they run they will
print informational messages to the Terminal window. Dont worry if there is sometimes a
pause in the messages. Be patient and let all the scripts run. The command prompt will
return when all the scripts have completed their work.
These are the scripts that are called:
1-createdomain.sh: This script calls the WLST script create_domain.py,
which creates a domain called wlsadmin in the directory /u01/domains/tshoot.
The WLST script configures the domain with an administration server running on
host01.example.com on port 7001. It also defines a cluster named cluster1,
which includes two managed servers: server1 running on host01.example.com
on port 7011, and server2 running on host02.example.com, also on port
7011.
2-packdomain.sh: This script runs the pack utility to create a managed server
template based on the new domain. The template file is named wlsadmin.jar and
placed in the /tmp directory.
3-copydomain.sh: This script uses secure copy to copy the template file over to
host02. It then uses secure shell to call the unpack script on host02.
4-unpackdomain.sh: This script runs on host02 and calls the unpack utility to
create a copy of the domain on this host.
5-startnodemanagers.sh: This script first calls the Node Manager start script
on host01, and then uses secure shell to call the Node Manager start script on
host02.

Chapter 2 - Page 4
startadmin.sh: This script, in the /practices/tshoot/utilities directory,

calls the domain script to start the administration server.
6-ohs.sh: This script copies the OHS configuration file, mod_wl_ohs.conf
(configuring OHS to be the proxy for the domains cluster), from the practice
directory over to the configuration directory of an OHS instance. It then uses
opmnctl to start that OHS instance. After OHS starts, the script calls opmnctl
again to print out the status. You should see that the instance of OHS, called ohs1,
is Alive and running HTTP on port 7777.
3. Take note of some utility scripts available to you.
a. In a Terminal window, navigate to /practices/tshoot/utilities. See what

scripts are there.
$> cd /practices/tshoot/utilities
$> ls
The utility scripts are:
killallnodemanagers.sh: This script calls the killnodemanager.sh script
twice: first on host01 and then, by using ssh, on host02.
killnodemanager.sh: This script finds the process ID of the Node Manager
process running on a host and kills that process.
startadmin.sh: This script calls the domain script to start the administration
server, if the administration server is not already running.
startallnodemanagers.sh: This script calls the startnodemanager.sh script
twice: first on host01 and then, by using ssh, on host02.
startnodemanager.sh: This script calls the script to start the Node Manager on a
machine, if it is not already running.
b. If you want to look at these scripts, you can open them by using the gedit editor. If
you do this, please ensure you close the editor without making any changes to the
scripts. For example, if you want to look at the script that starts the admin server:
$> gedit startadmin.sh
Note: Exit the editor by selecting File > Quit.
c. Close the Terminal window.
d. Exit the VNC Viewer.
4. Use the web browser on the gateway machine to access the WebLogic Server
administration console. Use the administration console to start the managed servers and
look around at the domain.
a. If the VNC Viewer is still open, close it.
Note: Although you can use a web browser on one of the host machines to access the
administration console, there is no need to access either host machine when you need
the administration console, because you can use the web browser on the gateway
machine.
b. On the gateway machine, double-click the Mozilla Firefox icon on the desktop.
Note: You can make Firefox your default browser if you want to.
c. When the web browser opens, enter the URL for the administration console:
http://host01.example.com:7001/console
Note: The URL for the administration console is the host and port of the administration
server followed by the context path of the admin console application.

Chapter 2 - Page 5
d. On the welcome page, enter the Username of weblogic and the Password of
Welcome1. Then click the Login button.
Note: You can have Firefox remember the password if you want to.
e. In the Domain Structure, expand Environment and select Servers.
f. Click the Control tab.
g. Select the check boxes next to server1 and server2. Then click the Start button.
When asked whether you are sure, click Yes.
h. Wait a moment, and refresh the browser. Wait until both managed servers have the
state of RUNNING.
i. Use the administration console to view the servers, machines, and clusters defined in

the domain. Answer the following questions:
What ports are the servers running on?
Which servers are on which machines?
What address and port does each Node Manager run on?
Hint: Remember that a Node Manager configuration is under a machines
configuration.
What messaging mode does the cluster use?
j. Log out of the administration console.
k. Close the web browser.

Chapter 2 - Page 6
Practice Solution: Accessing and Setting Up the Practice

Environment
There is no solution for this practice.

Chapter 2 - Page 7
Practice 2-2: <OPTIONAL> Running RDA

Overview
In this practice, you run the Oracle Remote Diagnostic Agent. You then view some of its output.
This practice can be performed at any time during the course, whenever you have extra time.
Assumptions
You completed Practice 2-1: Accessing and Setting Up the Practice Environment and the
domain has been successfully created.
All instances of WebLogic Server are running.

Tasks
1. Access host01.
the Enter key.
2. Run the RDA utility to create a configuration file.
a. Open a Terminal window, and run the domains script to set up environment variables.
$> /u01/domains/tshoot/wlsadmin/bin/setDomainEnv.sh
b. Navigate to the location of the RDA utility.
$> cd /u01/app/fmw/oracle_common/rda
c. Run RDA with the WebLogic Server profile to set up your configuration file.
$> ./rda.sh Sp WebLogicServer
--------------------------------------------
S00INI: Initializes the Data Collection
--------------------------------------------
RDA uses the output file prefix...
...
Note: As RDA asks questions or gives commands, you type your response and press
the Enter key. To accept the default, just press the Enter key. See the table in the next
step for the proper responses.
d. Use this table to answer the RDA questions and commands:
Step Question / Command Response
a. Enter the prefix to be used for all the (press Enter for the default of
generated files. RDA)
b. Enter the directory used for all the files to be /home/oracle/rda_output
generated.
c. Do you want to keep report packages from (press Enter)
previous runs (Y/N)?
d. Enter the Oracle home to be used for data /u01/app/fmw
analysis.
e. Enter the network domain name for this example.com

Chapter 2 - Page 8
Step Question / Command Response

server.
f. Do you want RDA to perform the network (press Enter)
ping tests (Y/N)?
g. Enter the location of the directory where the /u01/domains/tshoot
domains to analyze are located.
h. Data Collection Type: 3
1. Collect for a single server (offline mode)
2. Collect for a single server ()

3. Collect for multiple servers ()
Enter the item number.
i. *
Enter the number corresponding to the
domain directory name, or * to select all
domains, or . to terminate the list.
j. *
Enter the number corresponding to the
server directory name, or * to select all
servers, or . to terminate the list.
k. Enter the username for the Oracle WebLogic weblogic
Server connection for wlsadmin domain.
l. Are custom scripts used to start Oracle (press Enter)
WebLogic Servers (Y/N)?
m. Enter the maximum number of thread dumps 2
to capture when a data collection module
requires online collections.
n. Enter the interval (in milliseconds) between (press Enter)
the thread dumps.
e. Verify that the configuration file has been created.
$> ls setup.cfg
setup.cfg
3. Run RDA again to collect data.
a. In the same Terminal window, run RDA again.
$> ./rda.sh
--------------------------------------------
RDA Data Collection Started ...
--------------------------------------------
Processing ...
b. Various messages print out. Eventually RDA will ask for a password. Enter Welcome1
each time and press Enter.
Enter 'weblogic' user password for 'wlsadmin' domain: Welcome1
Please re-enter it to confirm: Welcome1

Chapter 2 - Page 9
Note: The password is not displayed.

c. Wait for RDA to collect data. This may take several minutes. When RDA is finished, it
prints out that the data collection is finished and the prompt returns.
...
Processing ...
--------------------------------------------
RDA Data Collection Ended ...
--------------------------------------------
You can review the reports by ...

...
$>
4. View the RDA output.
a. In the same Terminal window, list the contents of the output directory.
$> cd /home/oracle/rda_output
$> ls
Note: Many, many files and directories are present. One of them is the archive file,
RDA.RDA_host01.zip. Also notice the main HTML page, RDA__start.htm.
b. View the starting page of the HTML output by using the Firefox web browser (there are
two underscore characters in the file name).
$> firefox RDA__start.htm
Note: Notice the main index at the top left. The overview links are currently at the bottom
left. The main overview is in the display area. Ignore any messages in the Terminal
window.

Chapter 2 - Page 10
c. The top part of the index has links that show the machines configuration.

d. The part under Oracle WebLogic Server has links for WebLogic Server data.
Note:
The Oracle WebLogic Server Overview link has installation information such as the
version and patches applied.
The Domain link has links to domain scripts and configuration files.
The individual server links contain server information including log files, deployed
application deployment descriptors, diagnostic images, and thread dumps.
e. Select the link in the main index called server1 Server. Click some links in the
server1 Server frame (below the Main Index). Try some such as:
server1.state (under $SH/data/nodemanager) Note: This is the state of the
server as kept by Node Manager.
server1.log (under Log Files) Note: This is the latest server log.
JVM Runtime (WLST Collections > Server Runtime) Note: This is information
about the JVM under which this server is running.
f. Click around other areas as you have time and interest.
5. Clean up.
a. Close the web browser on host01.
b. Return to the host01 Terminal window.
c. Remove the rda_output directory.
$> cd /home/oracle
$> rm rf rda_output

Chapter 2 - Page 11
Warning: This command permanently deletes a directory, all its subdirectories, and files.
Use with caution.
d. Close the Terminal window.
e. Exit the VNC Viewer.

Chapter 2 - Page 12
Practice Solution: <OPTIONAL> Running RDA

Chapter 2 - Page 13

Chapter 2 - Page 14

WebLogic Server Diagnostic
Framework: Overview
Chapter 3
Practices for Lesson 3: WebLogic Server Diagnostic Framework: Overview

Chapter 3 - Page 1

Practices Overview
In these practices you set up WebLogic Diagnostic Framework harvesters, watches, and
notifications. You also use the Monitoring Dashboard to view diagnostic information as charts
and graphs.

Chapter 3 - Page 2
Practice 3-1: Harvesting Diagnostic Metrics

Scenario
Users have reported intermittent database problems with the contacts application. Before you
speak to the development team, you decide to gather some data. The WebLogic Server
Diagnostics Framework (WLDF) allows you to collect and record server and application metrics
over time for later analysis. WLDF also supports watch conditions that can trigger notifications,
such as posting a JMS message or capturing a diagnostic image.
Overview
In this practice you configure some archive policies. You also create a new system-level WLDF

diagnostic module. In that module you configure metric collectors (harvesters) for several
MBean types, trigger notifications under various MBean watch conditions, and access
diagnostic archives to inspect recorded metrics.
Assumptions
Tasks
1. Access host02 and run the database setup script.
the Enter key.
b. Open a Terminal window, navigate to the current practice directory and run the
database setup script, initdatabase.sh.
$> ./initdatabase.sh
Note:
You run the script on host02 because that is where the database is running.
This script calls a SQL*Plus script that initializes the database. Note that this script
first drops the sequence and table it creates, so it can be run multiple times.
Therefore, the first time it runs you will see messages that the sequence and table
do not exist. Ignore those messages.
d. Exit the host02 VNC Viewer.
2. Access host01 and run the main setup script.
the Enter key.
Note: You access the two Linux hosts from the gateway machine many times in these
practices. Rather than give specific directions each time, in this and subsequent
practices it will be assumed you know what to do if you are instructed to access
host01 or access host02.

Chapter 3 - Page 3
b. Open a Terminal window, navigate to the current practice directory and run the setup
script, setup.sh.
$> ./setup.sh
This script calls:
A WLST script to create a data source called datasource1
A WLST script to deploy a web application that uses the data source
A WLST script that sets up a JMS Server, a JMS module, and a JMS queue (for
WLDF notifications)
Note:

Ignore any messages about connecting to the admin server with an insecure
protocol. In the classroom, it does not matter that you access the admin server
through a nonsecure port.
Also ignore the message about WLContext.close() being called in a different
thread. This message always appears when you deploy with WLST and does not
have any real effect.
Wait until all the scripts run, and the prompt returns, before continuing.
d. Minimize the host01 VNC Viewer to use later.
3. Access the WebLogic Server administration console from the gateway machine.
a. On your gateway machines desktop, double-click the Mozilla Firefox web browser
icon.
b. In the web browser, enter the URL to the administration console:
http://host01.example.com:7001/console
c. On the console welcome page, enter the Username as weblogic and Password as
Welcome1. Then click the Login button.
Note: You access the administration console from the gateway machine web browser
many times in these practices. Rather than give specific directions each time, in this
and subsequent practices, it will be assumed you know what to do when you are
instructed to use (or access) the administration console.
4. Check the type and location of the WLDF diagnostic archives. Set the maximum size of a
servers archive file. Set a servers archive retirement policy.
a. In the Domain Structure, expand Diagnostics (click the + sign) and then select
Archives.
Note: In this and subsequent practices, the instructions for accessing something in the
Domain Structure will be abbreviated. For this case, the abbreviation is: In the Domain
Structure, select Diagnostics > Archives.
b. Notice the type of archive and their location for each server. The default archive type is
a file store archive.
c. Select server1 in the table. Under the Configuration tab, you can see that the Type of
archive is a File Store.
Note: The other type of archive is a database archive. In the admin console a
database archive displays its type as JDBC.
d. You want to make a change to the configuration, so lock it. In the Change Center, click
the Lock & Edit button.

Chapter 3 - Page 4
e. To save disk space, you decide to limit the size of this servers archive file to 50
megabytes. Set the Preferred Store Size to 50.
f. Save this change by clicking the Save button.
g. Activate the changes by clicking the Activate Changes button in the Change Center.
Note: You will make changes to the domain configuration many times in these
practices. In this and subsequent practices, it will be assumed you no longer need
specific directions when you are instructed to lock the configuration, save changes,
or activate changes.
h. Because you do not want to keep harvested diagnostic archive information forever, you
want old data to be retired. Scroll down and notice that the default is for Data
Retirement Enabled to be selected.

i. Create a new data retirement policy for harvested data.
Scenario: You want to retire harvested archive data after two days, start the first
retirement task at midnight, and run the retirement task every 8 hours.
j. Lock the configuration again. Click the New button above the Data Retirement Policies
table. Use the table below to fill in the information for the policy, and then click OK.
Step Field Choice or Value
a. Name retirement-policy-1
b. Archive Name HarvestedDataArchive
c. Retirement Age 48
d. Retirement Time 0
e. Retirement Period 8
f. Enabled Selected
k. Activate the changes when you are done.
5. Check the location of diagnostic images for server1 (running on host01). As a first step in
gathering information, perform an on demand capture of a diagnostic image.
a. Return to the Domain Structure and select Diagnostics > Diagnostic Images. Notice
the location of diagnostic images for all the servers, particularly server1.
b. Select the option button next to server1 and click the Capture Image button.
c. When you capture a diagnostic image, you are allowed to change the destination
directory and timeout. (The timeout specifies how long WebLogic Server waits, in
minutes, in between diagnostic image captures.) Make note of the location of
diagnostic images. Leave the defaults as they are and click OK.
d. Minimize the web browser to use later.

Chapter 3 - Page 5
e. Access host01. Use the File Browser on host01 to navigate to the location where the
image was saved:
/u01/domains/tshoot/wlsadmin/servers/server1/logs/
diagnostic_images.
Tip: The File Browser can be found in the panel under Applications > System
Tools. Once it opens, click File System under the Places panel to start your
navigation.

f. In the File Browser, right-click the .zip file (its name starts with
diagnostic_image_). Select Open with Archive Mounter. Then in the Places
panel of the File Browser (on the left), select the drive that has the same name as the
zip file.
Note: The information stored in the diagnostic image zip file is saved in various XML
files. Most have the extension of .img.
g. Look at the contents of the image.summary file. Right-click it and select Open with
gedit.
Note: If not given that choice in the pop-up menu, select Open with Other
Application. Then select gedit and click the Open button.
h. In this file, can you find the operating system and version? Can you find under which
JVM WebLogic Server is running?
Tips: Use the Search > Find menu option to search for os.name and os.version.
Then search for java.vendor and java.version.
i. Return to the File Browser and open the JDBC.img file with gedit to look at its
contents.
Note: Notice the information about the data source called datasource1.
j. Close the editor when you are finished.
k. In the File Browser, right-click the drive that represents the zip file. Select Unmount.
l. Close the File Browser.
m. Minimize the VNC Viewer to use later.
6. Create a new diagnostic system module.
Note: To create harvesters (metric collectors), watches, and notifications, you first create a
diagnostic module to contain them.
a. Return to the administration console and lock the configuration.
b. In the Domain Structure, select Diagnostics > Diagnostic Module.
c. Click the New button.
d. Give the module the Name server1-diagnostics. Then click OK.
e. Edit the new module.

Chapter 3 - Page 6
Tip: Click the name of the module in the table.

f. Click the Targets tab.
g. Select server1 (under cluster1) and click Save.
Note: Make sure you select server1. It may not be the first server in the list under
cluster1. When you select it, cluster1 and Part of the cluster are automatically
selected.
h. Do not activate changes yet.
7. Define a metric collector (harvester) in the diagnostic module.
a. Click the Configuration > Collected Metrics tabs of the diagnostic module.
b. Set the Sampling Period to 10000 (10 seconds) and click Save.

d. Leave the MBean Server location as ServerRuntime and click Next.
Note: Because this diagnostic module is targeted to a managed server, only the Server
Runtime MBeans apply.
e. Ensure that Select an MBean Type from the following list is selected. From the
drop-down list, select the weblogic.management.runtime.
JDBCConnectionPoolRuntimeMBean. Click Next.
f. Move the following attributes from the Available column to the Chosen column:
ActiveConnectionsCurrentCount (the number of connections currently in use by
applications)
CurrCapacity (the current count of JDBC connections in the connection pool in the
data source)
LeakedConnectionCount (the number of leaked connections)
Note: A leaked connection is a connection that was reserved from the data source but
was not returned to the data source.
g. Click Next.
h. Note the available values for Collected Instances, but make no selection.
Note: If you do not identify specific MBean instances, WLDF collects the selected
attributes from all available instances.
i. Click Finish.
j. Do not activate changes yet.
8. Create a diagnostic notification.
a. Click the Configuration > Watches and Notifications tabs of the diagnostic module.
b. Scroll down and click the Notifications tab.
d. For the Type field, select the JMS Message option. Click Next
e. Enter jms-notification as the Notification Name. Ensure that Enable
Notification is selected. Click Next.
f. Enter the following values and then click Finish:
a. JMS Destination JNDI wldfnotificationqueue
Name
b. Connection Factory weblogic.jms.XAConnectionFactory

Chapter 3 - Page 7

JNDI Name
Note: Be sure you type these values exactly as shown.
g. Do not activate changes yet.
9. Create a diagnostic watch.
a. Scroll down and click the Watches tab near the bottom of the Watches and
Notifications screen. Then click New.
b. Enter jdbcwatch for the Watch Name. Confirm that the Watch Type is Collected
Metrics. Confirm that Enable Watch is selected. Click Next.

c. Click the Add Expressions button.
d. Ensure ServerRuntime is selected and click Next.
e. Ensure that Select an MBean Type from the following list is selected. From the
drop-down list, select the same weblogic.management.runtime.
JDBCConnectionPoolRuntimeMBean that is being collected and click Next.
f. Click Next to watch all instances of this MBean.
g. Enter or select the following values and then click Finish:
Step Field Value
a. Message Attribute ActiveConnectionsCurrentCount
b. Operator >
c. Value 0
h. Click Next.
i. Select the Use a Manual Reset Alarm option and click Next.
j. Move the jms-notification option from the Available column to the Chosen
column.
k. Click Finish.
l. Activate the changes.
m. Minimize the web browser. You will be using the admin console later.
10. Trigger WLDF notifications.
a. Return to the VNC Viewer for host01. Open a Terminal window and navigate to the
current practice directory. Run the script that runs a client that accesses the database,
runclient.sh. The client will ask you to press Enter when you want to stop it. Do
not do that yet.
$> ./runclient.sh
...
This client is still running...
When you are ready to stop it, press Enter.
Note: This runs a stand-alone Java class that opens connections to the database but
does not close them. Wait a minute before proceeding to the next step.
b. Do not close the Terminal window.
c. Minimize the VNC Viewer.

Chapter 3 - Page 8
11. View the notification and collected metrics by using the admin console.
a. Access the admin console. In the Domain Structure select Services > Messaging >
JMS Modules.
b. Select the module called jmsmodule1.
c. Select the resource (queue) wldfnotificationqueue.
d. Select the Monitoring tab.
e. Notice that there is a message in the queue (Messages Current). You could select the
queue and click the Show Messages button, but the message is not a text message,
so viewing it by using the admin console is not very interesting.
f. From the Domain Structure panel, select Diagnostics > Log Files.

g. Locate and select the HarvestedDataArchive log file for the server1 server.
h. Click View.
i. Click the Customize this table link.
j. For Time Interval, ensure that Last 5 Minute(s) is selected.
Note: This may need to be increased, if you did not just run the client.
k. Under Column Display, update the Chosen column to contain only these fields:
Date
Attribute
Value
l. For Number of rows displayed per page, select 100.
m. Click Apply.
n. Browse the harvester entries. You may need to scroll through a few pages to get to the
JDBC connection pool metrics after the client was running. Verify that the MBean
attributes are being collected approximately every 10 seconds.
12. Stop the database client.
a. Return to the VNC Viewer for host01.
b. Find the Terminal window where the database client is running. Press Enter.

Chapter 3 - Page 9
Practice Solution: Harvesting Diagnostic Metrics

Perform the following tasks if you did not do this practice and want to use the finished solution.
Assumptions
Solution Tasks
1. Access host02 by using the VNC Viewer on the gateway machine. Run the database setup

script.
a. Open a Terminal window, navigate to the current practice directory and run the
database setup script, initdatabase.sh.
$> ./initdatabase.sh
2. Access host01 by using the VNC Viewer on the gateway machine. Run the solution script.
a. Open a Terminal window, navigate to the current practice directory and run the solution
script, solution.sh.
$> ./solution.sh
Note:
This script creates the WLDF system module with its harvester, watch, and
notification. These elements of the WLDF module are not enabled. You could
enable them manually by using the administration console.
Ignore any messages that state No stack trace available. The script checks for the
existence of a resource by trying to access the resource and catching an exception
when it does not exist. When there is no Java stack trace and there is also an
exception, then this message prints out.

Chapter 3 - Page 10
Practice 3-2: Monitoring Diagnostic Metrics

Overview
In this practice you access the Monitoring Dashboard to view harvested metric data.
Assumptions
You completed Practice 3-1: Harvesting Diagnostic Metrics (or have run the solution).
Tasks

1. Use the Monitoring Dashboard to monitor harvested data.
a. Access the administration console and log in.
b. On the Home page, click the Monitoring Dashboard link (under Charts and Graphs).
c. In the left panel, click the My Views folder.
d. Click the drop-down menu arrow (below the tabs).
e. Select New View. Type over the default name of the new view with: Harvested
Diagnostics.
f. Ensure the new view is selected and click the Metric Browser tab.
g. In the Servers drop-down list, select server1.
h. Select Collected Metrics Only, and click the Go button.
i. In the Types list of available MBeans on this server, locate and select
JDBCConnectionPool.
j. Under Instances, select datasource1.
k. Under Metrics, drag the three metrics onto the right panel. As the first one is dragged
in, a new chart is created. Drag the other two onto the same chart.
l. Change the charts default name by clicking the menu down-arrow in the chart title bar.
Select Properties. Under Chart Title, enter JDBC Chart. Click OK.
m. View the data in the chart.

Tip: If you see no data points in the chart, then you must set the time of the chart to
include when the metrics were actually collected (rather than the default time). Perform
the steps below, if that is the case:
Return to the admin console.
Under Diagnostics > Log Files, select HarvestedDataArchive for server1 and
click View. Click Customize this table. For Time Interval, select: Last 1

Chapter 3 - Page 11
hour(s) (or whatever time interval seems best based on when you did the
previous practice). Set Number of rows displayed per page to 1000.Click Apply.
Make note of the date and time of the collected JDBC attributes. (You may need to
scroll to find the ones of interest after the client was running.)
Return to the Monitoring Dashboard.
At the top-right of the chart, click the down arrow for the menu again, and select
Properties.
Under Time Range select Custom and enter the Start Time. Use the correct
format. For example: In the log file if the Date field was "10/11/13 18:50:41
820" you would enter it here as: "10/11/2013 6:50:41 PM" (do not include

the double quotation marks).
Enter a Duration of 5 minutes: 0 00:05
Click OK.
n. Look at the chart again. It should contain data points.
o. Close the Monitoring Dashboard.
p. Close the web browser.

Chapter 3 - Page 12
Practice Solution: Monitoring Diagnostic Metrics

Chapter 3 - Page 13
Chapter 3 - Page 14

Diagnostic Instrumentation
Chapter 4
Practices for Lesson 4: Diagnostic Instrumentation

Chapter 4 - Page 1

Practices Overview
In these practices, you create and configure an application-scoped diagnostic module which
includes a monitor. You also configure instrumentation on the module to trace a client request.

Chapter 4 - Page 2
Practice 4-1: Configuring and Monitoring Diagnostic Events

Scenario
Users have reported intermittent database problems with the contacts application. You talk to a
developer who guesses the problem might be in connecting to the database. You decide to
gather some specific information about this application and how long it takes to make a
database connection. You use application-scoped instrumentation to be able to gather this
information.
Overview
In this practice, you configure an application-scoped diagnostic monitor. Application-scoped

monitors are part of application-scoped diagnostic modules, so you add a diagnostic module to
an application. You select a delegating monitor (it has predefined code locations, but you
choose the action to take). After the action has been triggered, you look at the data stored in the
event archive.
Assumptions
You completed practice 3-1: Harvesting Diagnostic Metrics.
Tasks
1. Enable instrumentation on the server named server1.
Note: For instrumentation to be available for an application, instrumentation must be
enabled on the server to which the application is deployed.
a. Access the administration console, log in, and lock the configuration.
b. Navigate to your diagnostics module, server1-diagnostics.
Tip: In the Domain Structure: Diagnostics > Diagnostic Modules > server1-
diagnostics.
c. Click the Configuration > Instrumentation tabs.
d. Select the Enabled check box.
Note: It is here that you could add instrumentation to this system-scoped diagnostic
module. This practice does not have you do that, however, because you will instead
add instrumentation to an application-scoped diagnostic module.
e. Save and activate your changes.
f. Minimize the admin console for later.
2. Add a WLDF descriptor file to an application. Then redeploy the application.
a. Access host01.
b. Inspect the contents of the WLDF descriptor file provided under the current practice
directory:
META-INF/weblogic-diagnostics.xml
Tip: Open a Terminal window, navigate to the current practice directory, and use the
cat command.
$> cat META-INF/weblogic-diagnostics.xml
c. In a Terminal window run the jar utility to add the WLDF descriptor file to the
contacts application.

Chapter 4 - Page 3
$> /u01/app/jdk/bin/jar uf
/u01/domains/tshoot/wlsadmin/apps/contacts.war
META-INF
Note:
Enter all commands on one line. Commands are sometimes displayed on more than
one line due to space limitations.
The jar utility option uf means to update the file specified.
The file, contacts.war, is updated with the last parameter, the META-INF
directory.

d. Using the jar utility or the File Browser and Archive Mounter, inspect the
contacts.war file and confirm that the WLDF descriptor file was added to the WAR
files META-INF directory.
Tip: To use the jar utility, run the jar as before, but replace uf with tf (table of
contents for the file specified) and remove the META-INF at the end of the command.
$> /u01/app/jdk/bin/jar tf
/u01/domains/tshoot/wlsadmin/apps/contacts.war
e. Minimize the VNC Viewer to use later.
f. Return to the admin console and lock the configuration.
g. Select Deployments in the Domain Structure.
h. Select the check box next to the contacts application and click Update.
i. Click Finish.
j. Activate the changes.
3. Generate a deployment plan for updating the WLDF module in the application by using the
admin console.
a. In the Deployments table, click the name of the contacts application.
b. Click the Configuration > Instrumentation tabs.
c. Lock the configuration again.
d. Ensure Enabled is selected.
e. Scroll down and click the Add Monitor From Library button.
f. Move the JDBC_Around_GetConnection option from the Available column to the
Chosen column.
Note: This around monitor is triggered before and after a database connection is
retrieved.
g. Click OK.
Note: Because this is a change to the application-scoped diagnostic monitor, and a
deployment plan does not yet exist for this application, the Save Deployment Plan
Assistant is launched.
h. Set the Path to:
/u01/domains/tshoot/wlsadmin/apps/contacts-plan.xml
i. Click OK.
j. Return to the applications Configuration > Instrumentation tabs.
k. In the Included Classes text area enter:

Chapter 4 - Page 4
stcurr.*
Note: The class that gets the database connection is in the stcurr package.
l. Click Save.
m. Do not activate changes yet.
4. Configure the delegating monitor.
a. In the Diagnostic Monitors in this Module table, select the new monitor.
b. Under Actions, move the TraceElapsedTimeAction from the Available column to the
Chosen column.
Note: This action makes two event archive entries: one before and one after the
specified code location. It is only compatible with Around monitor types.

c. Click Save.
d. Activate all your changes.
e. Return to the contacts applications main configuration page and inspect the
Overview tab. Confirm that the Deployment Plan field now has a value.
Tip: Make note of the files name and location.
g. Return to host01. Use a Terminal window or the File Browser to find and inspect the
deployment plan file. The plan now has new variables, which will update the
weblogic-diagnostics.xml file. In the plan file, notice that there are now variables
that start with WLDFInstrumentationMonitor.
Note: If you view the deployment plan file by using the gedit editor, make sure you
do not make any changes to the file.
5. Deploy the latest version of the application, with the deployment plan.
a. Return to the admin console and lock the configuration.
b. In the Domain Structure, select Deployments.
c. Select the check box next to the contacts application, and click the Update button.
d. Select Update this application in place with new deployment plan changes.
Ensure that the Deployment plan path (under this option) is correct.
e. Click Finish.
f. Activate the changes.
g. Ensure that the contacts applications state is Active.
6. Run the application and inspect the generated events.
a. Use the Firefox web browser on the gateway machine. Enter the URL for the
application:
http://host01.example.com:7011/contacts
b. Click the browse all contacts link.
c. Click the [edit] link on a row.

d. Use the page to update some fields. Click the Update button.
e. Do this a few times.

Chapter 4 - Page 5
f. Return to the admin console. From the Domain Structure panel, select Diagnostics >
Log Files.
g. Locate and select the EventsDataArchive log file for the server1 server. Click View.
Note:
If you dont see any rows in the table, click the Customize this table link. Change
the Time Interval to ensure you are going back far enough to see the events
created. Click Apply.
If after doing that, you still dont see any rows, return to the task to deploy the latest
version of the application and do it again.
h. Browse the generated events. Notice the Date column. For each Before and After

event (look in the Type column) you can see how long connecting to the database took
by comparing the times. (The number on the end is milliseconds.)
7. View request performance in the admin console.
Note: Because instrumentation is enabled, and the monitor is an Around type, you can
view request performance data by using the admin console.
a. In the Domain Structure, select Diagnostics > Request Performance.
b. For Servers select server1. For Interval select Last 5 mins (or a time that will
include running the application).
c. Click the Refresh button.
d. View the results. To see what is under a request, click the + next to one.
e. To see the details of a request, click the string that identifies the request.
f. The details are displayed below the Requests tree.
Note: Above the details table are the WLDF context ID and the total elapsed time of
the request in milliseconds. The details include the class and method, the number of
times the method was called in the request, the total milliseconds of the calls, the

Chapter 4 - Page 6
average milliseconds of the calls, the application name, the application module, and
the method signature.
g. Close the admin console.

Chapter 4 - Page 7
Practice Solution: Configuring and Monitoring Diagnostic Events

Assumptions
You completed practice 3-1: Harvesting Diagnostic Metrics.
Solution Tasks
1. Access host01 by using the VNC Viewer on the gateway machine. Run the solution script.
a. Open a Terminal window, navigate to the practice directory and run the database setup

$> ./solution.sh
Note: This script:
Copies the new version of the application and its new deployment plan to the apps
directory under the domain.
Calls a WLST script to create a system-scoped diagnostic module (in case it does
not already exist).
Calls another WLST script to enable instrumentation in the module (and therefore
for the server to which it is targeted).
Calls another WLST script to deploy the application with its deployment plan.
(Ignore any messages about WLContext.close() being called in a different
thread.)

Chapter 4 - Page 8
Practice 4-2: Tracing a Client Request

Scenario
Users have reported intermittent database problems with the contacts application. After some
investigation, the problem seems to be JDBC related, but only for certain users. You want to be
able to trace a users request through JDBC operations. You also want to limit the diagnostic
events that are saved to a particular user by using their IP address.
Overview
When instrumentation is enabled on a diagnostic module, all WLDF events and log entries
associated with a particular client request will be tagged with a unique context ID. These IDs

allow administrators to trace all the diagnostic data for a specific request.
In some troubleshooting cases, you may wish to go a step further and limit the conditions under
which WLDF events are recorded. A diagnostic context ID supports various flags or dyes that
allow you to filter the types of requests that trigger instrumentation data.
You also enable some debug flags on the server so there is more detailed information saved in
the logs.
Assumptions
You completed practice 4-1: Configuring and Monitoring Diagnostic Events.
Tasks
1. Generate debug messages in the server log file.
a. Access the admin console. Lock the configuration.
b. In the Domain Structure, select Environment > Servers. Select server1.
c. Click the Debug tab.
d. Locate the weblogic > servlet > internal debug scope.
e. Select the check box for the internal scope. Click the Enable button.
Note: That scope and its children are now enabled. A couple of other scopes (that do
not seem like children, but are) are also enabled. That is expected.
f. Similarly, enable the weblogic > jdbc debug scope.

Note: That scope and its children are now enabled. (Not all the children are shown
here.)

Chapter 4 - Page 9
g. Activate your changes.

2. Run the application again and once again inspect the generated events.
a. Use the Firefox web browser on the gateway machine. Enter the URL for the
application:

c. Click the [edit] link on a row.
d. Use the page to update some fields. Click the Update button.
e. Do this a few times.
f. Return to the admin console. From the Domain Structure panel, select Diagnostics >
Log Files.
g. Locate and select the EventsDataArchive log file for the server1 server. Click View.
Note: If you dont see any rows in the table, click the Customize this table link.
Change the Time Interval to ensure you are going back far enough to see the events
created. Click Apply.
h. This time notice the Context ID field. Also notice that it changes with each request,
and sometimes there are more rows for one request than another.
Note: If the Context ID field is not displayed, click Customize this table. Add the
Context ID column to the chosen list and click Apply.
i. Write down the last five characters of one of the context IDs. For example, if the
context ID is 9d9fbb3a-0e57-4274-94fe-e1b1fb9aaa77-00014a17, write
down 14a17.
3. Correlate WLDF and server log messages.
a. Use the admin console to view the ServerLog for server1.
Tip: In the Domain Structure select Diagnostics > Log Files. Select ServerLog for
server1. Click View.
b. Click Customize this table. Add the Context ID column to the chosen list. Increase
the Number of Rows Displayed Per Page to 1000. Click Apply.
Tip: You might want to change the order of the columns so the Context ID is first or
second. In the Chosen column, select it and move it with the arrows.
c. Browse the recent log entries. Notice that some entries have been assigned context
IDs while others have not. The latter are internal messages not associated with a client
request.
d. Click Customize this table again.
e. For WLDF Query Expression, enter the following (where nnnnn is the last five
characters of the context ID that you wrote down).
CONTEXTID LIKE '%nnnnn'
f. Click Apply.

Chapter 4 - Page 10
Tip: If you do not see any entries, click Customize this table again and change Time
Interval to Last 15 minute(s) (or longer).
g. Browse the log entries. Notice that you can trace a particular request through
extremely detailed JDBC operations, thanks to the context ID and the debug
messages.
h. Minimize the admin console to use later.
4. Configure dye injection monitoring based on a dye filter, which is a particular IP address.
Note: You will pretend that the user having trouble is accessing the contacts application
from the web browser on host01.
a. Find out the IP address of host01. Access host01. Open a Terminal window and enter

the command ifconfig. Make note of the address.
$> ifconfig
eth0 Link encap:Ethernet HWaddr 00:16:3E:01:01:00
inet addr:192.0.2.11 Bcast:192.0.2.255 ...
Note: Look for the inet addr in the second line of the eth0 print out. Your IP
address may be different.
b. Close the Terminal window. Minimize the VNC Viewer to use later.
c. Return to the admin console. Lock the configuration.
d. Locate and edit the system diagnostic module, server1-diagnostics.
e. Under Configuration > Instrumentation click the Add/Remove button.
f. Move DyeInjection from Available to Chosen. Click OK.
g. In the table, select the dye injection monitor name to edit it. Ensure Enabled is
selected. In the Properties text area enter:
ADDR1=host01_ip_address
Note: For example, if the host01 IP address is 192.0.2.11, you enter:
ADDR1=192.0.2.11
h. Click Save.
i. Activate your changes.
j. Lock the configuration again.
k. Locate and edit the contacts application (deployment).
l. Click Configuration > Instrumentation.
m. Select the monitor JDBC_Around_GetConnection.
n. Select Enable Dye Filtering.
o. For the Dye Mask, move ADDR1 from Available to Chosen.
p. Click Save.
q. Return to the Deployments table.
Tip: You can use the breadcrumbs and select Summary of Deployments.
r. Select the check box next to contacts. Click Update.
s. Select Update this application in place with new deployment plan changes.
t. Click Finish.
u. Activate the changes.
Note: Now the application has been updated with a new version of its diagnostic
module, updated through its deployment plan.

Chapter 4 - Page 11
v. Do not close the admin console. Use the File menu of Firefox to open a New Tab or
New Window.
5. Test event filtering.
a. Retest the contacts application as before, by entering the application URL
(http://host01.example.com:7011/contacts) in the new tab or window of the
Firefox web browser of the gateway machine. Edit a couple of contacts.
b. Return to the admin console and once again view the EventsDataArchive for server1.
c. Click Customize this table and ensure that the Time Interval is long enough to
include the last testing you just did, but not so long as to include the previous testing.
d. You should not see any rows in the Events Log Entries table. That is because the IP

address of this browser does not match the dye filter IP address.
e. Keep the admin console available.
f. Close the window or tab of the Firefox browser accessing the contacts application.
g. Access host01.
h. Click the Firefox icon in the panel on host01.
i. Use this web browser to access the contacts application and edit a few contacts.
j. Close the web browser. Exit the VNC Viewer.
k. Return to the admin console and again view the EventsDataArchive for server1.
l. Click Customize this table and ensure that the Time Interval includes the very latest
testing.
m. Now there are rows in the Events Log Entries table, because the IP address of the
host01 browser matches the dye filter IP address.
6. Edit the server1-dianostics module and disable the instrumentation component.
a. Return to the admin console and lock the configuration.
b. From Domain Structure, select Diagnostics > Diagnostic Modules.
c. Select server1-diagnostics.
d. Select Configuration > Instrumentation.
e. Deselect Enabled.
f. Click Save.
7. Disable the debug flags.
a. Access the admin console again. Lock the configuration.
b. In the Domain Structure, select Environment > Servers. Select server1.
c. Click the Debug tab.
d. Locate the weblogic > servlet > internal debug scope.
e. Select the check box for the internal scope. Click the Disable button.
f. Similarly, disable the weblogic > jdbc debug scope.

Chapter 4 - Page 12
Practice Solution: Tracing a Client Request

Chapter 4 - Page 13

Chapter 4 - Page 14
Practices for Lesson 5: JVM Troubleshooting

Chapter 5 - Page 1
Chapter 5

Troubleshooting
Practices for Lesson 5: JVM


Practices Overview
In this practice, you use JVisualVM and Java Mission Control.

Chapter 5 - Page 2
Practice 5-1: Troubleshooting a JVM

Scenario
Users have reported that the new benefits application is running slowly. You investigate
WebLogic Servers use of the JVMs memory as well as CPU usage while the new application is
being accessed.
Overview
In this practice, you troubleshoot a HotSpot JVM by using Java Visual VM (JVisualVM) and
Java Mission Control.

Assumptions
You completed practice 4-1: Configuring and Monitoring Diagnostic Events.
Tasks
1. Run the setup script to deploy the new application.
a. Access host01. Open a Terminal window and run the setup script in the current
practice directory.
$> ./setup.sh
Note:
The script deploys the new benefits application. This application:
Overuses the CPU
Hogs memory
Please ignore any messages about:
An insecure protocol being used to connect to the admin server.
WLContext.close() being called in a different thread.
2. Find the process ID of server1 by using the JVM Process Status tool.
a. In a Terminal window, navigate to the bin directory of the JVM and run the jps
command with the -v argument.
$> cd /u01/app/jdk/bin
$> ./jps -v
Note: The jps (JVM Process Status) command lists running JVMs. The -v option tells
the command to output arguments passed to the JVM.
b. Look through the output. The first thing listed in each entry is the process ID. Two of
the listings should be WebLogic Servers (after the PID it says: Server). Find the one
that is server1 (within the arguments passed to the JVM is an option that gives the
server name: -Dweblogic.Name=server1).
c. Make note of the PID of server1.
3. Run the JVisualVM tool to view CPU and memory usage before simulating users accessing
the new benefits application.
a. In a Terminal window, navigate to the bin directory of the JVM and run the
jvisualvm command.

Chapter 5 - Page 3
$> ./jvisualvm
Note: If an informational window is displayed about calibration, click OK. Then click
OK again.
b. When the graphical user interface comes up, under the Applications tab, expand
Local (if it is not already expanded). Find the WebLogic JVM that matches the PID of
server1 which you discovered earlier. Double-click this JVM.

Note: The pid values and other processes in the list may not match this screenshot
exactly.
c. On the right side, there is now a WebLogic tab. It starts out displaying the Overview
subtab. Click the Monitor subtab.

Chapter 5 - Page 4
d. Leave JVisualVM running. You can minimize it, if you want. Also, do not close the
Terminal window that started JVisualVM, because this will exit the program.
4. Run a Grinder script to simulate users accessing the new benefits application.
a. In a new Terminal window, from the current practice directory, run the script to call the
Grinder.
$> ./rungrinder.sh
b. Do not close this Terminal window. You can minimize it, if you want.
5. Return to JVisualVM tool. View how the CPU and memory usage are affected by the
Grinder script. Use the tool to help figure out what is causing these effects.

a. Return to JVisualVM on the same screen as before: the server1 WebLogic JVM, and
the Monitor tab.
Note: Notice the CPU usage spikes and the memory usage goes up dramatically.
b. Use JVisualVM to look into the CPU issue. Click the Sampler tab. Then click the CPU
button. Watch for a little while. Are there any methods in the application using a lot of
CPU?
Note: The results may not be exactly the same for you as shown, but the service()
method of BenefitsServlet is probably using a large percentage of the CPU. You
have not looked at the code of the benefits application, but based on the name of this
Servlet, the odds are that it is part of the benefits application.
c. Click the Stop button.

Chapter 5 - Page 5
d. See if you can determine what is taking so much memory. Click the Memory button
under the Sampler tab. Watch for a little while.

Note: There are always many char arrays and String instances around. But it
seems in this case a Byte array is taking a large percentage of the memory.
e. Click the Stop button again.
f. Close JVisualVM.
Tip: You can exit the tool by using File > Exit in the menu.
6. Stop the Grinder script.
a. In a new Terminal window, navigate to the current practice directory and run a script to
kill the Grinder processes.
$> ./killgrinder.sh
b. Verify it worked by looking at the Terminal window in which you started the Grinder
script. There should be a message that things were killed and the prompt should
display again.
7. Set some flags so all of Java Mission Control is accessible in the JVM running server1.
a. Access the administration console, log in, and lock the configuration.
b. Navigate to server1.
c. Click Configuration > Server Start.
d. In the Arguments text area enter:
-XX:+UnlockCommercialFeatures
-XX:+FlightRecorder
Note: This will ensure the flight recorder in Mission Control will work. Be careful to
enter the arguments exactly as shown. There are no spaces in each argument. The
XX are capital letters.
e. Click Save.
f. Activate the changes.

Chapter 5 - Page 6
g. Even though the message says that no restarts are necessary, you do need to restart
server1 for these flags to take effect. Use the admin console to force shutdown and
then restart server1.
Tip: Environment > Servers > Control. Select the check box in front of server1 and
click Shutdown > Force Shutdown Now. Click Yes. Select the check box again and
click Start. Click Yes.
h. Wait for the server to be running before continuing.
8. Investigate the same issues as before by using Java Mission Control.
a. Because you stopped and restarted server1, it will now have a different process ID.
Return to host01. In a Terminal window, navigate to the bin directory of the JVM and
use the jps command again to find the new process of server1. This time use grep to

eliminate output not needed. Make note of server1s PID.
$> ./jps v | grep server1
b. Now run the jmc command.
$> ./jmc
Note: Ignore any messages or warnings in the Terminal window.
c. When the graphical user interface comes up, under the JVM Browser tab, locate the
WebLogic Server entry with the correct PID (the number in parentheses). Expand that
entry and double-click MBean Server. Under the new WebLogic Server tab, ensure
that the General icon is selected.
Tip: If you do not see the JVM Browser tab, click Window > Show View > JVM
Browser. You can also close the Welcome tab.
d. Leave Java Mission Control (JMC) running. You can minimize it, if you want. Also, do
not close the Terminal window that started JMC, because this will exit the program.
e. Run the Grinder script again.
Tip: In a different Terminal window, from the current practice directory, run the script
rungrinder.sh again.

Chapter 5 - Page 7
f. Leave the Grinder script window open and return to JMC. Watch for a while.

Note: Notice the CPU spikes and the memory usage goes up, as before.
9. <OPTIONAL> Create and view a Java Flight Recording.
a. First, set the diagnostic volume of server1 to High, so the flight recording has more
data. Access the admin console and lock the configuration. Navigate to Environment
> Servers > server1 > Configuration > General. Use the drop-down list to set
Diagnostic Volume to High. Save and activate your changes.
b. Return to Java Mission Control on host01.
c. Below the MBean Server entry of the server1 WebLogic Server, right-click on Flight
Recorder.
d. Select Start Flight Recording.

e. In the window that opens leave all the defaults and click Finish.
f. Click the arrow next to Flight Recorder in the JVM Browser so you can see the new
recording, My Recording (the default name).
g. The default recording time is 1 minute. Once it counts down and completes the
recording it will open a new tab to display the recording.
Note: It may take a little while to finish recording and appear.
h. Click around on the various icons and tabs to view the flight recording.
10. Exit Java Mission Control.
a. Exit JMC by clicking File > Exit.
b. Close the Terminal window in which you started JMC.

Chapter 5 - Page 8
11. Stop the Grinder script again.

a. In a new Terminal window, navigate to the current practice directory and run a script to
kill the Grinder processes, killgrinder.sh.
b. Verify the Grinder has been killed.
12. Clean up.
a. Return to the admin console and set the Diagnostic Volume of server1 back to Low.
Save and activate the change.
b. Return to host01. In a Terminal window in the current practice directory, run the
cleanup script.

$> ./cleanup.sh
Note: This script calls a WLST script to undeploy the benefits application.

Chapter 5 - Page 9
Practice Solution: Troubleshooting a JVM

Chapter 5 - Page 10

Troubleshooting Servers
Chapter 6
Practices for Lesson 6: Troubleshooting Servers

Chapter 6 - Page 1

Practices Overview
In this practice, you monitor threads and create and analyze a thread dump.

Chapter 6 - Page 2
Practice 6-1: Investigating Server Problems

Scenario
Soon after the latest version of an application is deployed, users report they are unable to
access the application. You investigate.
Overview
In this practice, you configure server overload conditions, monitor threads, create a thread
dump, and analyze a thread dump.
Assumptions

You completed Practice 3-1: Harvesting Diagnostic Metrics.
Tasks
1. Modify a servers overload configuration so that the server fails if too many threads are
stuck.
Note: The default is to never put a server into the FAILED state based on stuck threads.
a. Access the administration console and lock the configuration.
b. Locate and select server1.
c. Select the Configuration > Overload tabs.
d. Update the Stuck Thread Count to 5.
e. Also update the Max Stuck Thread Time to 30.
Note: The default is 600 seconds (10 minutes).
f. Save and activate the changes.
Note: Notice that the server needs to be restarted for the changes to take effect.
g. Shut down and restart server1. Wait for server1 to be running before continuing.
2. Run the setup script to deploy the latest version of the application.
a. Access host01. Open a Terminal window and run the setup script in the current
practice directory.
$> ./setup.sh
Note:
The setup script undeploys the current version of the application with its deployment
plan.
The script also deletes the deployment plan file. If you run the script more than
once, a message displays that the file cannot be removed (because it is no
longer there).
The script deploys a new version of the application. This version of the application:
Introduces some threading issues
No longer has a deployment plan
No longer contains an application-scoped diagnostic module
The setup script also disables the system diagnostic module, by targeting it to
nothing.

Chapter 6 - Page 3
Please ignore any messages about:

An insecure protocol being used to connect to the admin server
WLContext.close() being called in a different thread
3. Run a Grinder script to simulate users accessing the application.
a. In the same Terminal window, from the current practice directory, run the script to call
the Grinder.
$> ./rungrinder.sh
b. Let the Grinder run for about a minute before continuing.
Note: Do not wait longer than two minutes or you may have to run the Grinder again.

c. Do not close the Terminal window.
d. Minimize the VNC Viewer to use again.
4. Check the status of server1 and its threads.
a. Access the admin console. Navigate to the Servers table. You should see server1 is in
the FAILED state.
Note: If is not, wait a little while and refresh the web browser.
b. Click the name of server1. Click the Monitoring > Threads tabs.
c. In the Self-Tuning Thread Pool Threads table, you should see some threads that are
stuck (the Stuck column is true). Also, you should see threads that are hoggers (the
Hogger column is true). You may have to scroll to see these threads.
Note: Hogger threads are suspicious, but have not yet been busy long enough to be
stuck. After they are busy long enough to be stuck, the hogger flag is not reset (it
remains true).
d. Click the Dump Thread Stacks button.
e. Scroll down and view some of the threads that are marked as hogger or stuck. You
should also see some threads that are blocked.
f. Notice at the top of each stack trace, it always seems to show the same method:
stcurr.DataAccess.getConnection()
Note: Because all the threads that are blocked and stuck seem to have this method in
common, that is where developers should start looking to resolve the problem.
Tip: If you do not see any stuck threads in the thread stack, you waited too long to
press the Dump Thread Stacks button. If that is the case, you need to kill the Grinder
processes and run the Grinder script again.
Use the admin console to shut down and restart server1.
Open a new Terminal window, navigate to the current practice directory and run the
script killgrinder.sh.
$> ./killgrinder.sh
Return to the window in which you ran the Grinder script and run it again.
Try this task again: access the admin console, select server1, click the Monitoring
> Threads tabs, look for stuck threads, click the Dump Thread Stacks button, and
look through the thread dump.
5. Thread dumps go to the server log file, but you might also want to save them to their own
file. Save a thread dump to a file by using the HotSpot JVM utility called jstack.
a. Return to the VNC Viewer for host01.

Chapter 6 - Page 4
b. Open a new Terminal window.

c. Navigate to where the jstack utility resides, the bin directory under the JDK.
d. Find the process ID of server1 by using the ps (process status) command:
$> ps u oracle o pid,args | grep weblogic.Server
Note:
The -u option means to show only processes owned by the users listed (oracle).
The -o option is the format desired (the pid followed by the command args).
Using the pipe followed by grep weblogic.Server means to send the output of

the ps command to grep and only show those results that contain the string
weblogic.Server (notice that Server starts with a capital S).
e. Look through the output. You should have three items. One is the admin server, one is
server1, and the last one is the grep command itself. You can tell which WebLogic
Server is which by the option within the arguments that lists the servers name:
-Dweblogic.Name=server1
f. Note the number at the start of the server1 item. That is the PID of server1.
g. In the same Terminal window, run the jstack utility with that PID. Normally it prints
the thread dump to the Terminal window. Redirect it to a file to keep it.
$> ./jstack nnnnn > /home/oracle/server1_threads.txt
Note: Replace nnnnn with the PID of server1.
h. View the thread dump file by using the gedit editor.
$> gedit /home/oracle/server1_threads.txt
i. Try searching in the editor (Search > Find or Ctrl + F) for these strings:
stuck
blocked
Note: If you do not see any threads that are stuck or blocked, it may be that too much
time has passed since the Grinder script accessed the bad application.
j. Exit the editor.
k. Close the Terminal window.
6. Look at a thread dump with a tool. You will use an open source tool called Samurai that
looks through a server log file and displays thread dumps found in the log.
a. Still on host01, open a new Terminal window, and navigate to where the Samurai JAR
file resides:
$> cd /install/samurai
b. Set up the PATH and CLASSPATH by running the setWLSEnv.sh script:
$> source /u01/app/fmw/wlserver/server/bin/setWLSEnv.sh

Chapter 6 - Page 5
c. Run the Samurai JAR file to open the GUI tool:

$> java jar samurai.jar

d. You can drag a server log file into Samurai by using the File Browser, navigating to the
log file location, and dragging it from the File Browser into the Samurai window. Or you
can use the Samurai File > Open menu options to navigate to and select the log file.
The screenshot below is using the File menu. Remember, the log file is here:
/u01/domains/tshoot/wlsadmin/servers/server1/logs/server1.log

Chapter 6 - Page 6
e. Samurai reads the log file and puts any thread dumps into the Thread Dumps tab.
Select that tab. The information displayed starts out in Samurais Table View. Scroll to
the bottom to see the legend.
f. Each column in the table represents a thread dump. You are interested in one with
Blocked threads (red blocks). The thread dump of interest will be the last column (or
the only column). Select one of the Blocked threads by clicking its red block.

Note: You may not have as many thread dumps (columns) in your table as shown in
the screen shot. In fact, you may only have one.
g. This takes you to the Sequence View, to the part of this thread that has the problem.
You may notice that even though a thread is blocked and waiting, it does not have to
be stuck, as in this example.
Note: The ACTIVE and BLOCKED notations have been highlighted in yellow in the
screenshot.

Chapter 6 - Page 7
h. To return to the Table view, scroll up and click the Table link, or click the Table View
icon near the bottom of the screen.
Note: The icon has been highlighted in yellow in the screenshot.

i. Try selecting some other red blocks (blocked threads). Can you find any that are

stuck?
Tip: Scroll down and click one of the lower numbered ones. Or select the Thread
Dump view ( ) then use the Edit > Find options. In the Find window that
opens at the bottom of the screen, enter STUCK, select Match Case, and click the Next
button.
7. Clean up.
a. Close the Samurai window.
b. Close the Terminal window where you started Samurai.
c. Stop the Grinder processes, if they are still running. Open a new Terminal window, and
navigate to the current practice directory. Run the killgrinder.sh script.
$> ./killgrinder.sh
Killed all Grinder client processes.
Note: If the Grinder client processes have already finished running, the script prints
out:
No Grinder client processes found.
d. Minimize the host01 VNC Viewer.
8. Change server1s overload configuration. Shut down and restart server1.
Note: Although there were thread issues with the new version of the contacts application,
the overload configuration numbers you entered earlier might be too low, so you will update
them before shutting down and restarting the server.
a. Access the admin console.
b. Navigate to the Servers table. You should see that server1 is still in the FAILED state.
Even if its health may now be OK (the stuck threads are gone), the server was set to
fail if too many threads got stuck. After the server fails, it stays FAILED until it is shut
down and restarted.
c. Lock the configuration.
d. Select server1.
e. Select the Configuration > Overload tabs.
f. Update the Stuck Thread Count to 30.
g. Update the Max Stuck Thread Time to 300.
Note: Both of these are more realistic values.
h. Save and activate the changes.
Note: Notice that the server needs to be restarted for the changes to take effect.

Chapter 6 - Page 8
i. Shut down and restart server1.

Note: It needs to be shut down and started again because it is in the FAILED state,
anyway.
j. Wait for it to return to the RUNNING state before continuing.
9. Return to a previous, working version of the application.
a. Return to the host01 VNC Viewer.
b. In a Terminal window, navigate to the current practice directory and run the
deploygood.sh script.
$> ./deploygood.sh

Note: This script uses the same deploy_app.py WLST script as the setup.sh
script does. However, after it undeploys the bad version of the application, it copies the
original version over the bad version in the domains apps directory and then deploys
the original.

Chapter 6 - Page 9
Practice Solution: Investigating Server Problems

Assumptions
You completed Practice 3-1: Harvesting Diagnostic Metrics.
Solution Tasks
1. Access host01 and run the solution script.

$> ./solution.sh
Note: This solution script runs the setup script of the practice, followed by a script that
deploys a good version of the contacts application. The script does not modify the
overload configuration of the server, which is not required.

Chapter 6 - Page 10

Troubleshooting Applications
Chapter 7
Practices for Lesson 7: Troubleshooting Applications

Chapter 7 - Page 1

Practices Overview
In this practice, you debug an application error.

Chapter 7 - Page 2
Practice 7-1: Investigating Application Problems

Scenario
Developers have created a new version of the contacts application. The missing add a new
contact functionality has been added. The application also now does validation of some fields
to check format. After deploying the new version of the application, however, there are problems
with it.
Overview
In this practice, you look at a Java stack trace to debug an application error. You analyze a
CLASSPATH error and correct it.

Assumptions
You completed Practice 6-1: Investigating Server Problems.
Tasks
1. Run the setup script to deploy the latest version of the contacts application.
a. Access host01. Open a Terminal Window and run the setup script in the current
practice directory.
$> ./setup.sh
Note: This script copies the new version of the contacts application to the domains
apps directory and redeploys the application.
b. Minimize the VNC viewer for later.
2. Try the application to encounter an error.
a. From the gateway machine web browser, access the contacts application by entering
the URL:
c. Click the [edit] link next to one of the contacts.
d. Edit any field and click the Update button.
e. You should see Error 500--Internal Server Error.
3. Locate the error in the server1 log file.
a. Return to host01. Use a Terminal window or the File Browser to navigate to the server1
log file. Edit the file with the gedit editor.
Tip: Remember, the log file is here:
/u01/domains/tshoot/wlsadmin/servers/server1/logs/server1.log
b. Scroll to the bottom of the log file. Start looking from the bottom up for the last error
entry. It will look something like this:
####<Oct 29, 2013 12:25:12 PM UTC> <Error> <HTTP>
<host01.example.com> <server1>
...
Root cause of ServletException.
java.lang.NoClassDefFoundError: stcurr/utils/FieldValidator

Chapter 7 - Page 3
at
stcurr.DispatchServlet.validate(DispatchServlet.java:182)
Note: The timestamp and other details may be different.
c. This Java stack trace shows an uncaught exception. The exception is
ServletException.
d. Look below that and you can see the root cause of that exception is another
exception, NoClassDefFoundError. Notice that the missing class (and its package)
is listed: stcurr/utils/FieldValidator (in Java format it is written as
stcurr.utils.FieldValidator). Also notice that the exact line number in the
class where the error occurred (DispatchServlet.java) is indicated. This lets the

developers know exactly where the problem is.
e. This error happened because a class definition was not found. This is most often a
CLASSPATH issue. A class that is being used in the code (and was there during
compile time) is missing.
4. Investigate the application CLASSPATH.
Note: After talking to the development team you discover that the missing class,
stcurr.utils.FieldValidator, was part of some utility classes that the developers
created. In development, those classes were placed in a JAR file that was named
validator.jar.
a. On host01, use the File Browser to navigate to the apps directory under the domain.
Tip: /u01/domains/tshoot/wlsadmin/apps
b. Right-click contacts.war and select Open with Archive Mounter.
c. Click contacts.war in the Places list.
d. Look for the stcurr.utils.FieldValidator class under WEB-INF/classes.
Tip: The class file (FieldValidator.class) would be in the directory
stcurr/utils. You will not find this file. The package (directory structure) is not even
there.
e. Look for any JAR files in the web application library: WEB-INF/lib.
Tip: There are none.
f. Right-click contacts.war in the Places list and select Unmount.
5. Investigate the server CLASSPATH.
Note: Because the missing class (or JAR file that contains it) is not found in the application,
you search for it in the servers CLASSPATH.
a. On host01, check the lib directory of the domain for any JAR files. Use a Terminal
window or the File Browser to navigate to:
/u01/domains/tshoot/wlsadmin/lib
Tip: There are no JAR files there.
b. A server will place its CLASSPATH in the log when it first starts. To ensure you have
that information in the server log, use the admin console to shut down and restart
server1. Wait for the servers state to be RUNNING.
c. Check for any additions to the servers CLASSPATH. Return to host01. Use a Terminal
window or the File Browser and the gedit editor. Edit the latest server1 log file,
server1.log. Scroll to the bottom of the file. Use the Search > Find menu option to
search for a string. Select Search backwards. Enter java.class.path in the
Search for field and click the Find button.

Chapter 7 - Page 4
d. Look through the CLASSPATH (java.class.path). Is validator.jar listed

anywhere? (You can also search for validator.jar, now that you know the
CLASSPATH is in the log file.)
Tip: The CLASSPATH does not contain validator.jar.
e. Close the editor.
6. Correct the CLASSPATH issue.
Note:
You talk to the development team again to tell them that their utility JAR file cannot
be found. The team leader provides you with the file. This developer also indicates
that these utilities will be used in most applications, so the JAR file should be

accessible to all the applications deployed to the WebLogic Server instances of this
domain.
There is more than one way to make the JAR file classes accessible. WebLogic
Server start scripts could be modified to update the CLASSPATH. The JAR file
could be deployed as a shared library, which other deployed applications reference.
An easy way to make the JAR file classes accessible to all code running on
WebLogic Servers in a domain is to add the JAR file to the lib directory of the
domain.
a. On host01, access a Terminal window. The utility JAR file is in the current practice
directory. Copy the file to the lib directory of the domain.
$> cp validator.jar
/u01/domains/tshoot/wlsadmin/lib/validator.jar
Note: Enter the copy command on one line.
b. Double-check that the JAR file is there.
$> ls /u01/domains/tshoot/wlsadmin/lib
readme.txt validator.jar
c. This provides the JAR file to servers running on host01. But, server2 runs on host02.
The lib directory of the domain on that machine also needs the JAR file. Use the
secure copy command to copy the JAR file to the same location on host02.
$> scp validator.jar host02:/u01/domains/tshoot/wlsadmin/lib
d. Double-check that the JAR file made it to host02. In the same Terminal window, use
ssh to list the contents of the lib directory.
$> ssh host02 "ls /u01/domains/tshoot/wlsadmin/lib"
readme.txt
validator.jar
e. The downside of using the lib directory of a domain is that servers must be restarted
for the JAR files there to be added to the CLASSPATH of the servers. Use the admin
console to shut down and restart both server1 and server2.
Note: Wait until the servers are running before continuing.
f. Try the contacts application again. In the gateway machine web browser, enter the
contacts URL:

Chapter 7 - Page 5
g. Browse the contacts. Choose a contact to edit. Change the zip code to have too few
digits. Click Update. As you can see from the message, the validation code has been
found and is now working. Correct the zip code field and click the Update button again.
h. If you want to ensure it also works on server2, try the application again by using this
URL:
i. Once you have edited contacts, close the web browser.

Chapter 7 - Page 6
Practice Solution: Investigating Application Problems

Assumptions
You completed Practice 6-1: Investigating Server Problems.
Solution Tasks
1. Access host01 and run the solution script.

$> ./solution.sh
Note: This solution script runs the setup script of the practice to deploy the new version
of the contacts application. It also copies the missing JAR file to the lib directory of
the domain on both hosts. Finally, it kills and restarts the managed servers.

Chapter 7 - Page 7

Chapter 7 - Page 8

Troubleshooting JDBC
Chapter 8
Practices for Lesson 8: Troubleshooting JDBC

Chapter 8 - Page 1

Practices Overview
In this practice, you investigate some problems with an application that uses a data source to
access the database.

Chapter 8 - Page 2
Practice 8-1: Investigating JDBC Problems

Scenario
The developers have once more created a new version of the contacts application. After
deploying the new version of the application, however, database problems arise.
Overview
In this practice, you find and resolve a data source configuration problem, troubleshoot a
database connection leak, and configure a data sources connection timeout.
Assumptions

You completed Practice 7-1: Investigating Application Problems.
Tasks
1. Run the setup script.
practice directory.
$> ./setup.sh
Note: This script copies the new version of the application to the domains apps
directory and redeploys the application. It deploys a diagnostic module to server1, if it
is not already there. It also makes some changes to the configuration of the data
source (actually it deletes the old data source and creates a replacement).
2. Find a data source configuration error.
a. Use the admin console to shut down and restart server1.
b. After the server has started, wait a moment. Access the server1 server log file on
host01. Verify that a JDBC error message is logged periodically.
####<Oct 30, 2013 3:56:18 PM UTC> <Error> <JDBC>
<host01.example.com> <server1>
<[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default
(self-tuning)'>
<<WLS Kernel>> <> <> <1383148578819>
<BEA-001112> <Test "SELECT 1 FROM DIAL" set up for pool
"datasource1" failed with exception:
"java.sql.SQLSyntaxErrorException:
ORA-00942: table or view does not exist".>
Tip: Use the Find of gedit to search for: <Error> <JDBC>
Note: The timestamp and other details may be different.
c. What is the problem with the configuration?
d. Exit the editor.
3. Fix the data source configuration error.
a. Access the admin console again and lock the configuration.
b. Navigate to the data source name datasource1.

Chapter 8 - Page 3
c. Click the Configuration > Connection Pool tabs.

d. Click Advanced.
e. Correct the error in Test Table Name.
Tip: It should be: SQL SELECT 1 FROM DUAL
f. Save and activate your changes.
g. Use the admin console to shut down and restart server1 again.
h. <OPTIONAL> After the server has started, check the server1 log file to verify that the
error message no longer appears.
Tip: One way to do that is to use gedit to edit the server1.log file on host01. Go to
the bottom of the file. Use Find. Turn off Wrap around. Turn on Search backwards.

Search for: class.path. (That will be where the server started again.) Then turn off
Search backwards. Search for: <Error> <JDBC> (you should not find it).
4. Configure data source diagnostic profiling.
a. Access the admin console and lock the configuration.
c. Click the Configuration > Diagnostic tabs.
d. Select Profile Connection Usage, Profile Connection Reservation Wait, Profile
Connection Leak, and Profile Connection Reservation Failed.
e. Set the Profile Harvest Frequency Seconds to 15.
g. Minimize the admin console to use later.
5. Investigate a connection leak by using the admin console.
a. From a Terminal window on host01, run a Grinder script to simulate users accessing
the contacts application.
$> ./rungrinder.sh
b. Let the Grinder run for a minute or so before continuing.
c. Do not close the Terminal window in which the Grinder script is running.
d. In a new Terminal window (or the File Browser), use gedit to look at the server1 log
file again.
e. Use the editor to search for:
maximum capacity of pool
f. You should find multiple messages about the maximum capacity being reached for the
data source:
Reached maximum capacity of pool "datasource1"
g. Return to the admin console and navigate to the datasource1 data source.
h. Click the Monitoring > Statistics tabs.
i. Click Customize this table.
j. Ensure that the Chosen columns contain (the order does not matter):
Server
Enabled
Active Connections Current Count
Current Capacity

Chapter 8 - Page 4
Number Available
k. Click Apply.
l. Verify that the number of active connections is the same as the capacity. As you can
see, this means that there are no connections available.
m. For example:
6. View data source diagnostic profiling data in the data source log file.
a. On host01, use a Terminal window or the File Browser to navigate to the log directory

under the server1 directory.
Tip: /u01/domains/tshoot/wlsadmin/servers/server1/logs
b. Use gedit to edit the file datasource.log.
c. Scroll to the bottom of the file. Use the menu and click Search > Find. Turn on Search
backwards. Search for: WEBLOGIC.JDBC.CONN.
d. View some of the entries you find. Here is a sample entry:
####<datasource1> <WEBLOGIC.JDBC.CONN.RESV.FAIL> <timestamp>
<weblogic.common.resourcepool.ResourceLimitException: No
resources currently available in pool datasource1 to
java.lang.Exception
at weblogic.common.
<[ACTIVE] ExecuteThread: '45' for queue:
'weblogic.kernel.Default (self-tuning)'>
Note:
Notice each entry you find starts with ####.
Next, the data source name is listed.
Following that the profile type is displayed (in this example,
WEBLOGIC.JDBC.CONN.RESV.FAIL, which corresponds to the Profile
Connection Reservation Failed data source diagnostic profiling attribute selected
earlier).
Then a timestamp is displayed (actual timestamp not shown here).
Next, an exception and its message are shown, detailing what happened.
The stack trace is displayed.
Finally, the thread is listed.
The details of the entries you view may be different.
e. Exit the editor.
7. Update the maximum capacity of the data source connection pool and retest.
a. In a new Terminal window, run the script to stop all the Grinder processes.
$> ./killgrinder.sh
b. Use the admin console to update the maximum capacity of the data source. Lock the
configuration. Increase the datasource1 connection pool Maximum Capacity to 10.
Save and activate your changes.

Chapter 8 - Page 5
Note: You conferred with the DBA and 10 is the value you were told to use.
c. Use the admin console to shut down and restart server1. Wait for it to get to the
RUNNING state.
d. Return to the host01 Terminal window in which the rungrinder.sh script was run.
Run it again.
e. Wait a moment and return to the admin console data source monitoring screen from
earlier. Refresh the web browser. Do you have the same problem? Are the current
number of active connections equal to the capacity, and no connections are available?
Note: Increasing the maximum number of connections did not help. There must be an
issue with the application. After talking with the development group, they admit there is

a problem in their JDBC code, and they deliver an updated version of the application.
f. On host01, run the script to kill the Grinder processes again.
g. Run the script that deploys the fixed version of the application.
$> ./deploygood.sh
h. Use the admin console to shut down and restart server1. When the server stops, the
connections that are still in use are closed. When the server comes back up, the
connection pool starts fresh.
Note: Alternatively, you could shut down and restart just the data source: Navigate to
datasource1. Select the Control tab. Select the check box next to server1 and click
Shutdown > Force Shutdown. Click Yes. Select the check box next to server1 again
and click Start. Click Yes.
i. Verify that all is well with the data source by returning to its Monitoring > Statistics
screen. You should see that for server1 the active connections is 0, the capacity is 1 (it
will grow as needed), and the number available is 1.
8. Disable data source diagnostic profiling.
a. Access the admin console and lock the configuration.
c. Click the Configuration > Diagnostic tabs.
d. Deselect Profile Connection Usage, Profile Connection Reservation Wait, Profile
Connection Leak, and Profile Connection Reservation Failed.
e. Set the Profile Harvest Frequency Seconds to 300.
9. Configure data source connection timeouts.
Note: You already increased the maximum capacity of the data source. Now you will
ensure bad code that does not release connections will have those connections
automatically reclaimed by the data source.
a. In the admin console, lock the configuration.
b. Edit the datasource1 connection pool again.

Chapter 8 - Page 6
c. Under Advanced, set Inactive Connection Timeout to 30.

Note: The default value of 0 means that inactive connections never time out. Now after
30 seconds of inactivity* a connection will be automatically reclaimed by the
connection pool.
* Because of the way inactive connections are reclaimed, there could be, on average, a
delay of 50% more than the configured value.
d. Save and activate your changes.

Chapter 8 - Page 7
Practice Solution: Investigating JDBC Problems
There is no solution for this practice. At the end of the practice the good version of the
application is redeployed, which is how the system starts before doing this practice. Any data
source configuration changes are not required for later practices.

Chapter 8 - Page 8

Troubleshooting Node
Manager
Chapter 9
Practices for Lesson 9: Troubleshooting Node Manager

Chapter 9 - Page 1

Practices Overview
In this practice, you investigate some problems with Node Manager.

Chapter 9 - Page 2
Practice 9-1: Investigating Node Manager Problems

Scenario
Something has happened. A server, server2, is shut down, but when you try and restart it by
using the admin console, the Node Manager on its machine is unreachable.
Overview
In this practice, you find and resolve a problem with the communication between the admin
server and Node Manager, use the WLST nmEnroll() command, and configure Node
Manager to restart failed servers that it started.

Assumptions
Tasks
1. Run the setup script.
practice directory.
$> ./setup.sh
Note: This changes something (what it does exactly is not given here as it might give
away the problem you are about to troubleshoot). This script shuts down server2. It
also kills and restarts the Node Manager on host02.
2. Attempt to start server2 by using the admin console (which uses Node Manager).
a. Use the admin console to start server2 (through Node Manager).
b. What happens? What are the messages in the admin console?
Note: You should see a message that the Node Manager on machine2 is not
reachable.
c. In the admin console, navigate to machine2. Click Monitoring > Node Manager
Status. The information here is essentially what you already know, that the Node
Manager on machine2 is not reachable. It says the Node Manager is Inactive.
3. Access host02 and investigate the Node Manager problem.
a. You ran the setup script that killed and restarted the Node Manager on host02. Check
that Node Manager is really running on host02. Open a Terminal window on host02.
Use the ps command to look for the Node Manager process.
$> ps u oracle o pid,args | grep weblogic.NodeManager
Note:
The ps (process status) command here has two options: -u oracle (show
processes that belong to the oracle user) and -o pid,args (the output should
show the process ID and the arguments).
The output should show two processes. The first process output is lengthy. It is the
Java process that is Node Manager. The output contains a call to the JVM with the
class called weblogic.NodeManager (the name of the Java class that initiates the
Node Manager code). The second process is the grep process you ran.

Chapter 9 - Page 3
This shows that Node Manager is running. You must search further to find the
problem.
b. Navigate to the utilities directory and call the script to kill Node Manager.
$> ./killnodemanager
Note: You are killing Node Manager so you can start it in a Terminal window, to be
able to look at its output as it starts. (You could also look at the Node Manager log files,
but this is easier.)
c. Now navigate to the domains bin directory. Call the script to start Node Manager in
the Terminal window.

$> cd /u01/domains/tshoot/wlsadmin/bin
$> ./startNodeManager.sh
Note: The last of the output should include:
<Plain socket listener started on port 5556,
host host02.example.com>
d. Scroll up through the output.
Note: You should see that Node Manager loaded files from the Node Manager
directory, /u01/domains/tshoot/wlsadmin/nodemanager.
The domains from nodemanager.domains
Its configuration from nodemanager.properties
e. The configuration settings are also printed out. Look through them.
f. Look through all the output of Node Manager as it started. Do you see the problem?
Tip: The problem appears to be with the domains from which this Node Manager
expects communication.
g. In the Terminal window in which Node Manager is running, use the Terminal > Set
Title menu option. Change the title to Node Manager.
h. Minimize the Node Manager Terminal window.
4. Fix the Node Manager issue.
a. The Node Manager of machine2 has the wrong domain directory in the
nodemanager.domains file for the wlsadmin domain. To verify this (or see it for the
first time if you missed it in the Node Manager output), open a new Terminal window
and navigate to the Node Manager home directory on host02. Print out the
nodemanager.domains file.
$> cd /u01/domains/tshoot/wlsadmin/nodemanager
$> cat nodemanager.domains
wlsadmin=/u01/domains/wlsadmin
b. One way to enroll a Node Manager with a domain is with the WLST command
nmEnroll(). In that same Terminal window, start WLST.
$> /u01/app/fmw/oracle_common/common/bin/wlst.sh
...
c. Connect to the admin server.
connect('weblogic',
'Welcome1',
't3://host01.example.com:7001')

Chapter 9 - Page 4
Note:
Enter the command on one line.
The prompt is not shown here (or in subsequent WLST commands) due to space
limitations.
The first parameter is the username, the second is the password, and the third is the
URL to the admin server (using the proprietary t3 protocol).
Using the Tab key to automatically fill in file names or commands does not work in
WLST. Sorry. Type carefully.
d. Run the nmEnroll() command.
nmEnroll('/u01/domains/tshoot/wlsadmin',

'/u01/domains/tshoot/wlsadmin/nodemanager')
Note:
Enter the command on one line.
The first parameter is the domain directory and the second parameter is the Node
Manager home directory.
e. Exit WLST.
exit()
f. Verify that the nodemanager.domains file has been correctly updated.
$> cat nodemanager.domains
#
# <a timestamp>
wlsadmin=/u01/domains/tshoot/wlsadmin
5. Return to the admin console and try to start server2 again.
a. Use the admin console to start server2 (which uses Node Manager).
b. Wait until the state of server2 is RUNNING.
6. Ensure that Node Manager automatically restarts server2 if it fails. Test this feature.
a. In the admin console, navigate to server2.
b. Click Configuration > Health Monitoring.
c. What is the state of the attribute Auto Restart?
d. If Auto Restart is not enabled, lock the configuration and enable it. Save and activate
your changes.
e. Test that Node Manager will automatically restart server2 if it fails, by killing the server2
process. Return to host02. In a new Terminal window, run the ps command again.
$> ps u oracle o pid,args | grep server2
Note: Find the Java process that is server2. Within the output for this process you
should see -Dweblogic.Name=server2. The process ID is the first thing displayed.
Make note of it.
f. Find the Terminal window in which the machine2 Node Manager is running. Resize
windows and move them around so that you can see both the Node Manager window
and the window in which you ran the ps command. In the ps command window, run
the kill command with the process ID of server2.
$> kill -9 nnnn
Note:
Replace nnnn with the actual process ID.

Chapter 9 - Page 5
g. Watch the Node Manager Terminal window after you enter the kill command. You
should see in the Node Manager output that Node Manager notices that server2 has
failed and restarts it.
h. Return to the admin console and verify that server2 is back in the RUNNING state.
7. Clean up by closing windows, running Node Manager in the background (as it has been),
and starting servers.
a. Return to host02.
b. Open a Terminal window, navigate to the utilities directory, and call the script to kill
Node Manager.

$> ./killnodemanager.sh
c. Find the Terminal window in which Node Manager was running and close it.
d. Start Node Manager again, in the background, by running the start Node Manager
script in the utilities directory.
$> ./startnodemanager.sh
Note: Wait for the message that Node Manager has started.
e. Close the Terminal window.
f. Exit the VNC Viewer.
g. Access the admin console. Start any managed servers that are not running.

Chapter 9 - Page 6
Practice Solution: Investigating Node Manager Problems

Chapter 9 - Page 7

Chapter 9 - Page 8

Troubleshooting Clusters
Chapter 10
Practices for Lesson 10: Troubleshooting Clusters

Chapter 10 - Page 1

Practices Overview
In these practices, you troubleshoot a cluster. First you investigate issues with the OHS cluster
proxy. Then you investigate issues with session replication.

Chapter 10 - Page 2
Practice 10-1: Investigating Proxy Problems

Scenario
It has been reported that there may be some issues with the cluster. It is not certain if the
problem is with one of the servers in the cluster or with the OHS proxy in front of the cluster.
Overview
In this practice, you investigate the OHS cluster proxy by using the proxy log file and a special
proxy query parameter.
Assumptions

Tasks
a. Access host02. Open a Terminal window, navigate to the current practice directory and
run the database setup script, initdb.sh.
$> ./initdb.sh
Note:
You run the script on host02 because that is where the database is running.
This script calls a SQL*Plus script that creates a new table in the database which a
new application requires.
b. Exit the host02 VNC Viewer.
2. Access host01 and run the setup script.
a. Access host01 and open a Terminal window.
b. Navigate to the current practice directory and run the setup script.
$> ./setup.sh
This script:
Creates two new servers in cluster1: server3 and server4
Starts the new servers
Deploys a new application to the cluster that uses session replication
Copies a new version of the OHS configuration file to the correct location
Starts (or restarts) OHS and prints out its status, by using opmnctl. The status
output does not line up well, but it should indicate that OHS is Alive and running
HTTP on port 7777.
Note: Wait for all the scripts to finish before continuing.
3. Access the admin console to monitor sessions. From host01, run the Grinder script.
a. Access the admin console. Navigate to Deployments. Select the supplies
deployment. Click Monitoring > Web Applications.
b. Click Customize this table. Place only the following fields in the Chosen column, and
then click the Apply button:

Chapter 10 - Page 3
Application
Server
Machine
State
Sessions
Sessions High
Total Sessions
Note: The order does not matter.
c. Minimize the admin console to use in a moment.

d. Access host01. From a Terminal window navigate to the current practice directory and
run the script that runs the Grinder. The Grinder script simulates customers using the
supplies application.
$> ./rungrinder.sh
e. Wait for the script to finish. It may take a minute or two.
f. Return to the admin console and refresh the browser on the supplies monitoring page.
You should see that each server has some sessions. This indicates that each server
was handling some of the requests.
Note: The order of the servers and the number of sessions for each one may be
different in your admin console. The exact numbers do not matter. Also, if you run the
Grinder script more than once, the Total Sessions will increase.
g. Open a new web browser window or tab and enter the following URL:
http://host01.example.com:7777
/supplies/?__WebLogicBridgeConfig=true
Note:
Enter the URL all together, with no spaces.
There are two underscore characters (__) in the parameter name.
This URL asks the OHS proxy to display information about each server, as well as
configuration settings and communication statistics.
h. Using the display in the web browser, observe the status of the cluster members, from
the perspective of OHS.

Chapter 10 - Page 4
i. Find the Runtime Statistics section of the output.

Note: The exact numbers do not matter. But, notice that, even though all requests
were successful, there were some exceptions. The exceptions were READ_TIMEOUT
exceptions.
j. Back on host01, use gedit to edit the file /home/oracle/proxy.log.
k. Search for the text:
READ_TIMEOUT
l. Locate a message similar to the following:
2013-11-08T15:48:37.7362+00:00 <2765138392568712>
*******Exception type [READ_TIMEOUT] (no read after 30 seconds)
raised at line 240 of ../common/Reader.cpp
2013-11-08T15:48:37.7365+00:00 <2765138392568712> caught
exception in readStatus: READ_TIMEOUT [os error=0, line 240 of
../common/Reader.cpp]: no read after 30 seconds at line 688
2013-11-08T15:48:37.7365+00:00 <2765138392568712>
PROTOCOL_ERROR: Backend Server not responding - isRecycled:1
m. When OHS times out trying to connect to a backend server, it tries again. After a
certain number of tries, it marks the backend server as bad. Try to determine which
server is having a problem. Go to the top of the log file and search for as bad:
2013-11-08T15:48:07.7192+00:00 <2765138392568712> Marking
192.0.2.11:7013 as bad

Chapter 10 - Page 5
Note:
Notice the server, 192.0.2.11:7013.
Based on the port, it is either server3 or server4. (If the port was 7011, it would
be either server1 or server2.)
Which machine is it? Make note of the IP address. (Your IP address may be
different.)
n. Close the log file.
o. To determine which machine, on the gateway machine, open a Terminal window. Print
out the contents of the hosts file to see how IP addresses are mapped to names.

$> cat /etc/hosts
p. Find the IP address of the server having issues. In this case, the IP address is mapped
to host01.example.com.
Note: The server running on host01 at port 7013 is server3.
q. Close the Terminal window on the gateway machine.
4. Tune proxy connection settings. Stop and restart OHS for the change to take effect. Rerun
the Grinder script to see if the change keeps the timeout errors from occurring.
a. It appears that server3 is having problems and timing out. Because session replication
is set up, users will fail over, and should not notice. The cause of server3 being slow
needs to be investigated, but in the meantime, perhaps changing the timeout
properties of OHS can help.
b. Access host01. Open a new Terminal window. Navigate to where the OHS proxy
configuration file resides and use gedit to edit it.
$> cd /u01/app/fmw2/instances/webtier_1/config/OHS/ohs1
$> gedit mod_wl_ohs.conf
c. Find the WLIOTimeoutSecs attribute. Change its value to 200.
d. Save and close the file.
e. Navigate to the opmnctl executable. Stop and start OHS by using opmnctl.
$> cd /u01/app/fmw2/instances/webtier_1/bin
$> ./opmnctl stopall
opmnctl stopall: stopping opmn and all managed processes...
$> ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
Note: OHS must be restarted for the configuration file changes to take effect. Wait for
the command to complete before continuing.
f. Check the status of OHS by using opmnctl.
$> ./opmnctl status
Processes in Instance: webtier_1
----------------+---------------+-----+---------
ias-component | process-type | pid | status
----------------+---------------+-----+---------
ohs1 | OHS | 3092| Alive
Note: Your PID may be different.
g. Navigate to the current practice directory and run the Grinder script again.

Chapter 10 - Page 6
$> ./rungrinder.sh
h. Wait for the script to finish.
Note: The script may take quite a while to complete (over 10 minutes) because of the
increased timeout value. (This is an opportunity for you to take a break.)
i. Return to the web browser and once again enter the URL with the special query
parameter that asks the proxy to display information about each server, as well as
configuration settings and communication statistics.
Note: Is the value of READ_TIMEOUT exceptions now 0? With the larger timeout
value, it should be.

j. Close the web browser.
5. Cleanup.
a. Access host01. Navigate to the practices directory and run the cleanup script.
$> ./cleanup.sh
This script:
Deploys a good version of the supplies application. (The version you have been
using has code in it to cause server3 to respond slowly.)
Replaces the OHS proxy configuration file with one that has more realistic timeout
values (now that a corrected version of the application has been deployed)
Stops and restarts OHS so the configuration file takes effect

Chapter 10 - Page 7
Practice Solution: Investigating Proxy Problems

Assumptions
Solution Tasks
a. Access host02. Open a Terminal window, navigate to the current practice directory and

run the database setup script, initdb.sh.
$> ./initdb.sh
Note: This script calls a SQL*Plus script that creates a new table in the database,
which a new application requires.
b. Access host01. Navigate to the current practice directory and run the solution script.
$> ./solution.sh
Note: This script calls the setup script followed by the cleanup script of the practice.

Chapter 10 - Page 8
Practice 10-2: Investigating Cluster Replication Problems

Scenario
The issue with the slow server in the cluster has been resolved. After a new version of the
supplies application is deployed, some users report that sometimes their shopping cart
mysteriously loses items. You investigate.
Overview
In this practice, you configure HTTP session debugging, view session debug messages in a
server log, observe session replication failure, and investigate the cause of that failure.

Assumptions
You completed Practice 10-1: Investigating Proxy Problems.
All four managed server instances of WebLogic Server are running. OHS is configured, up, and
running.
Tasks
$> ./setup.sh
Note: This script deploys a different version of the supplies application.
2. Configure HTTP session debugging.
b. Lock the configuration.
c. Navigate to and edit server1. Click the Debug tab.
d. Select the check box for this debug attribute: weblogic > core > cluster >
DebugReplication.
e. Click the Enable button.
f. Select the check box for this debug attribute: weblogic > servlet > internal > session
> DebugHttpSessions.
g. Click the Enable button.
Note: Select the check box for the first attribute and click the Enable button. Then
select the check box for the other attribute and click the Enable button. It sometimes
does not work if you select both check boxes at once.
h. Do not activate the changes yet.
i. Enable the same debug attributes on the remaining managed servers.
j. After all servers have the same debug settings saved, activate your changes.
3. Monitor HTTP sessions by using the admin console.
a. In the admin console, navigate to and select the supplies deployment.
b. Click the Monitoring > Web Applications tabs.
c. Minimize the web browser.
d. Access host01.

Chapter 10 - Page 9
e. In a Terminal window, navigate to the current practice directory and run the Grinder
script.
$> ./rungrinder.sh
Note: This script simulates users accessing the supplies application.
f. Wait for the script to finish.
g. Return to the admin console.
h. Refresh the supplies > Monitoring > Web Applications screen.
Question: How many sessions were created? Is each server in the cluster involved?
i. Still on the supplies application, click the Monitoring > Sessions tabs.

j. On this screen, you should see the same number of sessions created, as well as their
creation time and the time they were last accessed.
4. Reset the session data in the admin console.
a. To reset the session information, while still in the supplies application, click the Control
tab.
b. Select the check box next to the supplies web application.
c. Click Stop > Force Stop Now. When asked if you are sure, click Yes.
d. Select the check box next to the supplies web application again. Click Start >
Servicing all requests. Click Yes.
e. Return to the Monitoring > Sessions screen. The table should be empty.
5. Observe session replication (primary and secondary servers) by looking at a server log file.
a. In a new web browser window or tab, access the supplies application:
http://host01.example.com:7777/supplies
b. Click the browse the catalog link.
c. Click the [add] link next to a couple of products.
d. Scroll down to see those products in your shopping cart.
Note: Remember what products are in your shopping cart.
e. Do not close this web browser window or tab, but return to the admin console.
f. Refresh the admin console on the supplies > Monitoring > Sessions screen. You
should see a session was created on the server that is handling your request.
Note: Make note of which server is handling your request and the creation time of the
session.
g. Do not close the web browser.
h. Access the host on which your request handling server is running.
Tip: server1 and server3 are on host01. server2 and server4 are on host02.
i. Use the gedit editor. Edit the latest server log for this server.
j. Scroll to the very bottom of the log file. Use the find capability of the editor. Select
Search backwards. Search for:
Creating new session
k. You should see a <DEBUG> message that matches the server and creation time you
noted earlier. For example, with the Server server1 and the session Creation Time
of 11/11/13 1:40:46 PM UTC, the message would look like this:

Chapter 10 - Page 10
####<Nov 11, 2013 1:40:46 PM UTC> <Debug> <HttpSessions>

<host01.example.com> <server1> <[ACTIVE] ExecuteThread: '5' for
queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <>
<> <1384177246784> <BEA-000000> <[HTTP Session:100046]Creating
new session with ID:
oGFHZKJAb5qXIyKbXskVWQKaD5xTAeFkTzRtj_gqFR4EjxBZ_USV for Web
application: /supplies.>
l. Immediately after the message stating that the session was created, find a replication
debug message that provides this sessions internal replication ID. For example:
####<Nov 11, 2013 1:40:46 PM UTC> <Debug> <Replication>

<> <1384177246784> <BEA-000000>
<[roid:-8699644743353115416]
Creating primary for application key /supplies>
m. Use the Find capability of the editor again. This time search from this point forward for
that replication ID (the number after roid:). Find the message with the replication ID
that says created secondary. This should show you the backup server. For
example:
####<Nov 11, 2013 1:40:46 PM UTC> <Debug> <Replication>
<> <1384177246793> <BEA-000000>
<[roid:-8699644743353115416]
Created secondary on
6002997145995730564S:host02.example.com:[7013,7013,-1,-1,-1,-1,-
1]:wlsadmin:server4>
Note: In this example, the secondary is server4. It could be another server in your
case.
n. From what you have observed so far, it seems session replication is working. A primary
server is servicing your requests, and a secondary server has been chosen to back it
up.
6. Test session replication. If it fails, investigate the cause for replication (and therefore
failover) not working.
a. Use the admin console to force shut down the primary server. (In this example,
server1.)
b. After the server is shut down, return to the supplies application. Remembering what is
already in your shopping cart, add one more item. Scroll down and view the cart.
c. Are all the items in the cart, or just the last one?
Note: Only the last item added is in the shopping cart. This means that failover did not
work. Although it appears that session replication is set up and working, you know it did not
work because you lost your previous shopping cart items.
d. Return to the log file of your primary server. Scroll to the bottom of the file. Search
backwards for <Error>.
Note: Because you are editing a file that can change, the editor may ask if you want to
reload the file. If it does, click the Reload button.

e. You should find an error written by the <Cluster> subsystem. The error complains:
Failed to replicate a non-serializable object.
7. Analyze the serialization error.
a. Still in the log file, search backwards for:
NotSerializableException
b. When you find this exception, it should indicate the class that has the problem.
java.io.NotSerializableException: stcurr.supplies.model.Item
Note:
It appears that an instance of the Item class in the stcurr.supplies.model

package is being placed in the session, but is not Serializable. Therefore, when
the session is replicated, these Item objects cannot be streamed over to the
secondary server and are lost.
This is not a problem an administrator corrects. You inform the development group
of your findings, so their code can be corrected and a new version of the application
supplied to you to deploy.
c. Close the editor.
8. Cleanup.
a. Use the admin console to restart whichever server you shut down earlier. Wait for the
server to be running before continuing.
b. Access a Terminal window on host01.
c. Navigate to the current practice directory and run the cleanup script.
$> ./cleanup.sh
This script:
Deploys a corrected version of the application
Disables the debug flags set earlier
Stops the two new servers: server3 and server4
Deletes those servers

Practice Solution: Investigating Cluster Replication Problems

Assumptions
You completed Practice 10-1: Investigating Proxy Problems.
All four instances of WebLogic Server are running. OHS is configured, up, and running.
Solution Tasks
1. Access host01. Navigate to the practices directory and run the cleanup script.

$> ./cleanup.sh



Troubleshooting Security
Chapter 11
Practices for Lesson 11: Troubleshooting Security

Chapter 11 - Page 1

Practices Overview
In this practice, you set up the auditing provider and enable domain configuration auditing.

Chapter 11 - Page 2
Practice 11-1: Troubleshooting Changes by Using Auditing

Scenario
Someone is making changes to the domain configuration in production. These changes are
causing problems. All administrators claim that they did not make the changes. You need to
discover who is making these changes.
Overview
In this practice, you configure the default auditing provider and configure auditing of domain
configuration changes.

Assumptions
Tasks
$> ./setup.sh
Note: This script creates two new users and adds them to the Administrators
group.
2. Set up the auditing provider.
a. Access the admin console. Lock the configuration.
b. Navigate to and select the security realm, myrealm.
c. Click the Providers > Auditing tabs.
d. Click the New button.
e. Enter the name change_auditor and click OK.
Note: The Type of DefaultAuditor is already selected.
f. Select the new auditing provider.
g. Click the Provider Specific tab.
h. Move the following Active Context Handler Entries from Available to Chosen:
com.bea.contextelement.jmx.AuditProtectedArgInfo
com.bea.contextelement.jmx.ObjectName
com.bea.contextelement.jmx.OldAttributeValue
com.bea.contextelement.jmx.Parameters
com.bea.contextelement.jmx.ShortName
com.bea.contextelement.jmx.Signature
Tip: Hover over an entry for a tool tip pop-up that displays the full name of the entry.
All of the JMX elements are together.
i. Choose Custom from the Severity drop-down list.

Chapter 11 - Page 3
Note: With custom chosen, the check boxes determine which severity levels are
audited.
j. Select each of the check boxes to enable all severity levels.
k. Save and activate your changes.
Note: Notice that servers must be restarted.
l. Use the admin console to force shut down the managed servers.
m. Now use the admin console to force shut down the admin server.
n. Close the web browser.
o. Access host01. In a Terminal window, navigate to the practice utility scripts. Run the
script to start the admin server.

$> ./startadmin.sh
p. After the admin server is running, access the admin console again and use it to start
the managed servers.
Note: Wait for the managed servers to be running before continuing.
3. Set up domain configuration auditing.
a. Use the admin console to lock the configuration.
b. Select the domain, wlsadmin.
c. Click the Configuration > General tabs.
d. Click Advanced.
e. Use the Configuration Audit Type drop-down list to select Change Log and
Audit.
Note: This writes domain configuration audit messages to both the admin server log
and the audit provider log.
Note: Notice this change does not require any server restarts.
4. Make some domain configuration changes.
a. Log out of the admin console.
b. Log in to the admin console as Fred. The username is fred and the password is
Welcome1.
c. Lock the configuration, make some innocuous domain configuration change, and
activate your changes.
Tip: For example, add a new server. Make the server name something unique, such as
fredserver1, which will be easy to find in the logs. Set the Server Listen Address to
either host01.example.com or host02.example.com. Use a port not already in
use (like 7015).
d. Log out of the admin console.
e. Log in to the admin console as Wilma. The username is wilma and the password is
Welcome1.
f. Lock the configuration and make some other innocuous change.
g. Log out of the admin console.
5. View the audit information.
a. Access host01.

Chapter 11 - Page 4
b. Use the File Browser or a Terminal window to navigate to the directory where the
admin server keeps its log files.
$> cd /u01/domains/tshoot/wlsadmin/servers/AdminServer/logs
c. Use the gedit editor to open the admin servers server log, and the audit log.
$> gedit AdminServer.log DefaultAuditRecorder.log
d. In the AdminServer.log file, use the Find tool to search for the name of the domain
resource that you created (or modified) earlier as one of the other administrators. In
this example, the user fred created a new managed server called fredserver1.
You should find some audit records of the resource being created and modified. For
example:

####<Nov 14, 2013 2:04:16 PM UTC> <Info> <Configuration Audit>
<host01.example.com> <AdminServer> <[ACTIVE] ExecuteThread: '0'
for queue: 'weblogic.kernel.Default (self-tuning)'> <fred> <> <>
<1384437856692> <BEA-159900>
<USER fred CREATED wlsadmin:Name=fredserver1,Type=Server>
####<Nov 14, 2013 2:04:16 PM UTC> <Info> <Configuration Audit>
<host01.example.com> <AdminServer> <[ACTIVE] ExecuteThread: '0'
for queue: 'weblogic.kernel.Default (self-tuning)'> <fred> <> <>
<1384437856705> <BEA-159904>
<USER fred MODIFIED com.bea:Name=fredserver1,Type=Server
ATTRIBUTE ListenAddress FROM TO host01.example.com>
e. Search for the same resource in the audit log, DefaultAuditRecorder.log. The
format is different, but the information is basically the same. In this example, it shows
the user fred creating and modifying the server fredserver1. Here is the audit
record of the servers creation:
#### Audit Record Begin <Nov 14, 2013 2:04:16 PM>
<Severity =SUCCESS>
<<<Event Type = Create Configuration Audit Event>
<Subject = Subject: 2
Principal = class
weblogic.security.principal.WLSUserImpl("fred")
Principal = class
weblogic.security.principal.WLSGroupImpl("Administrators")>
<Object = wlsadmin:Name=fredserver1,Type=Server>>> Audit Record
End ####
Note: By using domain configuration auditing, you will be able to tell which
administrative user makes the next configuration change that causes a problem.
f. <OPTIONAL> Use the editor to find the audit records of the resource the other user,
wilma, created or modified.
g. Exit the editor.
6. Clean up.
a. Access host01. In a Terminal window, navigate to the current practice directory and run
the cleanup script.
$> ./cleanup.sh

Chapter 11 - Page 5
Note: This script disables domain configuration auditing and sets the auditing provider
severity level to ERROR, so less activity is audited.
b. If you want to, you can use the admin console to delete any unneeded domain
resources you created to test configuration auditing.

Chapter 11 - Page 6
Practice Solution: Troubleshooting Changes by Using Auditing
There is no solution to this practice.

Chapter 11 - Page 7
Practice 11-2: <OPTIONAL> Recovering the Lost Admin Password

Scenario
In your domain, which uses the embedded LDAP authentication provider, you only set up one
administrative user (the one that is required to be configured when the domain is created). You
forget that users password, so you can no longer start servers or change the domain
configuration.
Overview
In this practice, you recover from the loss of the main admin-level users password by running
the AdminAccount utility.

Assumptions
You completed Practice 2-1: Accessing and Setting Up the Practice Environment. (This
practice can be done at any time after the domain has been created.)
The admin server is running.
Tasks
1. Shut down the domain.
b. Use the admin console to force shut down any managed servers that are running.
c. Now use the admin console to force shut down the admin server.
d. Close the web browser.
2. Back up the master (embedded) LDAP files.
a. Access host01. Use the File Browser. Navigate to the admin servers LDAP directory.
Tip: /u01/domains/tshoot/wlsadmin/servers/AdminServer/data/
b. Right-click on the ldap directory and select Copy.
c. Right-click in the data directory and select Paste.
d. Rename the ldap (copy) directory to ldap-backup.
Tip: Right-click on the directory and select Rename.
e. Do not close the File Browser.
3. Run the AdminAccount utility to create a new, temporary admin username and password
in the DefaultAuthenticatorInit.ldift file.
a. First, find the current DefaultAuthenticatorInit.ldift file. Use the File
Browser to navigate to:
/u01/domains/tshoot/wlsadmin/security
Note: Make note of the timestamp of the DefaultAuthenticatorInit.ldift file.
b. On host01, open a Terminal window.
c. Set the environment variables.
$> source /u01/app/fmw/wlserver/server/bin/setWLSEnv.sh
d. Run the AdminAccount utility.
$> java weblogic.security.utils.AdminAccount
tempadmin Password1
/u01/domains/tshoot/wlsadmin/security

Chapter 11 - Page 8
Note: The arguments passed to this Java class are the new username, the new
password, and the location of the DefaultAuthenticatorInit.ldift file.
e. Go back to the File Browser and reload the current location (View > Reload). Look at
the timestamp of the DefaultAuthenticatorInit.ldift file, which should have
just been updated.
f. Do not close the File Browser.
4. Remove the admin servers DefaultAuthenticatormyrealmInit.initialized file.
Remove the admin servers boot identity file.
a. Use the File Browser to navigate to:
/u01/domains/tshoot/wlsadmin/servers/AdminServer/data/ldap

b. Right-click on DefaultAuthenticatormyrealmInit.initialized and select
Rename.
c. Rename the file to:
DefaultAuthenticatormyrealmInit.initialized.backup
d. Use the File Browser to navigate to:
/u01/domains/tshoot/wlsadmin/servers/AdminServer/security
e. Right-click on boot.properties and select Rename.
f. Rename the file to:
boot.properties.backup
g. Do not close the File Browser.
5. Start the admin server by using a start script and the new username and password.
a. On host01, open a new Terminal window.
b. Navigate to the bin directory of the domain and run the admin server start script.
$> cd /u01/domains/tshoot/wlsadmin/bin
$> ./startWebLogic.sh
c. When prompted for the username to boot WebLogic Server, enter: tempadmin
d. When prompted for the password to boot WebLogic Server, enter: Password1
Note: The password will not display.
e. Use the Terminal window menu to set the name of the window to AdminServer.
Tip: Terminal > Set Title
f. Minimize the AdminServer Terminal window.
6. Reset the password of the main admin user. Delete the temporary admin user.
a. Access the admin console. On the welcome page, enter the Username of tempadmin
and the Password of Password1.
b. In the Domain Structure click Security Realms. In the table, click myrealm.
c. Click the Users and Groups tab.
d. Ensure the Users subtab is selected.
e. Click the weblogic user (the main admin user that was configured when the domain
was created).
f. Click the Passwords tab.
g. Enter Welcome1 in both password fields and click Save.
h. Return to the table of users.
i. Select the check box in front of tempadmin. Click Delete. Click Yes.

Chapter 11 - Page 9
j. Log out of the admin console.

k. Log back in with weblogic and Welcome1.
l. The admin console opens.
m. Close the web browser.
7. Clean up.
a. Return to the File Browser on host01. Navigate to the security directory of
AdminServer and rename the file back to boot.properties.
b. Right-click on the boot.properties file and select Open with gedit.
c. Type over the encrypted values for username and password with weblogic and
Welcome1. The file should look like this:

password=Welcome1
username=weblogic
Note: The order of these attributes does not matter. The timestamp can be deleted, if
you want. When the values are encrypted a new timestamp is written.
d. Save the file and close the editor.
e. In the File Browser, navigate to the data/ldap directory of AdminServer and delete
the file DefaultAuthenticatormyrealmInit.initialized.backup.
Tip: Right-click on it a select Move to Trash.
f. Navigate to the data directory of AdminServer and delete the directory ldap-
backup.
g. Close the File Browser.
h. Find the Terminal window in which the admin server is running. Press Ctrl + C to force
shut down the server.
i. In that same Terminal window, navigate to the utilities directory and start the
admin server with the script.
$> ./startadmin.sh
Note: Wait for the admin server to start. If it starts successfully, the
boot.properties file (with the updated username and password) is working.
j. Close the Terminal window.
k. Exit the VNC Viewer.
l. If you are continuing to work in the domain, use the admin console to start the
managed servers.

Practice Solution: <OPTIONAL> Recovering the Lost Admin Password
There is no solution to this practice.



D83175GC10 Ag PDF

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

D83175GC10 Ag PDF

Hochgeladen von

Copyright:

Verfügbare Formate

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY.

COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Restricted Rights Notice

Oracle University and In Motion Servicios S.A. use only

U.S. GOVERNMENT RIGHTS

Technical Contributors and Reviewers

This book was published using: Oracle Tutor

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 12c: Troubleshooting Workshop Table of Contents

Practice Solution: Investigating Cluster Replication Problems .......................................................................10-13

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 12c: Troubleshooting Workshop Table of Contents

Practices for Lesson 1: Course Overview

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Oracle University and In Motion Servicios S.A. use only

Practices for Lesson 1: Overview

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 1: Course Overview

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 2: Troubleshooting: Overview

Practices for Lesson 2: Overview

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 2: Troubleshooting: Overview

Practice 2-1: Accessing and Setting Up the Practice Environment

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 2: Troubleshooting: Overview

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 2: Troubleshooting: Overview

startadmin.sh: This script, in the /practices/tshoot/utilities directory,

Oracle University and In Motion Servicios S.A. use only

Practices for Lesson 2: Troubleshooting: Overview

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 2: Troubleshooting: Overview

Practice Solution: Accessing and Setting Up the Practice

There is no solution for this practice.

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 2: Troubleshooting: Overview

Practice 2-2: <OPTIONAL> Running RDA

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 2: Troubleshooting: Overview

Step Question / Command Response

Oracle University and In Motion Servicios S.A. use only

Practices for Lesson 2: Troubleshooting: Overview

Note: The password is not displayed.

You can review the reports by ...

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 2: Troubleshooting: Overview

Oracle University and In Motion Servicios S.A. use only

Practices for Lesson 2: Troubleshooting: Overview

Oracle University and In Motion Servicios S.A. use only