Oracle Weblogic Server

Best Practices for Troubleshooting Performance

Issues - CON8307
Laurent Goldsztejn
Fusion Middleware Proactive Support
September 2014
@weblogicsupport

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracles products remains at the sole discretion of Oracle.

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Program Agenda
1

Performance issues

Best practices with server logs

Best practices with thread dumps

Deconstructing Java threads

Examples & Demo

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Performance Issues

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Performance Issues
Server hanging

The server does not respond to new requests

Requests time out

Requests take longer and longer to process

The server is no longer reported as running

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Performance Issues
Issues that could lead to server hang or other severe performance issues
Application code or WLS internal issue

Too high number of file descriptors or open sockets

JVM spending too much time in GC

Long running JDBC requests

Lock contention between NM and Managed Server
Excessive or unneeded Remote JNDI lookup
WLS Session Replication problem
Memory leak

Deadlock
Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Performance Issues
Data needed to analyze a performance issue
Server logs, -verbose:gc to print information at every collection

Stdout redirected to server log:

Servers->Logging->General->Advanced-> Redirect Stdout Logging Enabled
-Dweblogic.log.RedirectStdoutToServerLogEnabled=true
Java Flight Recordings (JFR)
Collections of general data about the application (GCs, Heap, Optimizations, Objects stats etc)
RDA collections or configuration files
Thread dumps / Heap dumps
WLDF to collect metrics, setup watch and notifications, and to define instrumentation

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with server logs

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with server logs

Errors in Server Log file

Check for Stuck

Threads

Check for Too Many

Open Files

Check for
OutOfMemory
messages

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with server logs

Example of Stuck Threads
<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue:
'weblogic.kernel.Default(self-tuning)' has been busy for "196" seconds working on the request
"Http Request: /abc/services/MyServicesImpl", which is more than the configured time
(StuckThreadMaxTime) of "180" seconds.
oracle.jdbc.driver.OracleStatement.getColumnIndex(OracleStatement.java:3288)
oracle.jdbc.driver.OracleResultSetImpl.findColumn(OracleResultSetImpl.java:1914)
oracle.jdbc.driver.OracleResultSet.getTimestamp(OracleResultSet.java:1661)
weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.getTimestamp(Unknown Source)
com.foo.sqlmap.engine.type.DateTypeHandler.getResult(DateTypeHandler.java:44)

The thread is blocked and can't return to the thread pool in a given period of time
Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with server logs

Too Many Open Files
<Critical> <Server> <BEA-002616> <Failed to listen on channel "Default" on 172.1.2.3:7001, failure
count: 1, failing for 0 seconds, java.net.SocketException: Too many open files>

Socket connections use file descriptors

A socket can remain in TIME_WAIT after they are closed to guarantee that delayed packets in the network reached
the corresponding socket
tcp_time_wait_interval on Linux (default 4 minutes)
HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters on Windows (default 1 minute)

If many connections are being opened and closed quickly then socket's in TIME_WAIT may begin to accumulate on
a system
View sockets in TIME_WAIT using netstat

Support Pattern: How To Troubleshoot Too Many Open Files Problems [ID 867492.1]

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with server logs

OutOfMemory
JVM throws OOM if it is not able allocate memory to java objects
The application may handle this error and decide to shut itself down in a safe way or decide to run ignoring this error

-verbose:gc -XX:+PrintGCDetails, -XX:+PrintGCDateStamps and -

XX:+PrintGCTimeStamps to logs details on GC cycles

Memory Leak Detectors and heap dumps can be used to identify causes for leaks in applications

Support Pattern: Troubleshooting Out of Memory and Memory Leak Problems [ID 877172.1]

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with thread dumps

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with thread dumps

Definitions

A java thread dump is a snapshot that shows what every thread in the JVM
process is doing at a particular point in time

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with thread dumps

What does a Thread Dump contain?
Date and
time the
dump was
created

Classes
Packages
Method Calls
Locks

Time Stamp

JVM Info

Stack

Threads

Vendor
Version
Build
Architecture
Mode

State
Name,
Queue
tid, nid, prio
Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with thread dumps

Different ways to take thread dumps
Remote Diagnostic
Agent (RDA)

WLS Admin Console

Unix Command-line
kill -3 <pid>

Jstack <pid>

WLST threadDump()

jcmd <pid>
print_threads
Hotspot JDK 7

Java VisualVM

Different ways to take thread dumps in WebLogic Server [ID 1098691.1]

How to Use RDA to Generate WLS Thread Dumps At Specified Intervals?
https://blogs.oracle.com/fmwinstallproactive/entry/how_to_use_rda_to

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

jcmd
<pid> Thread.print
(Java 7 mission
control)

Best practices with thread dumps

Viewing a Thread Dump

Different JVM Vendors display the data in different formats

Markers for start/end of thread dumps
Reporting of locks
Thread states and method signatures

The underlying data exposed remains the same across vendors

Thread dumps can contain lots of data
Trying to read them in a text editor can be very tricky

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with thread dumps

Free parsing tools
Tool

Description

Samurai

Light weight open source tool that extract

http://yusuke.homeip.net/samurai/en/index.ht thread dumps and GC stats from log files,
ml
and presents them in different table views
with color code for easy visual analysis
Thread Dump Analyzer (TDA)
http://java.net/projects/tda/pages/Home

Thread Logic
http://java.net/projects/threadlogic/

Swing GUI for analyzing Thread Dumps and

Heap Information
Built upon TDA by adding logic for common
patterns found in application servers

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Troubleshooting
Thread Dump Analysis Five High level steps
Look at default and custom execute threads

Identity the cause of stuck threads by looking at the last method and java class invoked
Ignore the idle threads
Check thread dumps from different managed servers to find potential relationships between
the stuck threads
In most cases, WLS internal threads, such as muxer and listen threads, can be ignored

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads

Thread hanging on SocketRead from JDBC call
<[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)' id=76 idx=0x128 tid=15140 prio=1 alive, in native, daemon
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
oracle.net.ns.Packet.receive(Unknown Source)

oracle.net.ns.DataPacket.receive(Unknown Source)
oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)

oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:420)
oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
java.sql.DriverManager.getConnection(DriverManager.java:140)
com.foo.sbm.util.ConnectionPool.createConnection(Unknown Source)

id (or tid) is the thread identifier, a unique processwide number that identifies this thread within the
JVM process.
nid is the OS-level native thread identifier. It can
be used to correlate with high CPU usage threads
identified at the OS level
idx is the thread index in the threads array.
prio refers to the thread priority, a number
inherited from the thread that created it.
alive refers to the fact that this thread has not
ended yet and is still active.
in native means that the thread uses the operating
system's native ability to manage multi-threaded
processes.
daemon indicates that this thread can't prevent
the JVM from exiting.

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads

Thread.State
State
NEW

Description
Available thread ready to process work request for any work manager

BLOCKED

Thread blocked by another and waiting for the other thread to complete its operations and
release its locks. If more than one threads attempt to acquire the monitor of a particular
object then only one thread (selected by the JVM scheduler) is granted the monitor and all
other threads are put into BLOCKED state

RUNNING

Thread actively working processing a task and therefore using some CPU

TIMED_WAITING

Thread in sleep, wait, join or park method and in a state of monitoring (waiting on monitor
condition) - waiting for another thread to perform an action for up to a specified waiting
time is in this state

WAITING

The thread is waiting by using a wait, join or park method without any maximum - waiting
indefinitely for another thread to perform a particular action is in this state
Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads

Java monitors
Mechanism used by Java to support thread synchronization
Mutual exclusion
Supported via object locks
Enables multiple threads to independently work on shared data without interfering with each other

Cooperation
Supported via the wait and notify methods of class Object
Enables threads to work together towards a common goal

Package java.util.concurrent
Utility classes commonly useful in concurrent programming

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads

Java Monitors - Example
"Thread-0" in Object.wait()[]
at java.lang.Object.wait(Native Method)
- waiting on <id2> (a a.b.c.thread.MyClass)
at java.lang.Object.wait(Unknown Source)
at a.b.c.thread.Drop.take(ClassB.java:44)
- locked <id1> (a a.b.c.thread.ClassB)
at a.b.c.thread.exec(Test.java:15)
at java.lang.Thread.exec(Unknown Source)

"Thread-1" waiting on condition[]

at java.lang.Thread.sleep(Native Method)
at x.y.z.thread.run(ClassA.java:33)
at java.lang.Thread.run(Unknown Source)

"Thread-0" is waiting for a

notification after it called
Object.wait()

"Thread-1" is sleeping on a
condition after it called
Thread.sleep

Conditions provide a means for one thread to suspend execution and wait until
notified by another thread that some state condition may now be true
Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads

Java locks

Locks are used to prevent that more than one thread accesses and works the same
object
Only one thread can own a Lock object at a time
Lock objects support a wait/notify mechanism
A lock is called Thin with little or no contention
A lock is promoted to Fat when contention increases

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads

Thread Requesting Locks
If the lock is not held by another thread then the thread becomes the new holder of the
lock until it relinquishes it
If the lock is currently held then the thread requesting it joins the blocked list of
contenders for this lock
Threads holding locks should relinquish them as soon as they stop needing them to
avoid impairing on other threads

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads

Deadlocks
Deadlocks describe a situation where two or more threads are blocked waiting for each
other (circular dependency). Neither block will ever end because each thread is waiting
for the other
Deadlocks can occur when multiple threads need the same locks, at the same time, but
obtain them in different order
Restart of the server is the only option

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Examples & Demo

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Examples and demo

Thread hanging on SocketRead from JDBC call
<[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)' id=76 idx=0x128 tid=15140 prio=1 alive, in native, daemon
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
oracle.net.ns.Packet.receive(Unknown Source)

oracle.net.ns.DataPacket.receive(Unknown Source)

The thread is stuck while waiting to

receive data back from an Oracle
Database

oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)

oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:420)

oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)

This can be due to blocked

database sessions or a
communication issue with the
backend DB server leaving the
socket idle

java.sql.DriverManager.getConnection(DriverManager.java:140)
com.foo.sbm.util.ConnectionPool.createConnection(Unknown Source)

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Examples and demo

Thread hanging on SocketRead from SMTP call
[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=76 idx=0x128 tid=15140 prio=1 alive, in native, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)[optimized]
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]

The thread is stuck waiting to

receive data from the remote
mail server. Wily introscope is
used as instrumentation. The
communication with mail server
needs to be checked

at java/io/BufferedInputStream.read(BufferedInputStream.java:235)[optimized]
^-- Holding lock: java/io/BufferedInputStream@0x291bf2b8[thin lock]
at com/sun/mail/util/LineInputStream.readLine(LineInputStream.java:75)
at com/sun/mail/smtp/SMTPTransport.readServerResponse(SMTPTransport.java:1440)

at javax/mail/Service.connect(Service.java:105)
at javax/mail/Transport.send0(Transport.java:168)

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

A lock is placed on a
BufferInputStream object to
safeguard its content

Examples and demo

Thread hanging on LDAP response
"[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x000000004c8a2000 nid=0x7559 in Object.wait()
[0x000000004296c000]
The thread is waiting to enter a
synchronized block and acquire a
Java monitor.

java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

- waiting on <0x00002aaaeced3718> (a netscape.ldap.LDAPSearchListener)

at java.lang.Object.wait(Object.java:485)
at netscape.ldap.LDAPMessageQueue.waitForMessage(LDAPMessageQueue.java:200)
- locked <0x00002aaaeced3718> (a netscape.ldap.LDAPSearchListener)

at netscape.ldap.LDAPMessageQueue.waitFirstMessage(LDAPMessageQueue.java:101)

Its wait will be over when it

receives the message back from
the LDAP server.

The health of the LDAP server

needs to be checked to
understand why its performing
slowly

at weblogic.security.providers.authentication.LDAPAtnDelegate.authenticate(LDAPAtnDelegate.java:3625)
at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:220)
at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
at java.security.AccessController.doPrivileged(Native Method)
Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Examples and demo

Idle Thread waiting for new task
"[ACTIVE] ExecuteThread: '96' for queue: 'weblogic.kernel.Default (self-tuning)'" id=313028 idx=0x1d0 tid=4686 prio=5 alive, in native,
waiting, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0x739b42e0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]
at java/lang/Object.wait(Object.java:474)[inlined]
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized]
^-- Lock released while waiting: weblogic/work/ExecuteThread@0x739b42e0[fat lock]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

The thread is waiting to be

notified of a new task assignment
Several available or free threads
are waiting on the same
ExecuteThread object which
results in a fat lock

Examples and demo

Deadlock
"[ACTIVE] ExecuteThread: '44' for queue: 'weblogic.kernel.Default (self-tuning)'" id=372 idx=0xfa tid=4257
prio=5 alive, in native, blocked, daemon
-- Blocked trying to get lock: weblogic/utils/KeyTable@0x16939098[thin lock]
at jrockit/vm/Threads.sleep(I)V(Native Method)

Thread #134 is holding a lock

that thread #44 needs and at
the same time is requesting a
lock that thread #44 has locked

...
- Holding lock: weblogic/rjvm/ResponseImpl@0x4385fc78[fat lock]
at weblogic/rjvm/ResponseImpl.getTxContext()Ljava/lang/Object;(ResponseImpl.java:100)[inlined]
...
"[ACTIVE] ExecuteThread: '134' for queue: 'weblogic.kernel.Default (self-tuning)'" id=473785 idx=0x398 tid=32638 prio=5 alive, in native, blocked,
daemon
-- Blocked trying to get lock: weblogic/rjvm/ResponseImpl@0x4385fc78[fat lock]
Both threads are currently
blocked waiting to get a lock

at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
...
^-- Holding lock: weblogic/utils/KeyTable@0x16939098[thin lock]
...
Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

Drinks. Food. Fun.

My Oracle Support Monday Mix
Tonight!
Monday, September 29
6:00 to 8:00 p.m.
ThirstyBear Brewing Company
(only block from Moscone Center)

Join us for a relaxing Happy Hour after a busy day at Oracle OpenWorld!
Take a break and unwind with your peers

Get to know the Oracle support engineers you depend on

Meet My Oracle Support executives and developers
Enjoy drinks and hors doevres

Admission is free with your Oracle OpenWorld badge

Event details at:

www.oracle.com/goto/mondaymix

Copyright 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential Internal/Restricted/Highly Restricted

35

Copyright 2014, Oracle and/or its affiliates. All rights reserved. |