You are on page 1of 6

2/19/2017 JVM:HowtoanalyzeThreadDump|JavaCodeGeeks2017

JVM:HowtoanalyzeThreadDump
Postedby:PierreHuguesCharbonneau inCoreJava March30th,2012

ThisarticlewillteachyouhowtoanalyzeaJVMThreadDump
andpinpointtherootcauseofyourproblem(s).Frommy
perspective,ThreadDumpanalysisisthemostimportantskillset
tomasterforanyindividualinvolvedinJavaEEproduction
support.Theamountofinformationthatyoucanderivefrom
ThreadDumpsnapshotsisoftenmuchbeyondthanwhatyou
canthinkof.

MygoalistosharewithyoumyknowledgeonThreadDump
analysisthatIaccumulatedoverthelast10yearse.g.hundreds
ofThreadDumpanalysiscycleswithdozensofcommon
problempatternsacrossmanyJVMversionsandJVMvendors.

Pleasebookmarkthispageandstaytunedforweeklyarticles.
PleasealsofeelfreetosharethisThreadDumptrainingplanwithyourworkcolleaguesandfriends.

Soundsgood,IreallyneedtoimprovemyThreadDumpskillssowheredowe
start?

WhatImproposingtoyouisacompleteThreadDumptrainingplan.Thefollowingitemswillbecovered.I
willalsoprovideyouwithreallifeThreadDumpexamplesthatyoucanstudyandunderstand.

1)ThreadDumpoverview&fundamentals
2)ThreadDumpgenerationtechniquesandavailabletools
3)ThreadDumpformatdifferencesbetweenSunHotSpot,IBMJREandOracleJRockit
4)ThreadStackTraceexplanationandinterpretation
5)ThreadDumpanalysisandcorrelationtechniques
6)ThreadDumpcommonproblempatterns(Threadrace,deadlock,hangingIOcalls,garbagecollection/
OutOfMemoryErrorproblems,infiniteloopingetc.)
7)ThreadDumpexamplesviareallifecasestudies

IreallyhopethisThreadDumpanalysistrainingplanwillbebeneficialforyousopleasestaytunedforweekly
updatesandarticles!

ButwhatifIstillhavequestionsorstillstrugglingtounderstandthesetraining
articles?

Dontworryandpleaseconsidermeasyourtrainer.Istronglyencourageyoutoaskmeanyquestionon
ThreadDump( remember,therearenostupidquestions
)soIproposethefollowingoptionstoyouforfree
simplychosethecommunicationmodelthatyouaremorecomfortablewith:

https://www.javacodegeeks.com/2012/03/jvmhowtoanalyzethreaddump.html 1/6
2/19/2017 JVM:HowtoanalyzeThreadDump|JavaCodeGeeks2017

1)SubmityourThreadDumprelatedquestion(s)bypostingyourcomment(s)belowthearticle( pleasefeel
freetoremainAnonymous )
2)SubmityourThreadDumpdatatotheRootCauseAnalysisforum
3)EmailmeyourThreadDumprelatedquestion(s)@phcharbonneau@hotmail.com

CanIsendyoumyThreadDumpdatafrommyproductionenvironment/
servers?

Yes,pleasefeelfreetosendmeyourgeneratedThreadDumpdataviaemailorRootCauseAnalysis
forumifyouwishtodiscusstherootcauseofyourproblem(s).ReallifeThreadDumpanalysisisalwaysthe
bestwaytolearn.

IreallyhopethatyouwillenjoyandsharethisThreadDumpanalysistrainingplan.Iwilldomyverybestto
provideyouwithqualitymaterialandanswerstoanyquestion.

BeforegoingdeeperintoThreadDumpanalysisandproblempatterns,itisveryimportantthatyou
understandthefundamentals.ThepostwillcoverthebasicsandallowyoutobetteryourJVMand
middlewareinteractionwithyourJavaEEcontainer.

JavaVMoverview

TheJavavirtualmachineisreallythefoundationofanyJavaEEplatform.Thisiswhereyourmiddlewareand
applicationsaredeployedandactive.

TheJVMprovidesthemiddlewaresoftwareandyourJava/JavaEEprogramwith:

AruntimeenvironmentforyourJava/JavaEEprogram(bytecodeformat)
Severalprogramfeaturesandutilities(IOfacilities,datastructure,Threadsmanagement,security,
monitoringetc.)
Dynamicmemoryallocationandmanagementviathegarbagecollector

YourJVMcanresideonmanyOS(Solaris,AIX,Windowsetc.)anddependingofyourphysicalserver
specifications,youcaninstall1nJVMprocessesperphysical/virtualserver.

JVMandMiddlewaresoftwareinteractions

FindbelowadiagramshowingyouahighlevelinteractionviewbetweentheJVM,middlewareand
application(s).

ThisisshowingyouatypicalandsimpleinteractiondiagrambetweentheJVM,middlewareandapplication.
Asyoucansee,theThreadsallocationforastandardJavaEEapplicationaredonemainlybetweenthe
middlewarekernelitselfandJVM( therearesomeexceptionswhenapplicationitselforsomeAPIscreate
Threadsdirectlybutthisisnotcommonandmustbedoneverycarefully ).

Also,pleasenotethatcertainThreadsaremanagedinternallywithintheJVMitselfsuchasGC(garbage
collection)Threadsinordertohandleconcurrentgarbagecollections.

https://www.javacodegeeks.com/2012/03/jvmhowtoanalyzethreaddump.html 2/6
2/19/2017 JVM:HowtoanalyzeThreadDump|JavaCodeGeeks2017

SincemostoftheThreadallocationsaredonebytheJavaEEcontainer,itisimportantthatyouunderstand
andrecognizetheThreadStackTraceandidentifyitproperlyfromtheThreadDumpdata.Thiswillallowyou
tounderstandquicklythetypeofrequestthattheJavaEEcontainerisattemptingtoexecute.

FromaThreadDumpanalysisperspective,youwilllearnhowtodifferentiatebetweenthedifferentThread
PoolsfoundfromtheJVMandidentifytherequesttype.

ThislastsectionwillprovideyouwithanoverviewofwhatisaJVMThreadDumpfortheHotSpotVMandthe
differentThreadsthatyouwillfind.DetailfortheIBMVMThreadDumpformatwillbeprovidedinthepart4.

PleasenotethatyouwillfindtheThreadDumpsampleusedforthisarticlefromtherootcauseanalysis
forum.

JVMThreadDumpwhatisit?

AJVMThreadDumpisasnapshottakenatagiventimewhichprovidesyouwithacompletelistingofall
createdJavaThreads.

EachindividualJavaThreadfoundgivesyouinformationsuchas:

ThreadnameoftenusedbymiddlewarevendorstoidentifytheThreadIdalongwithitsassociatedThread
Poolnameandstate(running,stucketc.)

Threadtype&priorityex:daemonprio=3**middlewaresoftwarestypicallycreatetheirThreadsas
daemonmeaningtheirThreadsarerunninginbackgroundprovidingservicestoitsusere.g.yourJavaEE
application**
JavaThreadIDex:tid=0x000000011e52a800**ThisistheJavaThreadIdobtainedvia
java.lang.Thread.getId()andusuallyimplementedasanautoincrementinglong1..n**
NativeThreadIDex:nid=0x251c**CrucialinformationasthisnativeThreadIdallowsyouto
correlateforexamplewhichThreadsfromanOSperspectiveareusingthemostCPUwithinyourJVMetc.**
JavaThreadStateanddetailex:waitingformonitorentry[0xfffffffea5afb000]
java.lang.Thread.State:BLOCKED(onobjectmonitor)
**AllowstoquicklylearnaboutThreadstateanditspotentialcurrentblockingcondition**
JavaThreadStackTracethisisbyfarthemostimportantdatathatyouwillfindfromtheThread
Dump.ThisisalsowhereyouwillspentmostofyouranalysistimesincetheJavaStackTraceprovidesyou
with90%oftheinformationthatyouneedinordertopinpointrootcauseofmanyproblempatterntypesas
youwilllearnlaterinthetrainingsessions

JavaHeapbreakdownstartingwithHotSpotVM1.6,youwillalsofindatthebottomoftheThread
DumpsnapshotabreakdownoftheHotSpotmemoryspacesutilizationsuchasyourJavaHeap(YoungGen,
OldGen)&PermGenspace.ThisisquiteusefulwhenexcessiveGCissuspectedasapossiblerootcauseso
youcandooutoftheboxcorrelationwithThreaddata/patternsfound

https://www.javacodegeeks.com/2012/03/jvmhowtoanalyzethreaddump.html 3/6
2/19/2017 JVM:HowtoanalyzeThreadDump|JavaCodeGeeks2017

1 Heap
2 PSYoungGentotal466944K,used
178734K[0xffffffff45c00000,
0xffffffff70800000,0xffffffff70800000)
3 edenspace233472K,76%used
[0xffffffff45c00000,0xffffffff50ab7c50,0xffffffff54000000)
4 fromspace233472K,0%used
[0xffffffff62400000,0xffffffff62400000,0xffffffff70800000)
5 tospace233472K,0%used
[0xffffffff54000000,0xffffffff54000000,0xffffffff62400000)
6 PSOldGentotal1400832K,used
1400831K[0xfffffffef0400000,
0xffffffff45c00000,0xffffffff45c00000)
7 objectspace1400832K,99%used
[0xfffffffef0400000,0xffffffff45bfffb8,0xffffffff45c00000)
8 PSPermGentotal262144K,used
248475K[0xfffffffed0400000,
0xfffffffee0400000,0xfffffffef0400000)
9 objectspace262144K,94%used
[0xfffffffed0400000,0xfffffffedf6a6f08,0xfffffffee0400000)

ThreadDumpbreakdownoverview

Inorderforyoutobetterunderstand,findbelowadiagramshowingyouavisualbreakdownofaHotSpotVM
ThreadDumpanditscommonThreadPoolsfound:

AsyoucanthereareseveralpiecesofinformationthatyoucanfindfromaHotSpotVMThreadDump.Some
ofthesepieceswillbemoreimportantthanothersdependingofyourproblempattern(problempatternswill
besimulatedandexplainedinfuturearticles).

Fornow,findbelowadetailedexplanationforeachThreadDumpsectionasperoursampleHotSpotThread
Dump:

#Fullthreaddumpidentifier
Thisisbasicallytheuniquekeywordthatyouwillfindinyourmiddleware/standalongJavastandardoutput
logonceyougenerateaThreadDump(ex:viakill3<PID>forUNIX).ThisisthebeginningoftheThread
Dumpsnapshotdata.

1 FullthreaddumpJavaHotSpot(TM)64Bit
ServerVM(20.0b11mixedmode):

#JavaEEmiddleware,thirdparty&customapplicationThreads
ThisportionisthecoreoftheThreadDumpandwhereyouwilltypicallyspendmostofyouranalysistime.
ThenumberofThreadsfoundwilldependonyourmiddlewaresoftwarethatyouuse,thirdpartylibraries
(thatmighthaveitsownThreads)andyourapplication( ifcreatinganycustomThread,whichisgenerallynot
abestpractice ).

InoursampleThreadDump,Weblogicisthemiddlewareused.StartingwithWeblogic9.2,aselftuning
ThreadPoolisusedwithuniqueidentifierweblogic.kernel.Default(selftuning)

https://www.javacodegeeks.com/2012/03/jvmhowtoanalyzethreaddump.html 4/6
2/19/2017 JVM:HowtoanalyzeThreadDump|JavaCodeGeeks2017

1 "[STANDBY]ExecuteThread:'414'forqueue:
'weblogic.kernel.Default(self
tuning)'"daemon
prio=3tid=0x000000010916a800nid=0x2613in
Object.wait()[0xfffffffe9edff000]
2 java.lang.Thread.State:WAITING(on
objectmonitor)
3 atjava.lang.Object.wait(Native
Method)
4 waitingon<0xffffffff27d44de0>
(aweblogic.work.ExecuteThread)
5 at
java.lang.Object.wait(Object.java:485)
6 at
weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:160)
7 locked<0xffffffff27d44de0>(a
weblogic.work.ExecuteThread)
8 at
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

#HotSpotVMThread
ThisisaninternalThreadmanagedbytheHotSpotVMinordertoperforminternalnativeoperations.
TypicallyyoushouldnotworryaboutthisoneunlessyouseehighCPU(viaThreadDump&prstat/native
Threadidcorrelation).

1 "VMPeriodicTaskThread"prio=3
tid=0x0000000101238800nid=0x19waitingon
condition

#HotSpotGCThread
WhenusingHotSpotparallelGC(quitecommonthesedayswhenusingmultiphysicalcoreshardware),the
HotSpotVMcreatebydefaultorasperyourJVMtuningacertain#ofGCThreads.TheseGCThreadsallow
theVMtoperformitsperiodicGCcleanupsinaparallelmanner,leadingtoanoverallreductionoftheGC
timeattheexpenseofincreasedCPUutilization.

1 "GCtaskthread#0(ParallelGC)"prio=3
tid=0x0000000100120000nid=0x3runnable
2 "GCtaskthread#1(ParallelGC)"prio=3
tid=0x0000000100131000nid=0x4runnable
3

ThisiscrucialdataaswellsincewhenfacingGCrelatedproblemssuchasexcessiveGC,memoryleaksetc,
youwillbeabletocorrelateanyhighCPUobservedfromtheOS/Javaprocess(es)withtheseThreadsusing
theirnativeidvalue(nid=0x3).Youwilllearnhowtoidentifyandconfirmthisproblemisfuturearticles.

#JNIglobalreferencescount
JNI(JavaNativeInterface)globalreferencesarebasicallyObjectreferencesfromthenativecodetoaJava
objectmanagedbytheJavagarbagecollector.Itsroleistopreventcollectionofanobjectthatisstillinuse
bynativecodebuttechnicallywithnolivereferencesintheJavacode.

https://www.javacodegeeks.com/2012/03/jvmhowtoanalyzethreaddump.html 5/6
2/19/2017 JVM:HowtoanalyzeThreadDump|JavaCodeGeeks2017

ItisalsoimportanttokeepaneyeonJNIreferencesinordertodetectJNIrelatedleaks.Thiscanhappenif
youprogramuseJNIdirectlyorusingthirdpartytoolslikemonitoringtoolswhicharepronetonativememory
leaks.

1 JNIglobalreferences:1925

#JavaHeaputilizationview
ThisdatawasaddedbacktoJDK1.6andprovidesyouwithashortandfastviewofyourHotSpotHeap.I
finditquiteusefulwhentroubleshootingGCrelatedproblemsalongwithHIGHCPUsinceyougetboth
ThreadDump&JavaHeapinasinglesnapshotallowingyoutodetermine(ortoruleout)anypressurepoint
inaparticularJavaHeapmemoryspacealongwithcurrentThreadcomputingcurrentlybeingdoneatthat
time.AsyoucanseeinoursampleThreadDump,theJavaHeapOldGenismaxedout!

1 Heap
2 PSYoungGentotal466944K,used
178734K[0xffffffff45c00000,
0xffffffff70800000,0xffffffff70800000)
3 edenspace233472K,76%used
[0xffffffff45c00000,0xffffffff50ab7c50,0xffffffff54000000)
4 fromspace233472K,0%used
[0xffffffff62400000,0xffffffff62400000,0xffffffff70800000)
5 tospace233472K,0%used
[0xffffffff54000000,0xffffffff54000000,0xffffffff62400000)
6 PSOldGentotal1400832K,used
1400831K[0xfffffffef0400000,
0xffffffff45c00000,0xffffffff45c00000)
7 objectspace1400832K,99%used
[0xfffffffef0400000,0xffffffff45bfffb8,0xffffffff45c00000)
8 PSPermGentotal262144K,used
248475K[0xfffffffed0400000,
0xfffffffee0400000,0xfffffffef0400000)
9 objectspace262144K,94%used
[0xfffffffed0400000,0xfffffffedf6a6f08,0xfffffffee0400000)

IhopethisarticlehashelpedtounderstandthebasicviewofaHotSpotVMThreadDump.Thenextarticle
willprovideyouthissameThreadDumpoverviewandbreakdownfortheIBMVM.

https://www.javacodegeeks.com/2012/03/jvmhowtoanalyzethreaddump.html 6/6