KernelShark (quick tutorial)

Steven Rostedt srostedt@redhat.com rostedt@goodmis.org

trace-cmd

binary tool to read Ftrace's bu ers

Records into a trace.dat ile or later reads Reads the trace.dat ile

!an record on big endian" read in little" and vice versa

Reads the ra# bu ers using s$lice %ill automatically mount debug s i it is not mounted

trace-cmd record

&e ault" #rites to 'trace.dat(

-e sched ls -ltr /usr > /dev/null

[root@frodo ~]# trace-cmd record disable all enable sched offset=2f2000 offset=2f4000 [root@frodo ~]# trace-cmd record plugin function disable all offset=2f2000 offset=412000 [root@frodo ~]# trace-cmd record > /dev/null plugin function_graph disable all offset=2f2000 offset=460000 [root@frodo ~]# trace-cmd record ls -ltr /usr > /dev/null plugin function_graph disable all enable sched offset=2f2000 offset=461000

-o func.dat -p function ls -ltr /usr > /dev/null

-o fgraph.dat -p function_graph ls -ltr /usr \

-o fgraph-events.dat -e sched -p function_graph \

trace-cmd re$ort

&e ault" reads rom 'trace.dat(

tas%$ trace-cmd$61 6157$120$( ==) 0$1 tas%$ trace-cmd$61 0$120$+ ==) 615!$1 615!$,$, 5&00$ tas%$ trace-cmd$61 tas%$ trace-cmd$61 615!$120$+ ==) 5&0 tas%$ trace-cmd$61 tas% trace-cmd$615 5&00$0$( ==) 0$120 tas%$ ls$615! runt tas%$ ls$615! runt

[root@frodo ~]# trace-cmd report | head - ! version = 6 cpus=2 trace-cmd-6157 [000 !"#71"5!4$ sched_stat_runtime$ trace-cmd-6157 [000 !"#71"5&1$ sched_s'itch$ *idle)-0 [000 !"#71"646$ sched_stat_'ait$ *idle)-0 [000 !"#71"64!$ sched_s'itch$ ls-615! [001 !"#71"&"4$ sched_'a%eup$ ls-615! [001 !"#71"&"5$ sched_stat_runtime$ ls-615! [001 !"#71"&"7$ sched_stat_runtime$ ls-615! [001 !"#71"&"!$ sched_s'itch$ migration.1-5&00 [001 !"#71"&41$ sched_stat_'ait$ migration.1-5&00 [001 !"#71"&42$ sched_migrate_tas%$ migration.1-5&00 [001 !"#71"&47$ sched_s'itch$ ls-615! [000 !"#714067$ sched_stat_runtime$ ls-615! [000 !"#7146"6$ sched_stat_runtime$

KernelShark

) ront end reader o the trace-cmd trace.dat ile *ra$h vie# +ist vie# Sim$le and )dvance iltering

,ooming -n

+e t click and drag to the right

,oom .ut

+e t click and drag le t

/vent -n o 0ool 0i$

*ra$h 1arkers

1arker ) and 2 3sed to calculate the deltas

1arker )

+e t mouse click

1arker 2

+e t mouse click #ith shi t key held

*ra$h !ursor

&ouble click on gra$h 1oves the list vie# to the closest event to the timestam$ on #here the cursor is. !an be used or marking location on 4ooming in and out

*ra$h 5lots

!53 5lots

colors change de$ending on #hat task is running colors change de$ending on #hat !53 the task is on sho#s #ake u$ latency (hollo# green bo6) sho#s $reem$t latency (hollo# red bo6) can also be o$ened by menu o$tion #hen mouse is over a task in the !53 $lot

0ask 5lots

+ist o 0asks to $lot

!53 5lots

+ist vie#

Search the +ist

Search by column

!ontains Full match &oes not have

*ra$h ollo#s toggle

*ra$h ollo#s toggle

Filtering

Filter out tasks Filter in tasks Filter events Filter events based on content

Scheduling events

sched7s#itch sched7#akeu$ sched7#akeu$7ne# - a task in either side is to be dis$layed" then the event #ill be dis$layed

/vent Filters

/vent Filters

)dvanced /vent Filtering

)dvanced Filtering +anguage

FILTER := EVENTS | EVENTS ':' EXPRESSION EVENTS := EVENTS ',' EVENTS | SYSTEM '/' SYSTEM := any system name EVENT := any event name EXPRESSION := EXPRESSION BOOL EXPRESSION BOOL := '""' | '||' OPER!TION := '#' EXPRESSION | LV!L$E %MP %MP := '&' | ''' | '==' | '&=' | ''=' STR%MP := '==' | '#=' | '=(' | '#(' RV!L$E := )nte*e+ | FIEL, STRV!L$E := st+)n* (-./01e 2/.te- va1/e LV!L$E := FIEL, | EXPR EXPR := FIEL, OP RV!L$E | '(' EXPR ' ' FIEL, := a 3)e1- name .3 an event OP := '4' | '5' | '6' | '/' | '''' | EVENT | SYSTEM | EVENT | '(' EXPRESSION ' ' | OPER!TION RV!L$E | LV!L$E STR%MP STRV!L$E | '#=' | FIEL, | EXPR OP EXPR '&&' | '"' | '#'

Fields not in /vents

Field not in an event evaluates the local condition to alse but not the entire condition
s78e- : 9+ev:9)- #= ; s78e- : #(9+ev:9)- == ;

evaluates to8
s78e- : F!LSE s78e- : #(F!LSE

!om$aring Strings

Strings can com$are #ith regular e6$ressions

rege6(9) 3se :; or <;

s78e-:s<)t78 : ne=t:7.mm =( >?events/@ABCD>

/vent Filters #ith )dvanced

)dding /vents a ter )dvance

&eleting )dvanced Filters

Recording

Recording

Recording

Recording

Recording

Kernel Shark

&emo<

=uestions>