AIX VIOS DiskAndAdapterQueueTuningV1.2

IBM Americas Advanced Technical Skills
AIX/VIOS Disk and Adapter IO Queue Tuning AIX/VIOS Disk and Adapter IO Queue Tuning
Dan Braden Dan Braden

IBM AIX Advanced Technical Skills IBM AIX Advanced Technical Skills
Version: 1.2 Version: 1.2
Date: July ! 2"1# Date: July ! 2"1#

$ 2"1# International Business Machines! Inc.
A%&'()*+D,+M+'TS.............................................................................................................................-
D ISCLAIMERS............................................................................................................................................5
T RADEMARKS............................................................................................................................................5
F EEDBACK.................................................................................................................................................5
V ERSIO !"DATES....................................................................................................................................5
I'T.(D/%TI('.............................................................................................................................................0
1.T1+ AIX I( STA%& A'D 2/+/+S........................................................................................................3
#.#.T$E AI% IO STACK...............................................................................................................................&
#.'.$O( IO )!E!ES (ORK AD ASSOCIATED ATTRIB!TES..........................................................................*
#.+.C$A,I, DEVICE ATTRIB!TE VAL!ES................................................................................................##
#.-.M!LTI."AT$ IO CODE AD )!E!E T!I,..........................................................................................#'
#.-.#.SDD and SDD"CM.....................................................................................................................#'
2.T((*S T( M('IT(. T1+ 2/+/+S..................................................................................................1#
'.#.T$E IOSTAT COMMAD.........................................................................................................................#-
'.'.T$E SAR COMMAD..............................................................................................................................#5
'.+.T$E FCSTAT COMMAD.........................................................................................................................#5
'.-.T$E ITERACTIVE TO"AS COMMAD.....................................................................................................#&
'.5.T$E ITERACTIVE MO COMMAD......................................................................................................#*
'./.MO RECORDI,S............................................................................................................................#0
'.&.T$E SDD"CM "CM"AT$ COMMAD....................................................................................................''
'.*.T$E SDD DATA"AT$ COMMAD...........................................................................................................'+
4.T/'I', T1+ 2/+/+S.............................................................................................................................2#
+.# C$ECK FOR SA1STORA,E "ROBLEMS FIRST........................................................................................'-
+.'$O( TO T!E T$E )!E!ES...................................................................................................................'-
+.'.#.T2nin3 hdisk 42e2e5de67h...........................................................................................................'5
+.'.'.T2nin3 8cs Device )2e2e A77ri927es...........................................................................................'&
+.+.AFTER :O!R IITIAL T!I,...............................................................................................................'0
+.-.T!I, ORDER.....................................................................................................................................'0
+.5.($AT ARE ,OOD; REASOABLE AD "OOR IO SERVICE TIMES<..........................................................'0
+.5.#.S7=ra3e Cache Mana3emen7 Al3=ri7hm E88ec7s =n IO Service Times........................................+'
+./.T!I, )!E!E SI>ES I VIO EVIROMETS......................................................................................++
+./.#.Av=idin3 VIOS O27a3es..............................................................................................................+5
+./.'.T?= S7ra7e3ies 7= Limi7 In.8li3h7 IOs 7= 7he S7=ra3e...................................................................+/
+./.+.T2nin3 vSCSI )2e2es.................................................................................................................+&
+./.-.T2nin3 "IV )2e2es..................................................................................................................+*
+./.5.T2nin3 Shared S7=ra3e "==l )2e2es...........................................................................................+0
+./.T$EORETICAL T$O!,$TS O S$ARED VS. DEDICATED RESO!RCES.......................................................-@
#.+STIMATI', A55*I%ATI(' 5+.6(.MA'%+ IM5.(V+M+'T.............................................##
A55+'DIX: .+*AT+D 5/B*I%ATI('S..............................................................................................#-
A '@#- In7erna7i=nal B2siness Machines; Inc.
"a3e '
Ta7le o8 6i9ures:
6i9ure 1 : AIX I( stack and 7asic tuna7les...................................................................................................0
6i9ure 2 : to;as :D sa<;le out;ut.................................................................................................................10
6i9ure 4 : to;as ada;ter ;anel.......................................................................................................................13
6i9ure # : to;as virtual ada;ter ;anel..........................................................................................................13
6i9ure - : interactive n<on disk I( service statistics................................................................................1
6i9ure 0 : 'M(' analy=er disk read I( latency........................................................................................1>
6i9ure 3 : 'M(' analy=er disk read I( latency over ti<e......................................................................1>
6i9ure : 'M(' analy=er disk read I(5S over ti<e...............................................................................2"
6i9ure > : 'M(' analy=er disk ?ueue @ait ti<e! over ti<e....................................................................21
6i9ure 1" A Tunin9 Situations.......................................................................................................................2-
6i9ure 11 : I(5S 8or di88erent disk technolo9ies........................................................................................2>
6i9ure 12 : Sa<;le I(5S vs. I( service ti<e 9ra;h...................................................................................4"
6i9ure 14 : eM*% SSD ;er8or<ance............................................................................................................41
6i9ure 1- A '5IV Architecture.....................................................................................................................44
6i9ure 10 A VI( I( Stack Tunin9.................................................................................................................4#
6i9ure 13 A vS%SI */' li<its.......................................................................................................................43
6i9ure 1 A Shared Stora9e 5ool %luster.....................................................................................................4
6i9ure 1> A Ada;ter and ;ort hdisk ?ueue connections............................................................................4>
6i9ure 2" A Dedicated ada;ter ;ort ?ueue slot resource...........................................................................#"
6i9ure 21 A Shared ada;ter ;ort ?ueue slot resource................................................................................#"
"a3e +
Ackno@led9e<ents
Thank B=2 7= Brian $ar7; Cim Allen; S7even Lan3; ,r=ver Davids=n; Marc Ols=n; and =7hers ?h=Dve 6r=.
vided ed2ca7i=n and1=r assis7ance =n 7his s29Eec7.
Disclaimers
(hile IBM has made everB e88=r7 7= veri8B 7he in8=rma7i=n c=n7ained in 7his 6a6er; 7his 6a6er maB s7ill
c=n7ain err=rs. IBM makes n= ?arran7ies =r re6resen7a7i=ns ?i7h res6ec7 7= 7he c=n7en7 here=8 and s6eci8i.
callB disclaim anB im6lied ?arran7ies =8 merchan7a9ili7B =r 8i7ness 8=r anB 6ar7ic2lar 62r6=se. IBM as.
s2mes n= res6=nsi9ili7B 8=r anB err=rs 7ha7 maB a66ear in 7his d=c2men7. The in8=rma7i=n c=n7ained in 7his
d=c2men7 is s29Eec7 7= chan3e ?i7h=27 anB n=7ice. IBM reserves 7he ri3h7 7= make anB s2ch chan3es ?i7h.
=27 =9li3a7i=n 7= n=7i8B anB 6ers=n =8 s2ch revisi=n =r chan3es. IBM makes n= c=mmi7men7 7= kee6 7he
in8=rma7i=n c=n7ained herein 26 7= da7e.
Trademarks
The 8=ll=?in3 7erms are re3is7ered 7rademarks =8 In7erna7i=nal B2siness Machines C=r6=ra7i=n in 7he
!ni7ed S7a7es and1=r =7her c=2n7riesF AI%; AI%1L; AI%1L Gl=3=H; IBM; IBM Gl=3=H; 6Series; T=7al S7=r.
a3e; "=?er "C.
The 8=ll=?in3 7erms are re3is7ered 7rademarks =8 In7erna7i=nal B2siness Machines C=r6=ra7i=n in 7he
!ni7ed S7a7es and1=r =7her c=2n7riesF "=?er VM; Advanced Micr=."ar7i7i=nin3; AI% 5L; AI% / Gl=3=H;
Micr= "ar7i7i=nin3; "=?er Archi7ec72re; "O(ER5; "O(ER/; "O(ER &; Red9==ks; SBs7em 6; SBs7em
65; SBs7em 6/; SBs7em 6&; SBs7em S7=ra3e.
A 82ll lis7 =8 !.S 7rademarks =?ned 9B IBM maB 9e 8=2nd a7F
h776F11???.i9m.c=m1le3al1c=6B7rade.sh7ml
!I% is a re3is7ered 7rademark in 7he !ni7ed S7a7es; =7her c=2n7ries =r 9=7h.
Feedback
"lease send c=mmen7s =r s233es7i=ns 8=r chan3es 7= d9radenI2s.i9m.c=m.
Version Updates
Versi=n #.@ . ini7ial Techd=c
"a3e -
Versi=n #.# J re=r3aniKed and 26da7ed ?i7h ne? ma7erial
Versi=n #.' J added in8=rma7i=n a9=27 limi7 =n "IV vir72al ada67ers 8=r n2m5cmd5el.
ems
"a3e 5
Introduction
This 6a6er is in7ended 8=r IBM "=?er SBs7ems c2s7=mers; 2sin3 AI%; IBM Technical Sales S6ecialis7s
and c=ns2l7an7s ?h= are in7eres7ed in learnin3 m=re a9=27 72nin3 IO 42e2es 8=r =67imal 6er8=rmance; and
h=? 7= d= i7. I8 B=2r a66lica7i=n has a disk IO 9=77leneck; 7his 6a6er ?ill hel6 B=2 eval2a7e 7he 6r=9lem
and 6=7en7iallB im6r=ve 6er8=rmance.
The 6a6er eL6lains h=? IO 42e2in3 ?=rks; and eL6lains h=? 7= 72ne 7he 42e2es 7= im6r=ve 6er8=rmance;
incl2din3 in VIO envir=nmen7s. This ?ill hel6 ens2re B=2 d=nD7 have 2nnecessarB IO 9=77lenecks a7 7hese
42e2es. Thanks 7= M==reDs la?; disk IO is 3e77in3 rela7ivelB sl=?er c=m6ared 7= 6r=cess=rs and mem=rB;
and 9ec=min3 m=re and m=re a 9=77leneck 7= 6er8=rmance. Red2cin3 IO la7encB 8r=m 7he a66lica7i=nDs
6=in7 =8 vie? im6r=ves 6er8=rmance; and 72nin3 7hese 42e2es is im6=r7an7 8=r hi3h IO"S 7hr2627. This
d=c2men7 eLamines 72nin3 7he 42e2es 8=r 7he hdisk driver and ada67er drivers; incl2din3 in VIO envir=n.
men7s. I7 d=esnD7 eLamine IO 72nin3 8r=m 7he a66lica7i=n 7= 7he hdisk driver.
This 6a6er c=n7ains 9es7 6rac7ices ?hich have 9een c=llec7ed d2rin3 7he eL7ensive 6eri=d =8 7ime 7eam c=l.
lea32es and I have s6en7 ?=rkin3 in 7he AI% envir=nmen7. I7 is 8=c2sed =n AI% versi=ns 5.+; /.# and &.#.
"a3e /
1. The AIX IO Stack and Queues
1.1. The AIX IO Stack
F=ll=?in3 is 7he IO s7ack 8r=m 7he a66lica7i=n 7= 7he diskF
6i9ure 1 : AIX I( stack and 7asic tuna7les
This 6a6er is c=ncerned ?i7h 72nin3 7he AI% disk driver and ada67er driver 42e2e siKes. =7e 7ha7 even
7h=23h 7he disk is a77ached 7= 7he ada67er; 7he hdisk driver c=de is 27iliKed 9e8=re 7he ada67er driver c=de.
S= 7his s7ack re6resen7s 7he =rder s=87?are and hard?are c=me in7= 6laB =ver 7ime as 7he IO 7raverses 7he
s7ack. The disk s29sBs7em is 7B6icallB SA a77ached; and 7he disk s29sBs7em ?ill have i7s =?n in7ernal IO
s7ack. =7e 7ha7 7his IO s7ack als= eLis7s in 7he VIOS; 7h=23h 7hereDs n= a66lica7i=n; =7her 7han 7he VIOS
82nc7i=n =8 vir72aliKin3 IOs 8r=m VIOCs and sharin3 IO ada67ers. The 8ile sBs7em laBers and VMM als=
arenD7 2sed 8=r VIO vir72aliKed IO eLce67 8=r 8ile 9acked hdisks ?hich arenD7 c=vered in 7his 6a6er.
"a3e &
1.2. Ho IO !"e"es #ork and Associated Attrib"tes
AI%Ds disk and ada67er drivers each 2se a 42e2e 7= handle IO; s6li7 in7= an in.service 42e2e; and a ?ai7
42e2e. IO re42es7s in 7he in.service 42e2e are sen7 7= 7he s7=ra3e; and 7he 42e2e sl=7 is 8reed ?hen 7he IO
is c=m6le7e GAI% 3e7s ei7her 7he da7a 8=r a read; =r 7he ackn=?led3emen7 8=r a ?ri7e; and re72rns 7he res2l7
9ack 26 7he s7ackH. IO re42es7s in 7he ?ai7 42e2e s7aB 7here 2n7il an in.service 42e2e sl=7 is 8ree; a7 ?hich
7ime 7heB are m=ved 7= 7he in.service 42e2e and sen7 7= 7he s7=ra3e. IO re42es7s in 7he in.service 42e2e
are als= called in.8li3h7 8r=m 7he 6ers6ec7ive =8 7he device driver.
The siKe =8 7he hdisk driver in.service 42e2e is s6eci8ied 9B 7he 42e2e5de67h a77ri927e; ?hile 7he siKe =8
7he ada67er driver in.service 42e2e is s6eci8ied 9B 7he n2m5cmd5elems a77ri927e. And 3enerallB in 7his 6a.
6er 7he in.service 42e2e siKe is als= called 7he 42e2e siKe; 7he 42e2e de67h; =r n2m9er =8 42e2e sl=7s.
This 6a6er ?ill als= re8er 7= 7he 7radi7i=nallB 2sed Mada67er driverN . since ada67ers ini7iallB had =ne 6=r7;
7h=23h n=? =87en have m2l7i6le 6=r7s . ?hen i7 ac72allB re8ers 7= 7he 8cs device =r s6eci8icallB an ada67er
6=r7Ds driver.
$ereDs h=? 7= sh=? 7he 8cs ada67er 6=r7 and hdisk a77ri927esF
root # lsattr -EHl fcs0
attribute value description user_settable
intr_priority 3 Interrupt priority False
lg_term_dma 0x800000 ong term !"# $rue
max_xfer_si%e 0x&00000 "aximum $ransfer 'i%e $rue
num_cmd_elems (00 "aximum )umber of *+""#)! Elements $rue
s,_fc_class ( F* *lass for Fabric $rue
root # lsattr -EHl -dis.0
attribute value description user_settable
/*" /*"0friend0vscsi /at- *ontrol "odule False
algorit-m fail_over #lgorit-m $rue
-c-ec._cmd test_unit_rdy Healt- *-ec. *ommand $rue
-c-ec._interval 10 Healt- *-ec. Interval $rue
-c-ec._mode enabled Healt- *-ec. "ode $rue
max_transfer 0x20000 "aximum $3#)'FE3 'i%e $rue
pvid 00c2c1c4b35f(6440000000000000000 /-ysical volume identifier False
7ueue_dept- 3 8ueue !E/$H $rue
reserve_policy no_reserve 3eserve /olicy $rue
A77ri927es in 9l2e are =8 in7eres7 re3ardin3 72nin3 7he 42e2es. B=7h 6hBsical and vir72al disk drives 7B6i .
callB s266=r7 c=mmand 7a33ed 42e2in3; ?hich all=?s 2s 7= have m2l7i6le in.8li3h7 IO re42es7s 7= a sin3le
hdisk. This im6r=ves 6er8=rmance in several ?aBs.
A 6hBsical disk can =nlB d= =ne IO a7 a 7ime; 927 kn=?in3 several =8 7he IO re42es7s all=?s 7he disk 7= d=
7he IOs 2sin3 an eleva7=r al3=ri7hm 7= minimiKe ac72a7=r m=vemen7 and la7encB. Vir72al disks 7B6icallB
are 9acked 9B manB 6hBsical disks; s= can d= manB IOs in 6arallel. The eleva7=r anal=3B is a66r=6ria7e
8=r individ2al 6hBsical disk drives; ?here 6assen3ers are IO re42es7s; 7he eleva7=r is 7he disk arm; and 7he
8l==rs are 7he disk 7racks. I8 7he eleva7=r ?en7 7= 7he 8l==rs in 7he =rder 7he 9277=ns ?ere 62shed G=r 7he =r.
"a3e *
der =8 IO re42es7sH; ra7her 7han 7he ?aB 7heB n=rmallB ?=rk; 7he avera3e service 7ime 8=r each 6assen3er
?=2ld 9e l=n3er. Th2s; s29mi77in3 m2l7i6le in.8li3h7 IOs 7= a disk s29sBs7em all=?s i7 7= 8i32re =27 h=?
7= 3e7 7he m=s7 7hr2627 and 8as7es7 avera3e IO service 7ime. Increasin3 7he 42e2e siKe is like increasin3 7he
6assen3er ca6aci7B =8 7he eleva7=r.
The=re7icallB; 7he maLim2m IO"S =ne mi3h7 achieve 8=r a L! isF
MaLim2m L! IO"S O 42e2e5de67h1 Gav3. IO service 7imeH
Th2s; increasin3 42e2e5de67h; and c=nse42en7lB 7he n2m9er =8 in.8li3h7 IOs; increases 6=7en7ial IO"S and
7hr2627. SimilarlBF
MaLim2m ada67er IO"S O n2m5cmd5elems1 Gav3. IO service 7imeH
The maLim2m in.8li3h7 IOs a sBs7em ?ill s29mi7 7= SA s7=ra3e is 7he smalles7 =8 7he 8=ll=?in3
The s2m =8 7he hdisk 42e2e5de67hs
The s2m =8 7he ada67er n2m5cmd5elems
The maLim2m n2m9er =8 in.8li3h7 IOs s29mi77ed 9B B=2r a66lica7i=nGsH
As IOs 7raverse 7he IO s7ack; AI% needs 7= kee6 7rack =8 7hem a7 each laBer. S= IOs are essen7iallB 42e2ed
a7 each laBer; and 2sin3 42e2e sl=7s =r 9288ers 7= kee6 7rack =8 7hem. ,enerallB; s=me n2m9er =8 in.8li3h7
IOs maB 9e iss2ed a7 each laBer and i8 7he n2m9er =8 IO re42es7s eLceeds 7ha7 n2m9er; 7heB reside in a
?ai7 42e2e 2n7il 7he re42ired res=2rce 9ec=me availa9le. S= 7here is essen7iallB an in.service 42e2e and a
?ai7 42e2e a7 each laBer; ?i7h 7he siKe =8 7he in.service 42e2e limi7in3 7he n2m9er =8 in.8li3h7 IOs.
(hile 7his 6a6er d=esnD7 eLamine IO 72nin3 7he 8r=m 7he hdisk driver 7= 7he a66lica7i=n; =ne can als= d=
IO 72nin3 a7 7he 8ile sBs7em laBer; as 8ile sBs7em 9288ers limi7 7he maLim2m n2m9er =8 in.8li3h7 IOs 8=r
each 8ile sBs7em. Als= a7 7he LVM device driver laBer; hdisk 9288ers; 6928s; limi7 7he n2m9er =8 in.8li3h7
IOs 8r=m 7ha7 laBer. The hdisks have a maLim2m n2m9er =8 in.8li3h7 IOs 7ha7Ps s6eci8ied 9B i7s
42e2e5de67h a77ri927e. And FC ada67ers als= have a maLim2m n2m9er =8 in.8li3h7 IOs s6eci8ied 9B
n2m5cmd5elems. The disk s29sBs7ems 7hemselves 42e2e IOs and individ2al 6hBsical disks can acce67
m2l7i6le IO re42es7s 927 =nlB service =ne a7 a 7ime. There are manB 42e2es ?i7hin 7he =6era7in3 sBs7em
and 7he hard?are; incl2din3 ?i7h TC"1I"; mem=rB; and even 7he 42e2e =8 c=mmands 7= 2se 7he 6r=ces.
s=rs.
$ere is =ne 7B6e =8 hdiskPs de8a2l7 a77ri927es Ga77ri927es varB acr=ss di88eren7 disk s29sBs7emsHF
# lsattr -El -dis.33
/3_.ey_value none 3eserve 9ey $rue
location ocation abel $rue
lun_id 0x55&5000000000000 ogical :nit )umber I! $rue
lun_reset_spt yes 'upport '*'I :) reset $rue
max_transfer 0x20000 )0# $rue
node_name 0x5005041300c061ab F* )ode )ame False
pvid none /-ysical volume identifier False
7_type simple 8ueuing $;/E $rue
7full_dly (0 delay in seconds for '*'I $#'9 'E$ F: $rue
"a3e 0
7ueue_dept- (0 8ueue !E/$H $rue
reserve_policy single_pat- 3eserve /olicy $rue
r,_timeout 10 3E#!0<3I$E time out value $rue
scbsy_dly (0 delay in seconds for '*'I =:'; $rue
scsi_id 0x1(04&3 '*'I I! $rue
start_timeout &80 '$#3$ unit time out value $rue
,,_name 0x5005041300cd61ab F* <orld <ide )ame False
The de8a2l7 42e2e5de67h is '@; 927 can 9e chan3ed 7= as hi3h as '5/ as sh=?n ?i7hF
# lsattr -3l -dis.33 -a 7ueue_dept-
&>>>(51 ?@&A
This indica7es 7he val2e can 9e anB?here 8r=m # 7= '5/ in incremen7s =8 #. In 3eneral 8=r device a77ri927es
7ha7 have a 2ser5se77a9le val2e =8 Tr2e; B=2 can dis6laB all=?a9le val2es 2sin3F
# lsattr -3l BdeviceC -a BattributeC
$erePs a FC ada67erPs a77ri927esF
# lsattr -El fcs0
bus_intr_lvl 15403 =us interrupt level False
bus_io_addr 0xdec00 =us I0+ address False
bus_mem_addr 0xe8020000 =us memory address False
init_lin. al I)I$ in. flags $rue
intr_priority 3 Interrupt priority False
lg_term_dma 0x800000 ong term !"# $rue
max_xfer_si%e 0x&00000 "aximum $ransfer 'i%e $rue
num_cmd_elems (00 "aximum number of *+""#)!' to 7ueue to t-e adapter $rue
pref_alpa 0x& /referred #_/# $rue
s,_fc_class ( F* *lass for Fabric $rue
C2rren7lB de8a2l7 42e2e siKes Gn2m5cmd5elemsH 8=r FC ada67ers ran3e 8r=m '@@ 7= 5@@; ?i7h maLim2m
val2es =8 '@-* =r -@0/.
The FC ada67er als= direc7lB accesses a c=n8i32ra9le am=2n7 =8 sBs7em mem=rB 7= h=ld in.8li3h7 IO da7a;
?hich als= ac7s as a 42e2e since i8 7here isnD7 s288icien7 8ree mem=rB 7= h=ld 7he da7a; 7hen 7he IO ?ill 9e
9l=cked 2n7il 7here is. This DMA mem=rB area is c=n7r=lled 9B 7he maL5L8er5siKe a77ri927e; and c=n7r=ls
7?= se6ara7e 7hin3sQ 7h2s; is a 9i7 c=m6leL. This a77ri927eDs val2e s6eci8ies 7he maL IO siKe 7ha7 7he ada67er
?ill send 7= 7he disk s29sBs7em G6r=vided 7he L!Ds maL57rans8er a77ri927e isnD7 smallerH. The siKe =8 7he
DMA mem=rB area is n=7 e42al 7= 7he maL5L8er5siKe val2e. BB de8a2l7 8=r 7he * ,9 d2al 6=r7 ada67er;
?i7h 7he de8a2l7 maL5L8er5siKe =8 @L-@@@@O'5/ KB; 7he DMA mem=rB area siKe is #/ MB. !sin3 anB
=7her all=?a9le val2e 8=r maL5L8er5siKe increases 7he mem=rB area 7= #'* MB. T= 3e7 7he 82ll 9and?id7h
=8 7he ada67er; =ne needs 7he DMA mem=rB area =8 #'* MB; 7h=23h 8=r manB c2s7=mers; 7he de8a2l7 is
s288icien7. Chan3in3 7his val2e isnD7 al?aBs 6=ssi9le; de6ends =n 7he sBs7em hard?are; and has s=me
risks disc2ssed in 7he neL7 sec7i=n.
"a3e #@
Since 7he DMA mem=rB area siKes s266=r7ed varies acr=ss ada67ers; B=2 can 2seF
# fcstat D! Bfcs#C
T= see 7he val2e =8 7he DMA mem=rB siKe =8 B=2r ada67er; i8 B=2 have a recen7 versi=n =8 AI% =r =8 7he
VIOS OS. This all=?s B=2 7= 7rB 7he al7erna7ive maL5L8er5siKe se77in3s; and see i8 7he DMA 6==l siKe
chan3es. Sam6le =27627 is sh=?n in sec7i=n '.+ =n 8cs7a7.
1.$. %han&in& De'ice Attrib"te Val"es
In 3eneral =ne chan3es device a77ri927e val2es 2sin3 7he chdev c=mmandF
# c-dev Dl BdeviceC -a BattributeCEBne, valueC
$=?ever; 7= chan3e 7hese a77ri927es 8=r hdisk and 8cs devices; 7heB m2s7 n=7 9e in 2se. Th2s; 7B6icallB
7hese val2es are chan3ed d2rin3 se726 =r 6lanned main7enance. One can make 7he chan3e in 7he ODM via
7he J" 8la3; s= 7ha7 7he chan3e 3=es in7= e88ec7 ?hen 7he sBs7em is re9==7ed =r 7he device is rec=n8i32redF
# c-dev Dl BdeviceC -a BattributeCEBne, valueC -/
Be a?are 7ha7 a87er makin3 7he chan3e 7= 7he a77ri927e in 7he ODM; 7he lsa77r c=mmand dis6laBs 7he val2e
in 7he ODM; n=7 7he val2e 9ein3 2sed 9B 7he kernel. :=2Dll need 7= kee6 7rack =8 ?ha7Ds ac72allB in 2se 9B
7he kernel 8=r 72nin3.
Chan3in3 hdisk a77ri927es is 6=ssi9le 6r=vided 7heB arenD7 in a varied =n V,; =r =6ened and accessed via
=7her a66lica7i=ns.
Ada67er a77ri927es can 9e chan3ed 6r=vided n= disks are 2sin3 7he ada67er. Ass2min3 =ne has m2l7i6le
6a7hs 7= all disks via red2ndan7 ada67ers and B=2Dre 2sin3 m2l7i.6a7h c=de c=m6lian7 ?i7h 7he AI% "CM;
7hen B=2 can dBnamicallB chan3e 7he ada67er a77ri927esF B=2 s7=6 2sin3 =ne ada67er 9B 6277in3 all disk
6a7hs 2sin3 7he ada67er in7= 7he De8ined s7a7e; make chan3es 7= 7he ada67er a77ri927es; rec=n8i32re 7he
ada67er; 7hen rec=n8i32re 7he 6a7hs 9ack in7= 7he Availa9le and ena9led s7a7e via 7he c83m3r c=mmand.
This can 9e achieved viaF
# rmdev D3l Bfcs#C
# c-dev Dl Bfcs#C -a BattributeCEBne, valueC
# cfgmgr Dl Bfcs#C
D2e 7= limi7ed mem=rB =n "CI $=s7 Brid3e G"$BH chi6s; i7Ds n=7 al?aBs 6=ssi9le 7= increase
maL5L8er5siKe 8=r all FC ada67ers. This is less =8 an iss2e =n ne?er hard?are; and =n =lder sBs7ems
?here 8e?er ada67ers share "$Bs; s= i7 is c=n8i32ra7i=n de6enden7. The =nlB ?aB 7= 8ind =27 i8 B=2 can
chan3e 7he val2e 8=r each ada67er is 7= 7rB chan3in3 i7. $=?ever; 9e a?are 7ha7 i8 B=2Dre 9==7in3 8r=m
SA and B=2r c=n8i32ra7i=n d=esnD7 have s288icien7 "$B mem=rB; 7his can ca2se 9==7 8ail2re. Lack =8
"$B mem=rB res2l7s in hdisk devices c=n8i32rin3 in7= a De8ined; ra7her 7han Availa9le s7a7e; and i8 B=2r
9==7 disk is =ne =8 7hem; 9==7 ?ill 8ail. Rec=verB inv=lves 3=in3 in7= SMS and chan3in3 7he val2e 9ack
7= 7he de8a2l7; =r res7=rin3 8r=m a mksBs9.
"a3e ##
Ass2min3 B=2 can 9==7; 8ail2re 7= c=n8i32re hdisks =r 6a7hs in7= 7he Availa9le s7a7e ?ill lead 7= err=rs in
7he err=r l=3 7ha7 l==k likeF
#=EF !"#_E33
I!E)$IFIE3F 00530E#1
G
*lassF H
$ypeF :)9)
3esource )ameF /*I!"#
>>>
!escription
:)!E$E3"I)E! E33+3
/robable *auses
';'$E" I0+ =:'
'+F$<#3E /3+H3#"
#!#/$E3
!EII*E
3ecommended #ctions
/E3F+3" /3+=E" !E$E3"I)#$I+) /3+*E!:3E'
I8 B=2 3e7 7hese err=rs; B=2Pll need 7= chan3e 7he maL5L8er5siKe 9ack 7= 7he de8a2l7 val2e.
1.(. )"lti*path IO %ode and !"e"e T"nin&
The m2l7i.6a7h IO c=de 8=r 7he s7=ra3e is cl=selB in7e3ra7ed ?i7h 7he hdisk and ada67er drivers; as i7 6icks a
6a7h 7ha7 s6eci8ies 7he h=s7 6=r7 and s7=ra3e 6=r7 7ha7 ?ill handle 7he IO. The disk vend=r s6eci8ies ?ha7
m2l7i.6a7h c=de maB 9e 2sed ?i7h 7heir s7=ra3e; and in manB cases B=2 have a ch=ice; even ?i7h IBM s7=r.
a3e
Th2s; 72nin3 7he 42e2es means B=2 need 7= kn=? a9=27 7he m2l7i.6a7h c=de; h=? i7 a88ec7s 7he 42e2es; and
?ha7 re6=r7s i7 6r=vides 7= 8acili7a7e 72nin3. This 6a6er mainlB disc2sses 72nin3 8r=m 7he 6ers6ec7ive =8
IBM AI% s266=r7ed m2l7i.6a7h c=de ?hich incl2des; 7he AI% "CM Gaka. M"IOH; SDD"CM and SDD.
=7e 7ha7 SDD"CM is c=m6lian7 ?i7h 7he M"IO archi7ec72re; as is 7he AI% "CM ?hich c=mes ?i7h AI%;
s= s7andard AI% IO and 6a7h mana3emen7 c=mmands ?=rk ?i7h 9=7h. This incl2des c=mmands s2ch as
i=s7a7; ls6a7h; ch6a7h; rm6a7h; e7c.
This 6a6er d=esnD7 c=ver OEM m2l7i.6a7h c=de 7ha7 isnD7 c=m6lian7 ?i7h 7he M"IO archi7ec72re. "lease
re8er 7= =7her vend=rDs d=c2men7a7i=n 8=r 7heir m2l7i.6a7h c=de.
1.(.1. SDD and SDD+%)
SDD is n= l=n3er s7ra7e3ic; isnD7 c=m6lian7 ?i7h 7he M"IO archi7ec72re; and 7he a27h=r rec=mmends
chan3in3 7= ei7her SDD"CM =r 7he AI% "CM de6endin3 =n ?ha7Ds s266=r7ed 8=r 7he s6eci8ic m=del =8
s7=ra3e.
"a3e #'
SDD adds a laBer =8 42e2in3; ?hile SDD"CM E2s7 ch==ses a 6a7h 8=r each IO.
SDD has a v6a7h device 8=r each l=3ical disk 6l2s an hdisk device 8=r each 6a7h 7= i7; ?hile ?i7h SD.
D"CM ?e E2s7 have an hdisk ?i7h 6a7hs 7= i7 lis7ed ?i7h 7he ls6a7h c=mmand. There is als= a d6= device
8=r SDD. $erePs 7he d6= devicePs a77ri927es 8=r =ne release =8 SDDF
# lsattr -El dpo
Enterpr_maxlun 100 "aximum :)' allo,ed for Enterprise /roducts $rue
Iirtual_maxlun 5&( "aximum :)' allo,ed for Iirtuali%ation /roducts False
persistent_resv yes 'ubsystem 'upports /ersistent 3eserve *ommand False
7dept-_enable yes 8ueue !ept- *ontrol $rue
(hen 4de67h5ena9leOBes; SDD ?ill =nlB s29mi7 42e2e5de67h IOs 7= anB 2nderlBin3 hdisk G?here
42e2e5de67h here is 7he val2e 8=r 7he 2nderlBin3 hdiskPs 42e2e5de67h a77ri927eH. (hen 4de67h5ena9leOn=;
SDD E2s7 6asses =n 7he IOs direc7lB 7= 7he hdisk driver. S= 7he di88erence is; i8 4de67h5ena9leOBes G7he de.
8a2l7H; IOs eLceedin3 7he 42e2e5de67h ?ill 42e2e a7 SDD; and i8 4de67h5ena9leOn=; 7hen IOs eLceed 7he
42e2e5de67h ?ill 42e2e in 7he hdiskPs ?ai7 42e2e. In =7her ?=rds; SDD ?i7h 4de67h5ena9leOn= and SD.
D"CM d= n=7 42e2e IOs and ins7ead E2s7 6ass 7hem 7= 7he hdisk drivers. =7e 7ha7 a7 SDD #./; i7Ps 6re8er.
a9le 7= 2se 7he da7a6a7h c=mmand 7= chan3e 4de67h5ena9le; ra7her 7han 2sin3 chdev; as 7hen i7Ps a dBnamic
chan3e; e.3.; da7a6a7h se7 4de67h disa9le ?ill se7 i7 7= n=. S=me releases =8 SDD d=nP7 incl2de SDD 42e2.
in3; and s=me d=; and s=me releases d=nP7 sh=? 7he 4de67h5ena9le a77ri927e. Ei7her check 7he man2al 8=r
B=2r versi=n =8 SDD =r 7rB 7he da7a6a7h c=mmand 7= see i8 i7 s266=r7s 72rnin3 7his 8ea72re =88.
(i7h SDD =ne can s29mi7 42e2e5de67h L R 6a7hs 7= a L!; ?hile ?i7h SDD"CM; =ne can =nlB s29mi7
42e2e5de67h IOs 7= 7he L!. Th2s; i8 B=2 s?i7ch 8r=m SDD 2sin3 - 6a7hs 7= SDD"CM; 7hen B=2Pd ?an7
7= se7 7he SDD"CM =r AI% "CM hdisksD 42e2e5de67h 7= -L 7ha7 =8 SDD hdisks 8=r an e42ivalen7 e88ec.
7ive 42e2e de67h.
"a3e #+
2. Tools to Monitor the Queues
Basic c=mmands 7= m=ni7=r 7he 42e2es incl2deF
(hen 2sin3 7he AI% "CM =r m2l7i.6a7h c=de c=m6lian7 ?i7h 7he M"IO archi7ec72reF
o i=s7a7 8=r hdisk driver 42e2es
o sar 8=r hdisk driver 42e2es
o 8cs7a7 8=r ada67er driver 42e2es
o 7=6as
o nm=n Gin7erac7ive =r via MO rec=rdin3sH
(hen 2sin3 SDD"CM
o 6cm6a7h
(hen 2sin3 SDD
o da7a6a7h
=7e B=2 ?ill s7ill 2se i=s7a7; sar; 8cs7a7; 7=6as and nm=n ?i7h SDD and SDD"CM.
2..1 The iostat %ommand
F=r AI% i=s7a7 is 7he 9asic 7==l 7= m=ni7=r 7he hdisk driver 42e2es. The i=s7a7 .D c=mmand 3enera7es =27.
627 s2ch asF
-dis.1 xferF Jtm_act bps tps bread b,rtn
2>4 (>(" &6>0 0>0 (>("
readF rps avgserv minserv maxserv timeouts fails
0>0 0>0 0>0 0>0 0 0
,riteF ,ps avgserv minserv maxserv timeouts fails
&6>0 38>6 &>& &60>( 0 0
7ueueF avgtime mintime maxtime avg,7s% avgs7s% s7full
&5>0 0>0 83>4 0>0 0>0 &31
av3?4sK . avera3e ?ai7 42e2e siKe
av3s4sK . avera3e service 42e2e siKe
av37ime . avera3e 7ime s6en7 in 7he ?ai7 42e2e in ms
s482ll J ra7e =8 IOs s29mi77ed 7= a 82ll 42e2e 6er sec=nd
The s482ll val2e has chan3ed 8r=m ini7iallB 9ein3 a c=2n7 =8 7he 7imes ?ePve s29mi77ed an IO 7= a 82ll
42e2e; 7= n=? ?here i7Ps 7he ra7e =8 IOs 6er sec=nd s29mi77ed 7= a 82ll 42e2e. The eLam6le re6=r7 sh=?s
7he 6ri=r case Ga c=2n7 =8 IOs s29mi77ed 7= a 82ll 42e2eH; ?hile ne?er releases 7B6icallB sh=? l=?er val2es
and decimal 8rac7i=ns indica7in3 a ra7e.
I7Ps nice 7ha7 i=s7a7 .D se6ara7es reads and ?ri7es; as ?e ?=2ld eL6ec7 7he IO service 7imes 7= 9e di88eren7
?hen ?e have a disk s29sBs7em ?i7h cache. This hel6s 2s eval2a7e 7he IO service 7imes m=re acc2ra7elB.
The m=s7 2se82l re6=r7 8=r 72nin3 is E2s7 r2nnin3 Si=s7a7 .DlS ?hich sh=?s s7a7is7ics since sBs7em 9==7; as.
"a3e #-
s2min3 7he sBs7em is c=n8i32red 7= c=n7in2=2slB main7ain disk IO his7=rB Gr2n R lsa77r .El sBs@; =r R
smi77B ch3sBs 7= see i8 7he i=s7a7 a77ri927e is se7 7= 7r2eH. The a27h=rPs 6re8erred i=s7a7 c=mmand 8la3s areF
# iostat -3!$l K BintervalC K B#intervalsC LL
This lis7s 7he da7a in a l=n3 8=rma7 G=ne line 6er hdiskH. The .R 8la3 rese7s minim2m1maLim2m val2es 8=r
each in7erval; 7he JT adds a 7ime s7am6; and 3enera7es =27627 7ha7 is 7== ?ide 7= dis6laB here in 7his 8=n7;
927 i7 c=n7ains 7he da7a a9=ve 8r=m i=s7a7 JD =n =ne line 8=r each hdisk. $ereDs sam6le =27627F
# iostat -3!$l
'ystem configurationF lcpuE&( drivesE3 pat-sE8 vdis.sE2
!is.sF xfers read ,rite 7ueue time
-------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------
Jtm bps tps bread b,rtn rps avg min max time fail ,ps avg min max time fail avg min max avg avg serv
act serv serv serv outs serv serv serv outs time time time ,7s% s7s% 7full
-dis.0 0>5 2>19 0>4 &>(9 3>29 0>& &>( 0>& 30>0 0 0 0>1 8>3 &>( 16>2 0 0 &8>& 0>0 &2(>2 0>0 0>0 0>( 01F(0F52
-dis.& 0>0 (5>09 (>6 (5>09 5(>& (>6 0>2 0>& (&8>6 0 0 0>0 5>( 0>8 (28>2 0 0 0>0 0>0 0>0 0>0 0>0 0>0 01F(0F52
-dis.( 0>0 3>09 0>2 3>09 0>0 0>2 0>( 0>& &62>& 0 0 0>0 0>0 0>0 0>0 0 0 0>& 0>0 &62>& 0>0 0>0 0>2 01F(0F52
Me7rics in 9l2e are 7he main =nes eLamined 8=r 72nin3 7he 42e2es.
2..2 The sar %ommand
The sar .d c=mmand 3enera7es =27627 s2ch asF
&1F50F56 device Jbusy av7ue r@,0s 9bs0s av,ait avserv
&1F5&F00 -dis.& 0 0>0 0 0 0>0 0>0
-dis.0 0 0>0 0 0 0>0 0>0
av?ai7 J avera3e 7ime s6en7 in 7he ?ai7 42e2e in ms
avserv J avera3e IO service 7ime in ms
av42e J avera3e n2m9er =8 IOs in 7he ?ai7 42e2e
This c=mmand d=esnD7 dis7in32ish 9e7?een read and ?ri7e service 7imes; 2nlike i=s7a7.
2..$ The ,cstat %ommand
F=r 8i9re channel GFCH ada67er 42e2es; 7he 8cs7a7 c=mmand is 2sed 7= l==k 8=r 9l=cked IOs in 7he ada67erPs
42e2es; e.3.F
# fcstat fcs0
FI=3E *H#))E '$#$I'$I*' 3E/+3$F fcs0
>>>
'econds 'ince ast 3esetF 456432(
>>>
F* '*'I #dapter !river Information
)o !"# 3esource *ountF 0
)o #dapter Elements *ountF &02828
)o *ommand 3esource *ountF &36&5618
>>>
"a3e #5
The val2e = C=mmand Res=2rce C=2n7 is 7he n2m9er =8 7imes; since 7he ada67er ?as c=n8i32red G7B6i .
callB 9==7 7imeH; 7ha7 an IO ?as 7em6=rarilB 9l=cked ?ai7in3 8=r res=2rces; s2ch as an ada67er 9288er d2e
7= a n2m5cmd5elems a77ri927e val2e 7ha7 is 7== l=?. =n.Ker= val2es indica7e 7ha7 increasin3
n2m5cmd5elems maB hel6 im6r=ve IO service 7imes. O8 c=2rse i8 7he val2e incremen7s sl=?lB; 7hen 7he
im6r=vemen7 maB 9e verB small; ?hile 42icklB incremen7in3 val2es means 72nin3 is m=re likelB 7= have a
meas2ra9le im6r=vemen7 in 6er8=rmance. One can calc2la7e 7he ra7e =8 9l=cked IOs as 7he 9l=cked c=2n7
divided 9B 7he sec=nds since las7 rese7. $ere ?eDre 3e77in3 n= c=mmand res=2rce 9l=cks a7 a ra7e =8 a9=27
' 6er sec=nd. Ass2min3 ?e red2ce la7encB 9B # ms 9B 72nin3 7he 42e2e; =verall savin3s in an h=2r ?=2ld
9eF
' 9l=cks1s L # ms19l=ck L +/@@ s1hr O &.' sec each h=2r
S= 7his isnD7 a si3ni8ican7 9=77leneck and maB n=7 9e ?=r7h ?=rrBin3 a9=27. TB6icallB 7he am=2n7 =8 7ime
?ai7in3 8=r a 9288er 7= 8ree is verB small. (e are ?ai7in3 8=r 7he s7=ra3e 7= c=m6le7e neL7 in.8li3h7 IO.
The 8cs7a7 has recen7lB 9een enhanced ?i7h 7he JD 8la3 ?hich 6r=d2ces =27627 as 8=ll=?sF
# fcstat -! fcs0
FI=3E *H#))E '$#$I'$I*' 3E/+3$F fcs0
!evice $ypeF 8Hb /*I Express !ual /ort F* #dapter ?df&000f&&2&08a03A
?adapter0pciex0df&000f&&2&08a0A
>>>
<orld <ide )ode )ameF 0x(0000&(0F#0=6E!1
<orld <ide /ort )ameF 0x&0000060F#0=6E!1
>>>
/ort 'peed ?supportedAF 8 H=I$
/ort 'peed ?runningAF 2 H=I$
/ort F* I!F 0x1a0000
/ort $ypeF Fabric
'econds 'ince ast 3esetF 456432(
$ransmit 'tatistics 3eceive 'tatistics
------------------- ------------------
FramesF &886(3&8 &1&006(4
<ordsF 11(&25045( (313430188
>>>
I0+ !"# pool si%eF 0x&000000
F* '*'I #dapter !river 8ueue 'tatistics
)umber of active commandsF 0
Hig- ,ater mar. of active commandsF &80
)umber of pending commandsF 0
Hig- ,ater mar. of pending commandsF 6&
)umber of commands in t-e #dapter !river Held off 7ueueF 0
Hig- ,ater mar. of number of commands in t-e #dapter !river Held off 7ueueF 0
F* '*'I /rotocol !river 8ueue 'tatistics
"a3e #/
)umber of pending commandsF (
Hig- ,ater mar. of pending commandsF 48
>>>
)o #dapter Elements *ountF 0
)o *ommand 3esource *ountF &4&&643
>>>
#dapter Effective max transfer valueF 0x&00000
These s7a7is7ics are es6eciallB 2se82l in 72nin3 7he ada67er 42e2es. I7ems hi3hli3h7ed in 9l2e sh=? 7he s7a.
7is7ics =8 in7eres7 and =7her 2se82l in8=rma7i=n incl2din3 7he ada67er 7B6e; (("; s266=r7ed s6eed; 7he
r2nnin3 s6eed; 7he siKe =8 7he DMA 6==l mem=rB area; and 7he maL5L8er5siKe val2e c2rren7lB 2sed 9B 7he
kernel.
2..( The Interacti'e topas %ommand
S7ar7in3 7he in7erac7ive 7=6as c=mmand ?i7hF
# topas -!
Or E2s7 6ressin3 D ?hile in 7=6as ?ill Bield 7his disk de7ail 6anelF
Fi32re ' . 7=6as .D sam6le =27627
KB.R O KB read
ART O Avera3e Read IO service Time
KB.( O KB (ri77en
A(T O Avera3e (ri7e IO service Time
A)( O Avera3e )2e2e (ai7 O avera3e 7ime ?ai7in3 in 7he 42e2e
A)D O Avera3e )2e2e De67h O avera3e n2m9er =8 IOs in 7he ?ai7 42e2e
This c=mmand; like i=s7a7; re6=r7s 9=7h read and ?ri7e IO service 7imes; and 42e2e ?ai7 7ime as ?ell.
"ressin3 7he d keB ?hile in 7he disk de7ail 6anel; ?ill al7erna7e 8r=m 7he disk de7ail 6anel 7= 7he ada67er
6anelF
"a3e #&
Fi32re + . 7=6as ada67er 6anel
"ressin3 d 7hen v ?ill sh=? 7he vir72al ada67er 6anelF
Fi32re - . 7=6as vir72al ada67er 6anel
=7e 7ha7 7he 6er8=rmance me7rics re8er 7= avera3e val2es 8=r 7he L!s a77ached 7= 7he ada67er. Th2s 7he
A)( and A)D re8er 7= ?ai7 7ime 8=r 7he L!s in 7heir hdisk driver 42e2es; n=7 ?ai7 7ime in 7he ada67er
42e2e.
2..- The Interacti'e nmon %ommand
One en7ers 7he in7erac7ive nm=n 7==l viaF
# nmon
"ressin3 D ?ill cBcle 7hr2 8=2r di88eren7 disk screens; 7he 8irs7 sh=?in3 each diskDs 7hr2627; 7he sec=nd
sh=?in3 7he disk siKe; n2m9er =8 6a7hs and an ada67er c=nnec7ed 7= 7he disk; 7he 7hird screen sh=?n 9e.
l=?; and 7he 8=2r7h screen sh=?s disk 7hr2627 s7a7is7ics ?i7h a 3ra6h indica7=r =8 7hr2627. $ereDs 7he 7hird
screen sh=?in3 IO service 7imesF
"a3e #*
Fi32re 5 . in7erac7ive nm=n disk IO service s7a7is7ics
(ai7 O avera3e ?ai7 7ime in 7he 42e2e in ms
Serv) SiKe O Avera3e service 42e2e siKe
(ai7) SiKe O Avera3e ?ai7 42e2e siKe
Serv) F2ll O 2m9er =8 IO re42es7s sen7 7= a 82ll 42e2e 8=r 7he in7erval
I8 B=2Dve a l=7 =8 hdisks; B=2 can 6ress 7he M.N s29c=mmand 7= E2s7 sh=? 7he 92sB disks.
2... /)O/ 0ecordin&s
One can se726 MO rec=rdin3s 8r=m smi7 viaF
# smitty topas -C 'tart )e, 3ecording -C 'tart local recording -C nmon
$=?ever; 7he smi7 6anel d=esnD7 lis7 7he =67i=n ?e need 7= 3e7 disk IO service 7imes; s6eci8icallB 7he Jd
=67i=n 7= c=llec7 disk IO service and ?ai7 7imes. Th2s; =ne needs 7= 2se 7he c=mmand line ?i7h 7he nm=n
c=mmand 7= c=llec7 and re6=r7 7hese s7a7is7ics. $ereDs =ne se7 =8 =67i=ns 8=r c=llec7in3 7he da7aF
# nmon -#df9")+/I;M -, 2 -s BintervalC -c Bnumber of intervalsC
The keB =67i=ns here 8r=m a disk vie?6=in7 incl2deF
.d c=llec7 and re6=r7 IO service 7ime and ?ai7 7ime s7a7is7ics
.T incl2des 7he FC ada67er sec7i=n G?hich als= meas2res "IV 7ra88ic =n VIOS FC ada67ersH
.V incl2des 7he disk v=l2me 3r=26 sec7i=n
T= 3e7 7he rec=rdin3 7hr2 7he MO AnalBKer 7==l Ga s6readshee7 7==l 7ha7 r2ns =n "Cs and 3enera7es 6er.
8=rmance 3ra6hs; =7her =27627; and is availa9le via 7he in7erne7H; i7Ds rec=mmended 7= kee6 7he n2m9er =8
in7ervals less 7han +@@. Sam6les =8 7he main re6=r7s =8 in7eres7 8=r IO 42e2e 72nin3 8=ll=?F
"a3e #0
Disk Read Service Time msec/!er d"## 1$/2%/1&
0
2
4
6
8
10
12
14
h
d
i
s
k
1
h
d
i
s
k
0
h
d
i
s
k
2
Avg. WAvg. Max.
Fi32re / . MO analBKer disk read IO la7encB
Disk Read Service Time msec/!er d"## 1$/2%/1&
0
2
4
6
8
10
12
14
1
7
:
3
9
1
7
:
4
0
1
7
:
4
0
1
7
:
4
0
1
7
:
4
1
1
7
:
4
1
1
7
:
4
1
1
7
:
4
2
1
7
:
4
2
1
7
:
4
2
1
7
:
4
3
1
7
:
4
3
1
7
:
4
3
1
7
:
4
4
1
7
:
4
4
1
7
:
4
4
1
7
:
4
5
1
7
:
4
5
1
7
:
4
5
1
7
:
4
6
1
7
:
4
6
1
7
:
4
6
1
7
:
4
7
1
7
:
4
7
1
7
:
4
7
1
7
:
4
8
1
7
:
4
8
1
7
:
4
8
1
7
:
4
9
1
7
:
4
9
hdisk1 hdisk0 hdisk2
Fi32re & . MO analBKer disk read IO la7encB =ver 7ime
"a3e '@
Disk IO Reads 'er second d"## 1$/2%/1&
0
10
20
30
40
50
60
70
1
7
:
3
9
1
7
:
4
0
1
7
:
4
0
1
7
:
4
0
1
7
:
4
1
1
7
:
4
1
1
7
:
4
1
1
7
:
4
2
1
7
:
4
2
1
7
:
4
2
1
7
:
4
3
1
7
:
4
3
1
7
:
4
3
1
7
:
4
4
1
7
:
4
4
1
7
:
4
4
1
7
:
4
5
1
7
:
4
5
1
7
:
4
5
1
7
:
4
6
1
7
:
4
6
1
7
:
4
6
1
7
:
4
7
1
7
:
4
7
1
7
:
4
7
1
7
:
4
8
1
7
:
4
8
1
7
:
4
8
1
7
:
4
9
1
7
:
4
9
T
h
o
u
s
a
n
d
s
Fi32re * . MO analBKer disk read IO"S =ver 7ime
Similar re6=r7s eLis7 8=r ?ri7e service 7imes Gn=7 sh=?nH. Als= 6r=vided are 3ra6hs =8 avera3e ?ai7 7ime in
7he hdisk driver 42e2e 8=r IOsF

Disk (ait Queue Time msec/!er d"## 1$/2%/1&
0
10
20
30
40
50
60
70
80
90
h
d
i
s
k
0
h
d
i
s
k
2
h
d
i
s
k
1
Avg. WAvg. Max.
Fi32re 0 . MO analBKer disk 42e2e ?ai7 7ime
"a3e '#
Disk (ait Queue Time msec/!er d"## 1$/2%/1&
0
10
20
30
40
50
60
70
80
90
1
7
:
3
9
1
7
:
4
0
1
7
:
4
0
1
7
:
4
0
1
7
:
4
1
1
7
:
4
1
1
7
:
4
1
1
7
:
4
2
1
7
:
4
2
1
7
:
4
2
1
7
:
4
3
1
7
:
4
3
1
7
:
4
3
1
7
:
4
4
1
7
:
4
4
1
7
:
4
4
1
7
:
4
5
1
7
:
4
5
1
7
:
4
5
1
7
:
4
6
1
7
:
4
6
1
7
:
4
6
1
7
:
4
7
1
7
:
4
7
1
7
:
4
7
1
7
:
4
8
1
7
:
4
8
1
7
:
4
8
1
7
:
4
9
1
7
:
4
9
Fi32re 0 . MO analBKer disk 42e2e ?ai7 7ime; =ver 7ime
The ra? da7a is als= availa9le in case i7Ds di88ic2l7 7= read 7he 3ra6hs; e.3.; 7he avera3e disk ?ai7 7ime 8=r
hdisk@ G7he nearlB invisi9le verB narr=? 9l2e 9ar in 8i32re 0; is #.+ ms.H. MO rec=rdin3s have 7he
9ene8i7 7ha7 B=2 can 42icklB see ?hen 7he hdisk 42e2e is =ver8l=?in3; 7he d2ra7i=n; and h=? l=n3 7he IOs
are ?ai7in3 in 7he 42e2e.
2..1 The SDD+%) pcmpath %ommand
SDD"CM 6r=vides 7he S6cm6a7h 42erB devs7a7sS and S6cm6a7h 42erB ada67s7a7sS c=mmands 7= sh=?
hdisk and ada67er 42e2e s7a7is7ics. :=2 can re8er 7= 7he SDD1SDD"CM man2al 8=r sBn7aL; =67i=ns; eL.
6lana7i=ns =8 all 7he 8ields and manB =7her 2se82l c=mmands incl2din3 7he a9ili7B 7= disa9le all 6a7hs 7= a
s6eci8ic h=s71s7=ra3e 6=r7 ?i7h =ne c=mmand. $erePs 7he =27627 =8 M6cm6a7h 42erB devs7a7sN 8=r =ne
hdiskF
!evice #F 0 !EII*E )#"EF -dis.(
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
$otal 3ead $otal <rite #ctive 3ead #ctive <rite "aximum
I0+F (600450& 3034146 & 0 20
'E*$+3F 161&(20&5 &&0210510 8 0 (0280
$ransfer 'i%eF BE 5&( BE 2. BE &19 BE 129 C 129
(&266 &0684034 &886(0&0 &335568 806031
and herePs =27627 8=r =ne ada67er 8r=m M6cm6a7h 42erB ada67s7a7sN
"a3e ''
#dapter #F 0
EEEEEEEEEEEEE
I0+F 236160333 (24(1(5& 4 0 (58
'E*$+3F &0685&532 610&34&8( 108 0 &081(5
$ere; ?ePre mainlB in7eres7ed in 7he MaLim2m 8ield. F=r 7he hdisks; 7he maLim2m val2e re6resen7s 7he
maLim2m in.8li3h7 IOs sen7 7= 7he L!; and 7his ?=nD7 eLceed 42e2e5de67h. F=r ada67ers; 7he maLim2m
val2e re6resen7s 7he maLim2m n2m9er =8 IOs s29mi77ed 7= 7he ada67er =ver 7ime; and 7his can eLceed
n2m5cmd5elems. Th2s; 7he ada67er maLim2m 7ells 2s 7he val2e ?e sh=2ld assi3n 7= n2m5cmd5elems.
2..2 The SDD datapath %ommand
SDD 6r=vides 7he da7a6a7h c=mmand; ?hich is verB similar 7= 7he 6cm6a7h c=mmand ?here ?e 2se 7he
=27627 =8 Mda7a6a7h 42erB devs7a7sN and Mda7a6a7h 42erB ada67s7a7sN 7= eLamine device s7a7is7ics.
"a3e '+
&. Tunin) the Queues
$.1 %heck ,or SA/3Stora&e +roblems First
Be8=re 72nin3 7he 42e2es; B=2 sh=2ld make s2re 7here arenD7 6r=9lems ?i7h 7he SA =r 7he s7=ra3e inhi9i7.
in3 6er8=rmance. FiLin3 7hese 6r=9lems can res=lve 42e2e ?ai7 iss2es; as re8lec7ed in 7his 8=rm2laF
MaL L! IO"S O 42e2e5de67h 1 Uav3 IO service 7imeV
I8 ?e red2ce 7he IO service 7ime 9B 8iLin3 7he SA1s7=ra3e; maL IO"S ?ill increase; and IOs ?=nD7 9e 2s.
in3 7he 42e2e sl=7 as l=n3; red2cin3 2sa3e =8 7he 42e2e sl=7s; and red2cin3 ?ai7 7ime in 7he 42e2e. Fas7er
IOs 8r=m 7he s7=ra3e can 8iL B=2r 42e2e ?ai7 6r=9lem ?i7h=27 adE2s7in3 7he 42e2e siKes.
The 8cs7a7 c=mmand is 2sed 7= eLamine FC link heal7h ?here i7 sh=?s 7he 8=ll=?in3 s7a7is7icsF
# fcstat fcs0
FI=3E *H#))E '$#$I'$I*' 3E/+3$F fcs0
G
'econds 'ince ast 3esetF 554(40
G
I/ *ountF 0
)+' *ountF 0
Error FramesF 0
!umped FramesF 0
in. Failure *ountF 0
oss of 'ync *ountF 26
oss of 'ignalF 0
/rimitive 'e7 /rotocol Error *ountF 0
Invalid $x <ord *ountF 240
Invalid *3* *ountF 0
(e eL6ec7 a cer7ain am=2n7 =8 s=me =8 7hese err=rs G7B6icallB l=ss =8 sBnc and invalid T% ?=rdsH =ver
7ime. S= a 3==d ?aB 7= eLamine 7hem is 7= c=m6are 7he c=2n7s 8=r 7he ada67ers =n 7he L"AR; and 7= in.
ves7i3a7e anB 7ha7 have si3ni8ican7lB m=re err=rs 7han 7he res7 =8 7he 6=r7s.
$.2 Ho to T"ne the !"e"es
Firs7; =ne sh=2ld n=7 indiscrimina7elB increase 7hese val2es. I7Ps 6=ssi9le 7= =verl=ad 7he disk s29sBs7em =r
ca2se 6r=9lems ?i7h device c=n8i32ra7i=n a7 9==7. I7Ps 9e77er 7= de7ermine 7he maLim2m n2m9er =8 s29mi7.
7ed IOs 7= each 42e2e =ver 7ime; 8=r 72nin3; i8 6=ssi9le. (e d=nD7 al?aBs have 7his s7a7is7ic; and in s2ch a
case ?e have 7= 32ess h=? m2ch 7= increase 7he 42e2e siKes; 7= av=id 8illin3 7he 42e2es 26 and addin3 la.
7encB 7= IOs.
"a3e '-
(hen B=2 increase 7he 42e2e5de67hs and 7he n2m9er =8 in.8li3h7 IOs 7ha7 are sen7 7= 7he disk s29sBs7em;
7he IO service 7imes are likelB 7= sli3h7lB increase; 927 7hr=23h627 ?ill als= increase. I8 IO service 7imes
s7ar7 a66r=achin3 7he disk 7ime=27 val2e; 7hen B=2Pre riskin3 s29mi77in3 m=re IOs 7han 7he disk s29sBs7em
can handle. I8 B=2 s7ar7 seein3 IO 7ime=27s and err=rs in 7he err=r l=3 indica7in3 6r=9lems c=m6le7in3 IOs;
7hen 7his is 7he 7ime 7= l==k 8=r hard?are 6r=9lems =r 7= make 7he 6i6e smaller 9B red2cin3 42e2e siKes.
Once IO service 7imes s7ar7 increasin3; ?ePve 62shed 7he 9=77leneck 8r=m 7he AI% disk ada67er driver 7=
s=me?here 9el=? 7he hdisk driver in 7he s7ack. This c=2ld 9e 7he ada67er driver; 7he SA; a VIOS; =r 7he
disk s29sBs7em G=r c=m6=nen7s ?i7hin i7H i7sel8. TB6icallB i7 ?ill 9e 7he disk s29sBs7em and =87en 7he
6hBsical disks ?i7hin i7.
Be c=3niKan7 i8 7he IO is hi3h ra7e asBnchr=n=2s se42en7ial IO. In s2ch a case; ?e h=6e and eL6ec7 7he a6.
6lica7i=n 7= ini7ia7e manB IO 7hreads s= IOs are sen7 7= 7he s7=ra3e as s==n as 6=ssi9le 7= drive 26 7hr2627.
This 7B6icallB 8ills 7he 42e2es makin3 IOs ?ai7; s= ?e d=nD7 eL6ec7 7= 9e a9le 7= make 7he 42e2es 9i3
en=23h 7= av=id 8illin3; n=r d= ?e necessarilB need 7=. E.,.; c=nsider i8 ?e have a #@ ,B 7a9le; an a66li .
ca7i=n needs 7= read 2sin3 a 7B6ical maL IO siKe =8 '5/ KB. S= 7he a66lica7i=n iss2es -@;0/@ '5/ KB read
re42es7s asBnchr=n=2slB; 7= read 7he en7ire 7a9le. !nless ?e have l=7s =8 hdisks and verB 9i3
42e2e5de67hs; 7he 42e2es ?ill 9e 82ll 8=r a?hile Gin 7his case E2s7 a 8e? sec=ndsH. Then 7hereDs s7ill 7he
42es7i=n =8 ?he7her 7he s7=ra3e can handle 7ha7 manB in.8li3h7 IOs; ?he7her ?eDll r2n in7= in7erc=nnec7
9and?id7h limi7s; and ?he7her increasin3 42e2e5de67h hel6s 6er8=rmance. Increasin3 42e2e5de67h 7=
handle s2ch IO =nlB hel6s 7= 7he eL7en7 i7 increases 7hr2627; re3ardless =8 ?ai7 7ime in 7he 42e2e. Increas.
in3 42e2e5de67h here increases h=? manB IO re42es7s can 9e d=ne in 6arallel; ass2min3 7he s7=ra3e can
d= m=re IOs in 6arallel ?i7h=27 si3ni8ican7lB de3radin3 IO la7encB =r r2nnin3 in7= =7her SA1s7=ra3e 9=7.
7lenecks.
$.2.. 1 T"nin& hdisk 4"e"e5depth
T?= a66r=aches 7= 72nin3 42e2e de67h are #H 9ase 7he 42e2e de67hs =n ac72al IO re42es7s B=2r a66lica7i=n
3enera7e =r 'H 2se a 7es7 7==l and 72ne 7he 42e2es 9ased =n ?ha7 7he disk s29sBs7em can handle. The ndisk
7==l G6ar7 =8 7he ns7ress 6acka3e availa9le =n 7he in7erne7 a7 h776F11???.
0-#.i9m.c=m1c=lla9=ra7i=n1?iki1dis6laB1(iki"7B6e1ns7ressH can 9e 2sed 7= s7ress 7he disk s29sBs7em 7= see
?ha7 i7 can handle; and 7= 6r=vide 7he da7a 7= 3enera7e IO"S vers2s. IO service 7ime 3ra6hs 8=r 7he s7=ra3e.
The a27h=rPs 6re8erence is 7= 72ne 9ased =n B=2r a66lica7i=n IO re42iremen7s; es6eciallB ?hen 7he disk is
shared ?i7h =7her servers.
F=r 72nin3; ?e can ca7e3=riKe 7he si72a7i=n in7= =ne =8 / casesF
I( Ser:
vice Ti<e
,ood 5oor Ti<eout +rrors
"a3e '5
2ueues
=7 8ill.
in3
Fillin3 =7 8illin3 Fillin3 =7 8illin3 Fillin3
Tunin9
%ase and
Action:
#. =
72nin3
needed
'. Increase
42e2e
siKes
+. Add re.
s=2rces 9el=?
7he hdisk driv.
er
-. Add re.
s=2rces 9el=?
7he hdisk driv.
er. MaB9e
72ne
42e2e5de67h.
5. Check 7he sBs7em 8=r 9=77le.
necks 9el=? 7he hdisk driver.
T2ne 7he s7=ra3e; add s7=ra3e
res=2rces; and1=r red2ce 42e2e
siKes
/. Red2ce 42e2e
siKes and add re.
s=2rces 9el=? 7he
hdisk driver
Fi32re #@ J T2nin3 Si72a7i=ns
(e like 7= 72ne 7he 42e2es; s= 7he 42e2es d=nD7 8ill 26; and s7ill have 3==d IO service 7imes; leavin3 2s in
case #. Limi7ed s7=ra3e res=2rces =87en leave 2s in case + =r -; 3e77in3 7he m=s7 7hr2627 6=ssi9le 8=r
Case ' is verB c=mm=n. The de8a2l7 42e2e5de67h val2es arenD7 a66r=6ria7e 8=r everB=ne; s= increasin3
7he 42e2e siKes is im6=r7an7. F=r eLam6le 7he de8a2l7 vSCSI hdisk has a 42e2e5de67h =8 +; limi7in3 i7s
IO"S 9and?id7h 7= +1Uav3 IO service 7imeV. In ch==sin3 h=? m2ch 7= increase 42e2e de67h; s=me =8 7he
c=nsidera7i=ns incl2deF
$=? manB main7enance ?ind=?s d= B=2 3e7 7= 72ne 7his<
$=? m2ch d= B=2 need 7= im6r=ve 6er8=rmance<
(ha7 kind =8 6lanned 3r=?7h in IO 6er8=rmance is needed in 7he 8272re<
$=? m2ch m=re 9and?id7h d=es B=2r s7=ra3e have<
The avera3e IO service 7ime and 7he avera3e IO ?ai7 7ime in 7he 42e2e
$=? l=n3 are B=2r da7a in7ervals; and are 7here 6eaks ?i7hin 7h=se in7ervals are B=2 missin3<
F=r hdisks; 7here isnD7 a sim6le me7h=d 7= kn=? h=? m2ch 7= increase 7he 42e2e siKes. The l=n3er IOs are
?ai7in3 in 7he 42e2es; 7he m=re ?e need 7= increase 42e2e5de67h. S=me7imes 7he 42e2e siKes need 7= 9e
increased si3ni8ican7lB; and d=29lin3 7hem in manB cases isnD7 2nreas=na9le.
A reas=na9le chan3e ?=2ld 9e; a7 a minim2mF
ne? 42e2e5de67h O 42e2e5de67h L Uav3 ?ai7 7ime1av3 IO service 7imeV
Or al7erna7ivelBF
ne? 42e2e5de67h O 42e2e5de67h L G # W Uhdisk 9l=cked IO ra7eV1Uhdisk IO"S ra7eV H
!s2allB 7he c=nsidera7i=ns lis7ed a9=ve call 8=r increasin3 42e2e5de67h even m=re. And 7hese varia9les
are rela7ed in a n=n.linear 82nc7i=n; ?hile ?eDre 2sin3 a linear 6r=6=r7i=nal es7ima7i=n.
Case +; ?here ?e have 6==r IO service 7imes and ?eDre n=7 8illin3 7he 42e2es; indica7es a si72a7i=n in
?hich ?e need 7= eLamine 7he IO s7ack 9el=? 7he hdisk driver 8=r 9=77lenecks; and relieve 7hem 9B addin3
res=2rces. And 7his a66lies 7= cases -./ as ?ell; ?here IO service 7imes are 6==r =r ?=rse. This incl2des
l==kin3 a7 7he ada67er driver; 7he SA and 7he disk s29sBs7em incl2din3 i7s vari=2s c=m6=nen7s. I8 =ne is
2sin3 VIO; 7hen ?e als= need 7= eLamine 7he real and vir72al ada67ers; and 7he VIOS as ?ell 8=r s288icien7
"a3e '/
res=2rces. The s7=ra3e adminis7ra7=r ?ill a66recia7e 7he sBs7ems adminis7ra7=r checkin3 7he sBs7em 8=r
9=77lenecks 6ri=r 7= askin3 7hem 7= analBKe 7he s7=ra3e 8=r 9=77lenecks.
Case -; ?here IO service 7imes are 6==r and 7he hdisk driver 42e2es are 8illin3 26; ?e have a 9=77leneck in
7he s7=ra3e; and ?eDve eLamined 7he s7ack 9el=? 7he hdisk driver and didnD7 8ind a 9=77leneck =n 7he sBs.
7em; 7hen increasin3 42e2e5de67h maB =r maB n=7 im6r=ve 7hr2627. I7 mi3h7 im6r=ve 7hr2627 9B all=?in3
7he disk s29sBs7em 7= 2se i7s eleva7=r al3=ri7hms 7= re=rder IOs; 927 =nlB 7es7in3 ?ill 7ell. T2nin3 7he s7=r .
a3e =r addin3 s7=ra3e res=2rces is 7he s2re ?aB 7= im6r=ve 6er8=rmance.
Cases 5 and / are si72a7i=ns 7= av=id; ?here ?eDre 3e77in3 IO 7ime=27 err=rs. $ere 7he sBs7em is s29mi7 .
7in3 m=re IOs 7han 7he s7=ra3e can handle. I8 B=2 have 7his si72a7i=n =r are cl=se 7= i7 Gas sh=?n 9B maLi.
m2m IO service 7imes in 7he sec=ndsH; check 8=r 9=77lenecks =n 7he sBs7em 9el=? 7he hdisk driver; 7hen
?=rk =n im6r=vin3 7he s7=ra3e via 72nin3 i8 7ha7Ds an =67i=n; =r 9B addin3 s7=ra3e res=2rces.
All disks and disk s29sBs7em have limi7s re3ardin3 7he n2m9er =8 in.8li3h7 IOs 7heB can handle; mainlB
d2e 7= mem=rB limi7a7i=ns 7= h=ld 7he IO re42es7 and da7a. (hen 7he s7=ra3e l=ses IOs; 7he IO even72allB
?ill 7ime =27 a7 7he h=s7; rec=verB c=de mi3h7 9e 2sed 7= res29mi7 7he IO; 927 in 7he mean7ime 7ransac7i=ns
?ai7in3 =n 7ha7 IO ?ill 9e s7alled. This isnP7 a desira9le si72a7i=n; as 7he C"! ends 26 d=in3 m=re ?=rk 7=
handle IOs 7han necessarB; 7he a66lica7i=n ?ill likelB s7all ?ai7in3 =n 7he IO 7ime =27 G7B6icallB +@ =r /@
sec=ndsH; =r even72allB an IO 8ail2re maB 9e re72rned 7= 7he a66lica7i=n ?hich can lead 7= c=rr267 da7a =r
an a66lica7i=n crash. S=me a66lica7i=ns have 7heir =?n in7ernal 7ime =27s; ?hich are =87en sh=r7er 7han IO
7ime=27s. Checkin3 B=2r s7=ra3e d=c2men7a7i=n 7= 2nders7and i7s limi7s ?ill hel6 B=2 av=id 7his si72a7i=n.
S=me s7=ra3e vend=rs =r s7=ra3e adminis7ra7=rs maB rec=mmend B=2 d= n=7 chan3e B=2r hdisk
42e2e5de67h val2es. In s2ch a case; 9e s2re 7= 6lan 8=r en=23h hdisks 7= 3e7 7he IO"S 9and?id7h B=2r a6.
6lica7i=n needs; es7ima7in3 7he IO"S 9and?id7h 6er L! as s=me7hin3 less 7han 42e2e5de67h1Uav3. IO
service 7imeV.
TB6icallB ?eDll se7 =2r 42e2e siKes s= 7ha7 ?hen ?eDre 8illin3 7he 42e2es; rand=m read disk service 7imes
are a7 ?=rs7 avera3in3 ar=2nd +.5% reas=na9le IO service 7imes 8=r 7he disk 7echn=l=3B. E.,.; ?i7h #5 K
R"M $DDs in a lar3e disk s29sBs7em ?i7h read cache; ?e eL6ec7 7B6ical rand=m small 9l=ck read IO ser .
vice 7imes in 7he 5.#@ ms ran3e; and ?e d=nD7 ?an7 7hem 7= avera3e m=re 7han 5@ ms. And in s2ch a case;
B=2 have a si3ni8ican7 IO 9=77leneck. $avin3 a 3ra6h sh=?in3 IO"S vs. IO service 7ime 8=r 7he s7=ra3e
all=ca7ed 7= 7he L"AR ?ill hel6 B=2 decide ?ha7 a reas=na9le maLim2m IO service 7ime is 8=r B=2r s7=r .
a3e; s2ch as sh=?n in 8i32re #' G7his 3ra6h is 8=r a sin3le hdisk; 927 7he sha6e =8 7he 3ra6h is similar 8=r
anB s7=ra3e; incl2din3 en7ire disk s29sBs7ems; =r 8=r ?ha7Ds 9een all=ca7ed 7= an L"ARH. In 8i32re #'; ?e
mi3h7 saB a reas=na9le IO service 7ime is &.5 ms =r less; ?hile ?e d=nD7 ?an7 7= eLceed a9=27 '* ms.
(ha7 reas=na9le IO service 7imes are; 8=r B=2r s7=ra3e; is disc2ssed in sec7i=n -.5 What Are Good, Rea-
sonable and Poor IO Service Times?
$.2.. 2 T"nin& ,cs De'ice !"e"e Attrib"tes
Cases ' 7hr2 / 8i32re #@ all sh=? 6==r IO service 7imes; indica7in3 7ha7 7here is a 9=77leneck in 7he IO s7ack
9el=? 7he hdisk driver. C2s7 9el=? 7he hdisk driver is 7he ada67er driver; s= ?e sh=2ld check i7 7= make
s2re i7 isnD7 7he 9=77leneck; 9e8=re ?e s7ar7 l==kin3 in7= 7he SA =r disk s29sBs7em.
"a3e '&
The ada67er driver s7a7is7ics ca672re 7he 6eak ?=rkl=ad. F=r n2m5cmd5elems; 7he 8=ll=?in3 c=mmands
7ell 2s eLac7lB h=? 7= se7 n2m5cmd5elemsF
# fcstat D! Bfcs#C
# pcmpat- 7uery adaptstats
# datapat- 7uery adaptstats
F=r 7he AI% "CMF
ne? n2m5cmd5elems O Uhi3h ?a7er mark =8 ac7ive c=mmandsV W Uhi3h ?a7er mark =8 6endin3
c=mmandsV
E.,. ?i7h 7his =27627F
# fcstat D! fcs&
>>>
F* '*'I #dapter !river 8ueue 'tatistics
)umber of pending commandsF 0
Hig- ,ater mar. of pending commandsF 6&
)umber of commands in t-e #dapter !river Held off 7ueueF 0
Hig- ,ater mar. of number of commands in t-e #dapter !river Held off 7ueueF 0
>>>
(eDd se7 n2m5cmd5elems 7= #*@W0#O '&#.
F=r SDD"CM =r SDDF
ne? n2m5cmd5elems O maLim2m I1O val2e dis6laBed ?i7h 7he res6ec7ive a9=ve c=mmand
E.,.; ?i7h 7his =27627F
#dapter #F 0
EEEEEEEEEEEEE
I0+F 236160333 (24(1(5& 4 0 (58
(eDd se7 n2m5cmd5elemsO'5*.
I8 B=2Dre a7 a level =8 AI%1VIOS lackin3 7he 8cs7a7 JD c=mmand da7a; ?e can 2se 8cs7a7F
# fcstat fcs0
>>>
)o #dapter Elements *ountF 0
)o *ommand 3esource *ountF 0
>>>
"a3e '*
This 8cs@ da7a sh=?s an eLam6le =8 an ada67er 7ha7 has s288icien7 val2es 8=r n2m5cmd5elems and
maL5L8er5siKe. =n.Ker= val2es indica7e a si72a7i=n in ?hich IOs ?ai7ed in 7he ada67er driver 42e2e d2e
7= lack =8 res=2rces. Increase n2m5cmd5elems 8=r n=n.Ker= val2es =8 = C=mmand Res=2rce C=2n7
?hen 7he ra7e is si3ni8ican7. Increase 7he DMA 6==l siKe 8=r n=n Ker= val2es 8=r = DMA Res=2rce ?hen
7he ra7e is si3ni8ican7; i8 6=ssi9le. =7e 7ha7 chan3in3 maL5L8er5siKe isnD7 al?aBs 6=ssi9le; and can lead
7= 9==7 8ail2re. "lease see sec7i=n #.+ Changing Device Attribte !ales 7ha7 disc2sses 7his i8 B=2 havenD7
alreadB.
$ere; similar 7= increasin3 42e2e5de67h; ?eDd increase n2m5cmd5elems 7= s=me7hin3 m=re 7hanF
ne? n2m5cmd5elems O n2m5cmd5elems L U9l=cked IO ra7eV1UIO"S ra7eV
= adE2s7men7 =8 7he DMA mem=rB siKe is 6=ssi9le =n vir72al FC ada67ers; E2s7 =n 7he real ada67er.
$.$. A,ter 6o"r Initial T"nin&
A87er chan3in3 7he val2es; r2n B=2r a66lica7i=n and l==k a7 7he s7a7is7ics 8=r 6eak IO 6eri=ds and 72ne
a3ain; i8 needed. (hile en7irelB elimina7in3 42e2e ?ai7 7ime is 7he 3=al; 7he re72rns 8r=m inves7in3 7ime in
7his 72nin3 diminish as 8e?er IOs ?ai7 in 7he 42e2es; and as 7heB ?ai7 less 7ime as ?ell.
$.(. T"nin& Order
Re3ardin3 7he =rder 7= 72ne 7he s7ack; =ne 3enerallB 72nes 42e2e5de67h; n2m5cmd5elems; and
maL5L8er5siKe 7=3e7her in =ne main7enance ?ind=?. T2nin3 7he l=?er laBers =8 7he s7ack 8irs7 Ge.3.; 72n.
in3 7he ada67er 42e2e siKes 9e8=re 7he hdisk 42e2e siKesH can res=lve 42e2e ?ai7s hi3her in 7he s7ack. On
7he =7her hand; 72nin3 hi3her laBers in 7he s7ack 7B6icallB increase 7he n2m9er =8 in.8li3h7 IOs sen7 7= 7he
l=?er laBers ?hich can res2l7 in m=re 9l=ckin3 7here. S= 72nin3 7hese is an i7era7ive 6r=cess.
$.-. #hat Are 7ood8 0easonable and +oor IO Ser'ice Times9
(ha7 is 3==d; reas=na9le; =r 6==r is a 8ac7=r =8 7he s7=ra3e 7echn=l=3B; 7he s7=ra3e cache siKes; and 7he IO
?=rkl=ad 8=r 7he disk s29sBs7em. Ass2min3 n= IOs are 42e2ed 7= a $DD; a 7B6ical read ?ill 7ake s=me.
?here 8r=m @ 7= #5 ms; =r s=; de6endin3 =n h=? 8ar 7he ac72a7=r has 7= 7ravel Gseek 7imeH; h=? l=n3 i7
7akes 7he disk 7= r=7a7e 7= 7he ri3h7 sec7=r Gr=7a7i=n 7ime 9ased =n s6indle s6eedH; and h=? l=n3 i7 7akes 7=
read 7he da7a G7rans8er 7imeH. Then 7he da7a m2s7 m=ve 8r=m 7he s7=ra3e 7= 7he h=s7. TB6icallB 7he 7ime is
d=mina7ed 9B seek 7ime W r=7a7i=n 7ime; 7h=23h 8=r lar3e IOs 7rans8er 7ime als= can 9e si3ni8ican7. This in.
cl2des 7imes 7= 7rans8er 7he IO 7=18r=m 7he disk; as ?ell as sendin3 i7 acr=ss vari=2s in7er8aces Gs7=ra3e
9ackend; s7=ra3e 7= h=s7 in7er8ace; e7c. incl2din3 FC; SAS; iSCSI and FC=EH. S=me7imes 7he da7a ?ill 9e
in disk s29sBs7em read cache; in ?hich case 7he IO service 7ime is ar=2nd # ms =r 9e77er. TB6icallB 8=r
lar3e disk s29sBs7ems 7ha7 arenP7 =verl=aded ?i7h #5K R"M disks; read IO service 7imes ?ill avera3e
"a3e '0
ar=2nd 5.#@ ms. (hen small rand=m reads s7ar7 avera3in3 3rea7er 7han #5 ms ?i7h #5K R"M disks; 7his
indica7es 7he s7=ra3e 27iliKa7i=n is hi3h.
(ri7es 7B6icallB 3= 7= ?ri7e cache Gass2min3 i7 eLis7sH ?i7h avera3e IO service 7imes 7B6icallB less 7han
a9=27 '.5 ms; 927 7here are eLce67i=ns. I8 7he s7=ra3e is sBnchr=n=2slB mirr=rin3 7he da7a 7= a rem=7e si7e;
?ri7es can 7ake m2ch l=n3er as ?e have 7= add 7he in7er.si7e r=2nd 7ri6 la7encB. And i8 7he IO is lar3e GsaB
/- KB =r lar3erH 7hen 7he 7rans8er 7ime 9ec=mes m=re si3ni8ican7 and 7he avera3e 7ime is sli3h7lB ?=rse. I8
7herePs n= cache; 7hen ?ri7es 7ake a9=27 7he same as reads. S=me disk s29sBs7ems d=nD7 cache lar3e ?ri7e
IOs and send 7hem direc7lB 7= 7he s7=ra3e.
$ereDs a 7a9le sh=?in3 eL6ec7ed IO service 7imes and IO"S 8=r li3h7lB l=aded disk 7echn=l=3iesF
Fi32re ## . IO"S 8=r di88eren7 disk 7echn=l=3ies
These 7imes are sim6lB calc2la7ed as IO"S O #1 Gr=7a7i=nal la7encB W av3. seek 7imeH
In reali7B; $DDs can d= m=re =r less IO"S a7 hi3her1l=?er IO service 7imes res6ec7ivelB. And =ne can
charac7eriKe a $DD in an IO"S vs. IO service 7ime char7 as 8=ll=?s Gin 7his case 8=r a &'@@ R"M SATA
disk d=in3 - KB IOsHF
"a3e +@
Fi32re #' . Sam6le IO"S vs. IO service 7ime 3ra6h
S= ?hile 7his &'@@ R"M SATA disk can achieve =ver #*@ IO"S; ?e maB n=7 ?an7 7= drive i7 a7 7ha7 ra7e;
and 6re8er a l=?er IO"S ra7e and a 9e77er IO service 7ime. The 6ercen7a3es =n 7he 3ra6hed line are 7he
$DD 27iliKa7i=n. Th2s; ?ha7Ds reas=na9le de6ends =n 7he ?=rkl=ad and 7he c2s7=merDs 6er8=rmance re.
42iremen7s. F=r a $DD like 7his; an IO service 7ime =8 '@ ms is reas=na9le; 2nless B=2Dve 6lanned 7= d=
8e?er IO"S and 3e7 9e77er IO service 7imes 7= mee7 B=2r a66lica7i=n 6er8=rmance re42iremen7s. Similar
char7s eLis7 8=r disk s29sBs7ems; and =ne can make s2ch a char7 8=r s7=ra3e all=ca7ed 7= an L"AR.
I8 7he IO is lar3e 9l=ck se42en7ial; 7hen 9esides 7he increased 7rans8er 7ime; ?e eL6ec7 IOs 7= 42e2e a7 7he
6hBsical disk; and IO service 7imes 7= 9e m2ch l=n3er =n avera3e. E.,.; i8 an a66lica7i=n s29mi7s 5@ IOs
GsaB 5@ /- KB IOs readin3 a 8ile se42en7iallBH 7hen 7he 8irs7 8e? IOs ?ill have reas=na9lB 3==d IO service
7imes; ?hile 7he las7 IO ?ill have had 7= ?ai7 8=r 7he =7her -0 7= 8inish 8irs7; and ?ill have a verB l=n3 IO
service 7ime. (e h=6e 7= 42e2e 26 a 92nch =8 IOs 8=r se42en7ial IO 7hreads; and s= ?e h=6e 7heB ?ill 9e
?ai7in3 in 7he 42e2e readB 7= 3=. Th2s; ?e d=nD7 3e7 c=ncerned ?i7h l=n3 IO service 7imes 8=r se42en7ial
IO; and ins7ead 8=c2s =n 7hr2627.
IOs 7= SSDs18lash are 7B6icallB less 7han # ms and 8=r SSDs18lash in disk s29sBs7ems; 7B6icallB less 7han '
ms; and =ccasi=nallB hi3her. IO la7encies 8=r 8lash have 9een im6r=vin3; as sh=?n in 7his char7F
"a3e +#
Fi32re #+ . eMLC SSD 6er8=rmance
(here 7he 8lash is im6lemen7ed als= ma77ers. The n2m9ers in 7he char7 are 8=r SAS a77ached SSDs. I8 7he
8lash is in a disk s29sBs7em; add a9=27 @.5 ms 7= 7he la7encB 8=r 7he IO 7= 3= 7hr2 7he disk s29sBs7em 9ack
end in7er8aces and als= 7hr2 7he SA 8=r ?ha7 is eL6ec7ed. I8 2sin3 VIO; 7hen add @.@5 ms G5@ micr= sec.
=ndsH 7= la7encB 7= 3= 7hr2 7he VIOS. The FC a77ached IBM FlashSBs7em can achieve @.+ ms 8=r IOs
7hanks 7= i7s 2se =8 a da7a 6a7h 2sin3 hard?are ra7her 7han s=87?are.
!nders7andin3 ?ha7 reas=na9le IO service 7imes are 8=r B=2r s=l27i=n is im6=r7an7 in 72nin3 7he 42e2es; s=
B=2 kn=? i8 B=2 have a 9=77leneck 9el=? 7he hdisk driver; =r n=7.
$.-..1 Stora&e %ache )ana&ement Al&orithm :,,ects on IO Ser'ice Times
S=me disk s29sBs7ems 2se cache mana3emen7 al3=ri7hms; ?hich s=me7imes 7ake a?aB ?ri7e cache 8=r a
L!. This =cc2rs ?here 7he IO"S ?ri7e ra7e 7= a L! is verB l=?; and ?here 7here is a hi3h read ra7e
8r=m 7he L!; =r heavB 2se =8 7he cache else?here. In s2ch cases; 7he disk s29sBs7em maB 7ake all =r
nearlB all 7he ?ri7e cache 8=r a L! and 2se i7 else?here. The res2l7 ?hen an a66lica7i=n sends a ?ri7e in
7his si72a7i=n is 7he disk s29sBs7em cache mana3emen7 al3=ri7hms have 7= 7ake 7he 7ime 7= 8ree 26 and all=.
ca7e s=me cache 8=r 7he ?ri7e; and 6lace 7he ?ri7e da7a in 7he cache s= 7he disk s29sBs7em can 7hen send
"a3e +'
9ack an ackn=?led3emen7 7ha7 7he ?ri7e is c=m6le7e. This adds si3ni8ican7 la7encB 7= 7hese IOs; and B=2r
i=s7a7 da7a maB l==k likeF
# iostat Dl!
!is.sF xfers read ,rite
-------------- -------------------------------- ------------------------------------ ------------------------------------
Jtm bps tps bread b,rtn rps avg min max time fail ,ps avg min max time fail
act serv serv serv outs serv serv serv outs
-dis.0 0>3 (1>49 3>& &6>39 4>59 &>2 &>4 0>2 &6>8 0 0 &>1 0>8 0>1 1>6 0 0
-dis.& 0>& 508>1 0>& 343>0 &35>1 0>& 8>& 0>5 (2>4 0 0 0>0 0>8 0>1 &>0 0 0
-dis.( 0>0 14>8 0>0 0>0 14>8 0>0 0>0 0>0 0>0 0 0 0>0 0>8 0>4 &>0 0 0
-dis.3 &>& 34>39 2>2 (5>&9 &(>(9 (>0 0>8 0>3 &0>2 0 0 (>2 2>2 0>1 138>2 0 0
-dis.2 80>& 33>1" 56(>5 33>1" 38>(9 586>2 (>2 0>3 853>1 0 0 3>& 1>5 0>5 450>3 0 0
-dis.5 53>( &1>6" 302>( &1>6" (&>59 30(>( 3>0 0>3 &>0' 0 0 (>0 &1>2 0>4 426>3 0 0
-dis.1 &>& (&>49 2>( &>69 &6>89 0>& 0>1 0>5 0>8 0 0 2>0 (>4 0>1 265>1 0 0
=7e 7he hi3h ?ri7e IO service 7imes in red and 7he hi3h read IO"S in 9l2e. Overall; 7he L"AR is d=in3
0@+.* IO"S; 927 la7encB is =nlB a88ec7ed 8=r &.- IO"SQ 7h2s; less 7han #X =8 7he IOs are a88ec7ed.
The disk s29sBs7em mi3h7 =88er s=me 72nin3 7= 6reven7 7his; 927 i8 n=7; 7here isnD7 anB easB me7h=d 7= deal
?i7h i7. C=nsider 7ha7 since 7he im6ac7 is 7= a small 8rac7i=n =8 7he IOs; 7ha7 =verall i7 isnD7 likelB 7= have a
si3ni8ican7 6er8=rmance im6ac7. Th2s; =ne needs 7= 9e a?are =8 7his; 6ri=r 7= decidin3 7he ?ri7es are 7==
sl=?.
$... T"nin& !"e"e Si;es in VIO :n'ironments
IO 7hr2 VIO is d=ne ei7her ?i7h vSCSI =r "IV; 2sin3 a vir72al SCSI =r a vir72al Fi9re Channel GvFCH
ada67er as sh=?n in 8i32res #- and #5 res6ec7ivelB.
Fi32re #- J vSCSI Archi7ec72re
"a3e ++
Fi32re #-5 J "IV Archi7ec72re
These dia3rams sh=? 7he l=3ical c=nnec7i=ns =8 7he vir72al and real hard?are; 927 7heB d=nD7 sh=? 7he IO
s7ack; s= le7Ds l==k a7 7ha7F
"a3e +-
Fi32re #5/ J VIO IO S7ack T2nin3
=7e 7ha7 ?e d= n=7 3= 7hr2 7he hdisk driver in 7he VIOS ?i7h "IV. Th2s; "IV IO s7a7is7ics are n=7
ca672red via i=s7a7; sar =r =7her c=mmands a7 7he VIOS 7ha7 access 7he hdisk driver s7a7is7icsY (e can s7ill
ca672re IO s7a7is7ics via 8cs7a7 and nm=n 7ha7 access 7he ada67er driver s7a7is7ics.
(i7h vSCSI; ?e 72ne hdisk 42e2e5de67h a7 9=7h 7he VIOS and 7he VIOC. And ?e als= 72ne 7he real FC
ada67ers c=nnec7ed 7= 7he s7=ra3e. The ada67er als= has a 8iLed 42e2e; s= 7= av=id 42e2in3 7here; ?e limi7
7he n2m9er =8 hdisks 2sin3 7heir ada67ers 9ased =n 7heir 42e2e siKes.
(i7h "IV; ?e 72ne 7he hdisk 42e2e de67hs a7 7he VIOC; 7he vFC ada67ers a7 7he VIOCs; and 7he real
ada67ers a7 7he VIOSs. One 7hin3 ?e canD7 chan3e 8=r 7he vFC is 7he DMA mem=rB siKe; ?hich is 8iLed
re3ardless =8 7he maL5L8er5siKe a77ri927e val2e. I7Ds im6=r7an7 7= kn=? 7he vFC ada67ers have an e88ec.
7ive limi7 =8 '5/ in.8li3h7 IOsQ 7h2s; increasin3 n2m5cmd5elems 9eB=nd '5/ ?=nD7 have anB a88ec7.
$.... 1 A'oidin& VIOS O"ta&es
T= chan3e device a77ri927es means 7heB cann=7 9e in 2se. Disks ma66ed 7hr2 vSCSI =r vFCs ma66ed 7hr2
real FC ada67ers are in 2se. And =87en 7here are several vFCs 2sin3 a sin3le real FC. Th2s; chan3in3 7he
real FC ada67ers s266=r7in3 "IV; ?i7h=27 a VIOS =27a3e; is di88ic2l7.
A sim6le a66r=ach is 7= 2se Live "ar7i7i=n Mi3ra7i=n GL"MH 7= m=ve 7he L"ARs; 2sin3 7he VIOS; 7= an.
=7her sBs7em; make 7he chan3e 7= 7he real ada67er a77ri927es; 7hen 2se L"M 7= m=ve 7he L"ARs 9ack.
"a3e +5
An=7her a66r=ach is essen7iallB 7= re6lace 7he ada67er; see
h776F116ic.dhe.i9m.c=m1in8=cen7er16=?ersBs1v+r#m51indeL.Es6<7=6icOX'Fi6hakX'Fi6hak.vi=s.rr.h7m; =r
7= add a ne? ada67er; ne? vFCs; chan3es a7 7he s7=ra3e and SA; 7hen rem=ve 7he =ld vFCs and ada67er.
These a66r=aches are 7ime c=ns2min3.
In d2al VIOS envir=nmen7s; ?e can make chan3es 7= 7he ODM 8=r device a77ri927es Gvia 7he J" 8la3 =8 7he
chdev c=mmandH; and re9==7 8=r a sh=r7 =27a3e =8 =ne VIOS.
"IV hdisk a77ri927es are chan3ed a7 7he VIOC; s= =ne =nlB needs 7= s7=6 2sin3 7hem 7= chan3e 7he a7.
7ri927es.
VSCSI hdisks can 9e similarlB chan3ed a7 7he VIOC. Chan3in3 7hem a7 7he VIOS re42ires s7=66in3 7heir
2se a7 7he VIOC and 2nma66in3 7hem 8r=m 7he clien7 a7 7he VIOS.
F=r c2s7=mers ?h= can 7ake =27a3es =n 7he VIOS; i7Ds =87en desira9le 8=r each hdisk 7= have 7he same
hdisk 42e2e siKe a7 9=7h 7he VIOS and VIOCQ 7h2s; ?hen chan3in3 i7 a7 7he VIOC =ne als= chan3es i7 a7
7he VIOS.
$.... 2 To Strate&ies to <imit In*,li&ht IOs to the Stora&e
As 6revi=2slB men7i=ned; i7Ds im6=r7an7 7= limi7 7he n2m9er =8 in.8li3h7 IOs 7= 7he s7=ra3e. B27 7here are
di88eren7 ?aBs 7= d= 7his. IBM has 6revi=2slB rec=mmended 7ha7 =ne se7 42e2e5de67h 8=r vSCSI hdisks 7=
ma7ch a7 7he VIOC and VIOS. And ?e 2s2allB rec=mmend 72nin3 9=7h 7he hdisk and ada67er drivers 7=
E2s7 have en=23h res=2rces 7= handle 7he 6eak ?=rkl=ad. $=?ever; ?e can als= 7ake 7he a66r=ach ?here.
9B ?e limi7 7he IOs a7 7he hdisk drivers; 7he ada67er drivers; =r a7 7he VIOCs.
As s7a7ed in sec7i=n #.'F
The maLim2m in.8li3h7 IOs a sBs7em ?ill s29mi7 7= SA s7=ra3e is 7he smalles7 =8 7he 8=ll=?in3
The s2m =8 7he hdisk 42e2e5de67hs
The s2m =8 7he ada67er n2m5cmd5elems
The maLim2m n2m9er =8 in.8li3h7 IOs s29mi77ed 9B B=2r a66lica7i=nGsH
Th2s; ?e can achieve =2r =9Eec7ive 9B limi7in3 7he hdisk 42e2e5de67hs s= 7ha7 7heir s2m is less 7han 7he
maLim2m in.8li3h7 IOs 7he s7=ra3e ?ill handle. SimilarlB; ?e can limi7 7he s2m =8 7he n2m5cmd5elems
s= 7ha7 i7 is less 7han 7he s7=ra3e limi7s. And ?e can limi7 7he in.8li3h7 IOs 9B adE2s7in3 hdisk 42e2e5de67h
a7 ei7her 7he VIOC =r 7he VIOS; and similarlB ?e can limi7 7he IOs 9B adE2s7in3 n2m5cmd5elems a7 7he
VIOS =r a7 7he VIOCs ?hen 2sin3 "IV.
This 3ives 2s 7?= s7ra7e3ies 8=r limi7in3 in.8li3h7 IOsF
"a3e +/
#. Se7 all 7he 42e2e siKes 7= 7he minim2m siKe 7ha7 can handle 7he 6eak ?=rkl=ad. Se7 vSCSI hdisk
42e2e5de67h 7= ma7ch a7 7he VIOS and 7he VIOC. T2ne ada67er n2m5cmd5elems and DMA mem.
=rB siKe a7 7he VIOS and n2m5cmd5elems a7 7he VIOC; 7= handle 7he 6eak ?=rkl=ads.
'. Se7 7he 42e2e siKes a7 7he VIOS 7= 7heir maLim2m val2es; and limi7 7hem a7 7he VIOCs.
S7ra7e3B # has 7he vir72e =8 minimiKin3 7he 2se =8 res=2rces Gm=s7lB mem=rB 7= handle increased 42e2e
siKesH. S7ra7e3B ' has 7he vir72e =8 minimiKin3 VIOS =27a3es 8=r 72nin3 and red2cin3 adminis7ra7=r ?=rk;
927 2ses eL7ra mem=rB. B=7h a66r=aches ins2re 7ha7 ?e d=nD7 have hdisk driver 42e2e 9=77lenecks a7 7he
VIOS.
C2s7=mers maB 7ake 7he a66r=ach 7ha7 9es7 8i7s 7heir needs.
$.... $ T"nin& 'S%SI !"e"es
(hen 2sin3 vSCSI; =ne c=n8i32res vir72al SCSI ada67ers 8=r 7he VIOC 7= access s7=ra3e c=n8i32red =n 7he
VIOS. F=r each vir72al SCSI ada67er; 7here ?ill 9e a vscsi device in a VIOC; and a ma7chin3 vh=s7 device
in 7he VIOS. (e d=nD7 have kn=9s 7= chan3e 42e2e siKes 8=r 7he vscsi ada67ers; ins7ead ?e ch==se h=?
manB ?e crea7e and ?hich hdisks 2se ?hich ada67ers.
These vir72al ada67ers have a 8iLed 42e2e de67h. There are 5#' c=mmand elemen7s =8 ?hich ' are 2sed 9B
7he ada67er; + are reserved 8=r each vSCSI L! 8=r err=r rec=verB and 7he res7 are 2sed 8=r IO re42es7s.
Th2s; ?i7h 7he de8a2l7 42e2e5de67h =8 + 8=r vSCSI L!s; 7ha7 all=?s 8=r 26 7= *5 L!s 7= 2se an ada67er
?i7h=27 9l=ckin3 7hereF G5#' . 'H 1 G+ W +H O *5 r=2ndin3 d=?n. S= i8 ?e need hi3her 42e2e de67hs 8=r 7he
devices; 7hen 7he n2m9er =8 L!s 6er ada67er is red2ced. E.,.; i8 ?e ?an7 7= 2se a 42e2e5de67h =8 '5;
7ha7 all=?s 5#@1'*O #* L!s. (e can c=n8i32re m2l7i6le vSCSI ada67ers 7= handle manB L!s ?i7h hi3h
42e2e de67hs. Each vSCSI ada67er 2ses addi7i=nal mem=rB and res=2rces =n 7he hB6ervis=r. F=r a 8=rm2.
la; 7he maLim2m n2m9er =8 L!s 6er vir72al SCSI ada67er Gvh=s7 =n 7he VIOS =r vscsi =n 7he VIOCH is
OITG5#@1G)W+HH ?here ) is 7he 42e2e5de67h =8 all 7he L!s Gass2min3 7heB are all 7he sameH. $ereDs a
7a9le 8=r 32idanceF
vS*SI hdisk
+ueue de'th
Ma hdisks
'er vscsi
ada'ter,
& - de!ault ./
1$ &0
2% 1.
&2 1%
#% 1
1$$ %
12. &
2/2 2
2/# 1
"a3e +&
Fi32re #/& J vSCSI L! limi7s
Z MaLim2m 7= ens2re n= 9l=ckin3 =8 IOs d2e 7= lack =8 vscsi 42e2e sl=7s
(hen 2sin3 8lash 9acked L!s; ?here B=2 eL6ec7 rela7ivelB hi3h IO"S; ?e c2rren7lB rec=mmend 2sin3 a
sin3le vSCSI ada67er 6er L!. This is d2e 7= =7her limi7a7i=ns ?i7h 7he vSCSI ada67er and 9eca2se 8lash
IOs are s= 8as7.
F=r LV vSCSI hdisks; ?here m2l7i6le VIOC hdisks are crea7ed 8r=m a sin3le VIOS hdisk; 7hen =ne maB
7ake a dedica7ed res=2rce; shared res=2rce =r an in 9e7?een a66r=ach 7= 7he VIOS hdisk 42e2e sl=7s. See
7he sec7i=n -.5 Theoretical Thoghts on Shared vs" Dedicated Resorces.
$.... ( T"nin& /+IV !"e"es
(hen 2sin3 "IV; ?e have vFC and real FC ada67er 6=r7s; and =87en have m2l7i6le vFCs 7ied 7= a sin3le
real FC ada67er 6=r7.
I8 B=2 increase n2m5cmd5elems =n 7he vir72al FC GvFCH ada67er 6=r7; 7hen B=2 sh=2ld make s2re 7he se7.
7in3 =n 7he real FC ada67er 6=r7 is e42al =r 3rea7er. (i7h m2l7i6le vFCs 6er real FC; 7he real FC 6=r7Ds re.
s=2rces 9ec=me a shared res=2rce 8=r 7he vFCs. Kee6 in mind 7ha7 n2m5cmd5elems is e88ec7ivelB limi7ed
7= '5/ =r smaller; s= 7here isnD7 anB 9ene8i7 7= increasin3 n2m5cmd5elems 7= 3rea7er 7han '5/ 8=r vFCs.
:=2 can 2se 7he 8cs7a7 c=mmand 8=r 9=7h 7he vir72al ada67er 6=r7 as ?ell as 7he real ada67er 6=r7 8=r 72nin3;
and de7erminin3 ?he7her =r increasin3 n2m5cmd5elems =r maL5L8er5siKe mi3h7 increase 6er8=rmance.
S= 8=r heavB IO and es6eciallB 8=r lar3e IOs Gs2ch as 8=r 9ack26sH i7Ps rec=mmended 7= se7 maL5L8er5siKe
s2ch 7ha7 ?e 3ive 7he maLim2m DMA mem=rB 7= 7he 8cs device; ?hen 7he sBs7em c=n8i32ra7i=n ?ill al.
l=? i7.
$.... - T"nin& Shared Stora&e +ool !"e"es
$ereDs a dia3ram =8 a Shared S7=ra3e "==l GSS"H cl2s7erF
"a3e +*
Fi32re #&* J Shared S7=ra3e "==l Cl2s7er
The SS" and SS" cl2s7er re6=si7=rB L!s are all=ca7ed 8r=m a RAID 6r=7ec7ed SA disk s29sBs7emGsH 7=
all 7he VIOSs in 7he SS" cl2s7er. The SS" L!s are 7B6icallB verB lar3e; ?hile 7he cl2s7er re6=si7=rB
L! is a9=27 #@ ,B. These L!s are c=n8i32red as hdisks a7 7he VIOS. The VIOS adminis7ra7=r crea7es
vir72al hdisks 8r=m 7he SS" and ma6s 7hem 7= VIOCs 8=r ne? =r eLis7in3 L"ARs. This rem=ves 7he need
8=r 7he s7=ra3e1SA adminis7ra7=rs 7= 3e7 inv=lved; and s6eeds 7he 6r=cess. The vir72al disks 7he VIOS
adminis7ra7=r crea7es are s6read =27 acr=ss 7he L!s in 7he SS" 7= 9alance 7he IO ?=rkl=ad acr=ss 7hem.
SS"s als= =88er 7hin 6r=visi=nin3 and sna6sh=7 ca6a9ili7B as ?ell.
Fr=m a 72nin3 6ers6ec7ive; ?e 72neF
FC ada67er 42e2es a7 7he VIOS
42e2e5de67h =n 7he SS" hdisks a7 7he VIOS
42e2e5de67h =8 7he vir72al disks a7 7he VIOCs
(e =nlB need 7= s7=6 2sin3 7he hdisks a7 7he VIOC 7= chan3e 7heir a77ri927es. T= chan3e a77ri927es a7 7he
VIOS ?ill 7B6icallB re42ire a VIOS re9==7.
One sh=2ld als= 3ive s=me 7h=23h7 7= makin3 s2re 7he servers d=nD7 =verr2n 7he s7=ra3e ?i7h IO re42es7s.
One a66r=ach 7= 7his is 8=r 7he s7=ra3e adminis7ra7=r 7= m=ni7=r 7he s7=ra3e res=2rce 2se; and aler7 7he
server adminis7ra7=rs a9=27 i7. Then 7he server adminis7ra7=rs can inves7i3a7e and red2ce 42e2e siKes as
necessarB; 7B6icallB ?hile 7he s7=ra3e 7eam ?ill 9e 3e77in3 ne? res=2rces 7= handle 7he demand.
"a3e +0
Desi3nin3 7he 42e2e siKes 3e7s in7= sharin3 vs. dedica7in3 s7=ra3e res=2rces; 8r=m 7he maLim2m in.8li3h7
42e2e sl=7s ?i7hin 7he disk s29sBs7emGsH 7= 7he vir72al disk 42e2e sl=7s a7 7he VIOCs.
Fr=m a shared res=2rce 6ers6ec7iveF
The vir72al disks a7 7he VIOCs share 7he SS" hdisk 42e2es a7 7he VIOS
The SS" hdisks share 7he FC ada67er 42e2es a7 7he VIOS
The VIOSs share 7he disk s29sBs7em 42e2es 8=r i7s limi7 =8 in.8li3h7 IOs
(e can a88ec7 h=? 7hese res=2rces are dedica7ed =r shared 9B ?here and h=? m2ch =ne limi7s 7he 42e2e
siKes.
$... Theoretical Tho"&hts On Shared Vs. Dedicated 0eso"rces
)2e2e siKes limi7 7he maLim2m n2m9er =8 in.8li3h7 re42es7s 8=r s7=ra3e. S= red2cin3 7he 42e2e siKe =8
saB an hdisk; 3ives 7he =7her hdisks access 7= m=re 42e2e sl=7s 82r7her d=?n in 7he s7ack; s2ch as 7he
ada67er 42e2e sl=7s; all 7he ?aB 7= 7he disk s29sBs7emGsH 42e2e sl=7s. Th2s 9B red2cin3 42e2e siKes 8=r
hdisks; L"ARs; VIOSs; ada67ers and sBs7ems; ?e 6r=vide m=re =8 7he s7=ra3e res=2rces 7= 7he =7her
hdisks; L"ARs; VIOSs; ada67ers and sBs7ems res6ec7ivelB.
TB6icallB ?e have manB hdisk drivers sharin3 m2l7i6le ada67ers and ada67er drivers; 7h2s; 7he FC 42e2e
sl=7s are a shared res=2rce 8=r 7he hdisk driversF
Fi32re #*0 J Ada67er and 6=r7 hdisk 42e2e c=nnec7i=ns
Th2s; i7Ps 6=ssi9le 7= ens2re 7ha7 ?e never 8ill 7he ada67er 6=r7 42e2es; 9B makin3 S!MGhdisk@
42e2e5de67h; hdisk# 42e2e5de67h; ... hdiskM 42e2e5de67hH UO S!M G8cs@ n2m5cmd5elems; 8cs#
n2m5cmd5elems; ... 8cs n2m5cmd5elemsH. This ass2mes 7ha7 IOs are evenlB s6read acr=ss 7he ada67ers.
And m=s7 m2l7i.6a7h c=de d=es 9alance IOs acr=ss 7he ada67ers G=r a7 leas7 canH.
"a3e -@
Th=23h envir=nmen7s =87en have manB m=re hdisks 7han FC 6=r7s; and ens2rin3 ?e ?=nP7 8ill 7he ada67er
drivers can lead 7= small val2es 8=r 42e2e5de67h; and 82ll 42e2es =n 7he hdisk drivers.
S= 7here is 7he dedica7ed res=2rce a66r=ach; 7he shared res=2rce a66r=ach; and in 9e7?een dedica7ed and
shared. Takin3 7his sim6le eLam6le ?here ) re6resen7s 7he 42e2e siKe 8=r 7he device driverF
Fi32re #0@ J Dedica7ed ada67er 6=r7 42e2e sl=7 res=2rce
This ?=2ld 9e c=nsidered a dedica7ed res=2rce a66r=ach; ?here #@ =8 7he ada67er driver 42e2e sl=7s are
dedica7ed 7= each hdisk driver. $ere ?e kn=? ?ePll never s29mi7 an IO 7= a 82ll 42e2e =n 7he ada67er driv.
er.
Al7erna7ivelBF
Fi32re '@# J Shared ada67er 6=r7 42e2e sl=7 res=2rce
This ?=2ld 9e c=nsidered a shared res=2rce a66r=ach ?here 7he #@ ada67er 42e2e sl=7s c=2ld 9e 8illed 26
8r=m a sin3le hdisk driver.
And herePs an eLam6le =8 s=me7hin3 in 9e7?eenF
"a3e -#
Fi32re '' J Ada67er 6=r7 42e2e sl=7 res=2rce J 6ar7lB dedica7ed; 6ar7lB shared
$ere; 7here ?ill al?aBs 9e a7 leas7 5 42e2e sl=7s availa9le in 7he ada67er driver 8=r ei7her hdisk driver.
There are 6r=s and c=ns 7= each a66r=ach. The 9ene8i7 =8 7he dedica7ed res=2rce a66r=ach is 7ha7 7he re.
s=2rces all=ca7ed ?ill al?aBs 9e availa9le 927 7B6icallB 7here ?ill 9e 8e?er res=2rces availa9le 7= each 2ser
=8 7he res=2rce Ghere 7he res=2rce ?ePre c=nsiderin3 is 7he ada67er 42e2e sl=7s; and 7he 2sers =8 7he re.
s=2rce are 7he hdisk drivers sendin3 IO re42es7s 8r=m 7he hdisk 42e2esH. The 9ene8i7 =8 7he shared re.
s=2rce a66r=ach is 7ha7 ?ePll have m=re res=2rces 8=r an individ2al 2ser =8 7he res=2rce ?hen i7 needs i7
and i7 ?ill 9e a9le 7= 3e7 3rea7er 7hr2627 7han in 7he dedica7ed res=2rce a66r=ach. The a27h=r 3enerallB
6re8ers a shared res=2rce a66r=ach; as i7 3enerallB 6r=vides 7he 9es7 =verall 7hr2627 and 6rice 6er8=rmance.
=7e 7ha7 7his si72a7i=n =8 shared res=2rces =cc2rs in several 6=ssi9le ?aBs 9eB=nd hdisk drivers 2sin3
ada67er drivers. I7 is als= inv=lved ?henF
Several LV vSCSI hdisks 8=r a sin3le hdisk =n a VIOS
Several vFC ada67ers 2sin3 a sin3le real FC ada67er
Several L"ARs 2sin3 7he same disk s29sBs7em
Several sBs7ems 2sin3 SS"s
vSCSI hdisks 2sin3 a vscsi ada67er
S= similar a66r=aches can 9e 2sed 8=r sharin3 =r dedica7ed 7h=se res=2rces as ?ell.
(eDve 7aken 7?= a66r=aches 7= 72nin3 42e2es here. One a66r=ach 2ses IO s7a7is7ics 7= de7ermine h=? 9i3
7he 42e2es sh=2ld 9e; ?hile 7he =7her a66r=ach l==ks a7 7he 6=7en7ial n2m9er =8 IOs 7ha7 mi3h7 arrive 7= a
42e2e. The la7er a66r=ach ?ill 9e m=re c=nserva7ive in av=idin3 42e2e ?ai7; and ?ill als= have lar3er av.
era3e n2m9er =8 re42es7s in 7he service 42e2e.
In 6rac7ice =ne sh=2ld 3enerallB siKe 7he servers and s7=ra3e 9ased =n 7he a66lica7i=n res=2rce needs; se726
7he envir=nmen7; 7es7 i7s 6er8=rmance; and se7 7he 42e2es a66r=6ria7elB 7hen; 7akin3 in7= acc=2n7 6lanned
3r=?7h.
Ma7hema7icallB; 7he IO 42e2es here are cl=ses7 7= an M1D1c 42e2e. M re6resen7s IO arrivals and is as.
s2med 7= have a "=iss=n dis7ri927i=n. D re8ers 7he 8ac7 7ha7 IOs arrivin3 7= 7he 42e2e ?ill need s=me 7ime
7= service 7hem; and c re8errin3 7= 7he n2m9er =8 42e2e sl=7s ?here9B 26 7= c IOs can 9e serviced in 6aral.
"a3e -'
lel. An M1D1c 42e2e m=del is de8ined as a s7=chas7ic 6r=cess ?h=se s7a7e s6ace is 7he se7 [@;#;';+;...\
?here 7he val2e c=rres6=nds 7= 7he n2m9er =8 c2s7=mers in 7he sBs7em; incl2din3 anB c2rren7lB in service.
In 42e2in3 7he=rB; an M1D1c 42e2e re6resen7s 7he 42e2e len37h in a sBs7em havin3 c servers; ?here ar.
rivals are de7ermined 9B a "=iss=n 6r=cess and E=9 service 7imes are 8iLed Gde7erminis7icH. See
h776F11en.?iki6edia.=r31?iki1M1D1k542e2e.
"a3e -+
%. 2stimatin) A''lication 3er!ormance Im'rovement
"r=Eec7in3 a66lica7i=n 6er8=rmance im6r=vemen7 9ased =n disk la7encB im6r=vemen7; 8r=m 7he a66lica.
7i=nDs 6=in7 =8 vie?; has challen3es. S=me =8 7he challen3es incl2deF
Disk IOs are =87en d=ne in 6arallel
ManB IOs are d=ne asBnchr=n=2slB and d=nD7 9l=ck 7he a66lica7i=n; ?hile 7he a66lica7i=n is
9l=cked 8=r manB =7her IOs
(ri7es 7end 7= 9l=ck a66lica7i=ns m=re 7han reads
IO la7encB im6r=vemen7s 8=r reads and ?ri7es di88er
ever7heless; 9B makin3 s=me ass2m67i=ns; ?e can crea7e an es7ima7e; =r a ran3e =8 6=7en7ial im6r=ve.
men7; 2sin3 AmdahlDs la?.
Ass2min3 ?e can im6r=ve IO la7encB LX and 7he disk 6ar7 =8 7he E=9 is DX; 7hen 7he 6=7en7ial 6er8=r.
mance im6r=vemen7 G8=r a 9a7ch E=9; =r 8=r a66lica7i=n res6=nseH is DLL. E.,.; ass2min3 IO service 7ime
avera3es & ms; ?ai7 7ime in 7he 42e2e avera3es + ms; and 7ha7 ?e can elimina7e 7ime in 7he 42e2e; 7hen LO
+1G&W+H O +@XQ 7h2s im6r=vin3 la7encB +@X 8r=m 7he a66lica7i=nDs 6=in7 =8 vie?. Ass2min3 7he disk
?=rk is 5@X =8 7he E=9 7ime; 7hen ?e can r2n 7he E=9 in 5@X L +@X O #5X less 7ime.
(i7h=27 kn=?in3 7he 6=r7i=n =8 7he E=9 7ha7 is disk rela7ed; ?e d=nD7 kn=? 7he ac72al im6r=vemen7; 927 i7
627s 7he 6=7en7ial im6r=vemen7 in7= 7he ran3e =8 @.L; and a c=26le =8 da7a 6=in7s a7 di88eren7 la7encies Gas
seen 9B 7he a66lica7i=nH can 6r=vide an es7ima7e =8 D.
"er8=rmance im6r=vemen7 7= a 9a7ch E=9 is re8lec7ed in 7he r2n 7ime. F=r OLT" a66lica7i=ns; red2c7i=ns
in IO la7encB are re8lec7ed in a66lica7i=n res6=nse 7ime 7= a 42erB126da7e; =r 7= maLim2m a66lica7i=n
7hr2627. Since 7he n2m9er =8 in.8li3h7 7ransac7i=ns is =87en limi7ed 9B mem=rB; 7he a9ili7B 7= d= 7ransac.
7i=ns 8as7er means ?e essen7iallB increase 7he maL T"S 8=r an L"AR 9B red2cin3 IO la7encB G=r al7erna.
7ivelB ?e c=2ld red2ce sBs7em res=2rces and kee6 7he same T"S ?i7h 8as7er IOsH.
An=7her me7ric ?e can calc2la7e is h=? m2ch IO 7ime ?e can save 9B elimina7in3 7he 42e2e ?ai7 7imeF
IO"S L Uav3 7ime in 7he 42e2e 6er IOV O IO la7encB savin3s 6er sec=nd
C2s7 9e a?are 7ha7 7his isnD7 7ime savin3s 8=r 7he a66lica7i=n. E.,.; i8 ?eDre ?ai7in3 in 7he 42e2e =n aver .
a3e ' ms 8=r each IO; and ?eDre d=in3 #@;@@@ IO"S; 7he savin3s in IO la7encB ?=2ld 9e '@ sec=nds 6er
sec=nd. $=?ever ?e canD7 save m=re 7han # sec=nd everB sec=nd; and 9eca2se 7he IOs are d=ne in 6aral .
lelQ 7h2s; 7he real a66lica7i=n savin3s is s=me7hin3 less 7han =ne sec=nd everB sec=nd. B27 7his me7ric is
an indica7=r ?he7her 72nin3 ?ill hel6 6er8=rmance si3ni8ican7lB and ?he7her 7he 72nin3 e88=r7 is ?=r7h 7he
7ime.
"a3e --
"a3e -5
A''endi4 Related 3u"lications
The 629lica7i=ns lis7ed in 7his sec7i=n are c=nsidered 6ar7ic2larlB s2i7a9le 8=r a m=re de7ailed disc2ssi=n =8
7he 7=6ics c=vered in 7his ?hi7e 6a6er.
MAI% 5.+ "er8=rmance Mana3emen7 ,2ideN; IBM
MAI% /.# "er8=rmance Mana3emen7 ,2ideN; IBM
MAI% &.# "er8=rmance Mana3emen7 ,2ideN; IBM
S"=?erVM Vir72aliKa7i=n =n IBM SBs7em 6F In7r=d2c7i=n and C=n8i32ra7i=nS; IBM Red9==ks
h776F11???.red9==ks.i9m.c=m1red6ieces1a9s7rac7s1s3'-&0-@.h7ml
S"=?erVM Vir72aliKa7i=n =n IBM SBs7em 6F Mana3in3 and M=ni7=rin3S; IBM Red9==ks
h776F11???.red9==ks.i9m.c=m1red6ieces1a9s7rac7s1s3'-&50@.h7ml
Fi9re.channel I1O "er8=rmance T2nin3 =n AI% 2sin3 8cs7a7Q A $=?.7= and !sa3e ,2ide
h776F11?+.@+.i9m.c=m1s266=r717echd=cs1a7smas7r.ns81(e9IndeL1TD#@/#''
IBM SBs7em 6 and AI% In8=rma7i=n Cen7er
h776F11629li9.9=2lder.i9m.c=m1in8=cen7er16series1v5r+1indeL.Es6 GAI% 5.+H
h776F11629li9.9=2lder.i9m.c=m1in8=cen7er1sBs7ems1sc=6e1aiL1indeL.Es6 GAI% /.#H
h776F11629li9.9=2lder.i9m.c=m1in8=cen7er1aiL1v&r#1indeL.Es6 GAI% &.#H
"a3e -/

AIX VIOS DiskAndAdapterQueueTuningV1.2

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

AIX VIOS DiskAndAdapterQueueTuningV1.2

Hochgeladen von

Copyright:

Verfügbare Formate

IBM Americas Advanced Technical Skills

Das könnte Ihnen auch gefallen