IBM Power Systems Performance

PowerVM Best Practice Guide

POWER7 Virtualization Best Practice Guide

Version 2.0

Sergio Reyes, Mala Anand and Pete Heyrman STG Power System Performance and Develo ment

IBM STG !ross Platform Systems Performance Copyright 2012 by International Business Machines Corporation. No part of this document may be reproduced or transmitted in any form without written permission from IBM Corporation. roduct data has been re!iewed for accuracy as of the date of initial publication. roduct data is sub"ect to change without notice. #his information may include technical inaccuracies or typographical errors. IBM may ma$e impro!ements and%or changes in the product&s' and%or programs&s' at any time without notice. (eferences in this document to IBM products) programs) or ser!ices does not imply that IBM intends to ma$e such products) programs or ser!ices a!ailable in all countries in which IBM operates or does business. #*+ IN,-(M.#I-N (-/I0+0 IN #*I1 0-C2M+N# I1 0I1#(IB2#+0 3.1 I13 4I#*-2# .N5 4.((.N#5) +I#*+( +6 (+11 -( IM 7I+0. IBM +6 (+1175 0I1C7.IM1 .N5 4.((.N#I+1 -, M+(C*.N#.BI7I#5) ,I#N+11 ,-( . .(#IC27.( 2( -1+ -( N-N8IN,(IN9+M+N#. IBM shall ha!e no responsibility to update this information. IBM products are warranted according to the terms and conditions of the agreements &e.g.) IBM Customer .greement) 1tatement of 7imited 4arranty) International rogram 7icense .greement) etc.' under which they are pro!ided. IBM is not responsible for the performance or interoperability of any non8IBM products discussed herein. #he performance data contained herein was obtained in a controlled) isolated en!ironment. .ctual results that may be obtained in other operating en!ironments may !ary significantly. 4hile IBM has re!iewed each item for accuracy in a specific situation) there is no guarantee that the same or similar results will be obtained elsewhere. 1tatements regarding IBM:s future direction and intent are sub"ect to change or withdrawal without notice) and represent goals and ob"ecti!es only. #he pro!ision of the information contained herein is not intended to) and does not) grant any right or license under any IBM patents or copyrights. In;uiries regarding patent or copyright licenses should be made) in writing) to< IBM 0irector of 7icensing IBM Corporation North Castle 0ri!e .rmon$) N5 10=0>81?@= 2.1... IBM) +nterprise 1torage 1er!er) +1C-N) ,IC-N) ,lashCopy) #otal1torage) 1ystem 1torage< 1ystem A) 1ystem i) 1ystem p) and A%-1 are trademar$s of International Business Machines Corporation in the 2nited 1tates) other countries) or both.

age 2

IBM STG !ross Platform Systems Performance

Preface
#his document is intended to address -4+(? ower/M best practices to attain best 7 .( performance. #his document by no means co!ers all the ower/M best practices so this guide should be used in con"unction with other ower/M documents.

T"e following is a list of #BM reference and documents t"at are good references$
A#% on P&'(R ) Performance *A+ http<%%www.ibm.com%common%ssi%cgi8bin%ssialiasB subtypeC4*DinfotypeC1.DappnameC1#9+E -E -E21+NDhtmlfidC -40F0>G21+NDattach mentC -40F0>G21+N. 0, #BM System Advanced P&'(R Virtuali,ation Best Practices Red-oo.$

"tt $//www0red-oo.s0i-m0com/red a ers/ dfs/red 12310 df

Virtuali,ation Best Practice$

"tt $//www0i-m0com/develo erwor.s/wi.is/dis lay/virtuali,ation/Virtuali,ation4Best4Pr actice

!onfiguring Processor Resources for System 5 S"ared6Processor Pool Micro6Partitions$

"tt $//www0i-msystemsmag0com/ai7/administrator/systemsmanagement/!onfiguring6 Processor6Resources6for6System6 56S"ar/

An 8PAR Review$

"tt $//www0i-msystemsmag0com/ai7/administrator/l ar/An68PAR6Review/

Virtuali,ation Tric.s$

"tt $//www0i-msystemsmag0com/ai7/trends/w"atsnew/Virtuali,ation6Tric.s/
A !om arison of PowerVM and 79:6Based Virtuali,ation Performance$

"tt $//www6 ;<0i-m0com/systems/ ower/software/virtuali,ation/w"ite a ers/ owervm=79:0"tml

#BM #ntegrated Virtuali,ation Manager$

"tt $//www6;<0i-m0com/systems/ ower/"ardware/w"ite a ers/ivm0"tml

Ac"ieving Tec"nical and Business Benefits t"roug" Processor Virtuali,ation$

"tt $//www0i-m0com/common/ssi/fcgi6-in/ssialias> infoty e?SA@su-ty e?'H@a name?STG(=P&=P&=AS(B@"tmlfid?P&8;<;CDAS( B@attac"ment?P&8;<;CDAS(B0PD*

age F

IBM STG !ross Platform Systems Performance

1.1 Introduction
PowerVM "y ervisor as well as A#% o erating system EA#% :02 T8 5 and a-ove versionsF on PowerD "ave im lemented en"anced affinity in a num-er of areas to ac"ieve o timi,ed erformance for wor.loads running in a virtuali,ed s"ared rocessor logical artition ESP8PARF environment0 !ustomers -y leveraging t"e -est ractice guidance descri-ed in t"is document can attain o timum a lication erformance in a s"ared resource environment0 T"is document covers -est ractices in t"e conte7t of PowerD systems t"erefore t"is section can -e used as an addendum to ot"er PowerVM -est ractice documents0

1.2 Virtual Processors

A virtual rocessor is a unit of virtual rocessor resource t"at is allocated to a artition or virtual mac"ine0 PowerVMG "y ervisor can ma a w"ole "ysical rocessor core or can time slice a "ysical rocessor core PowerVM Hy ervisor time slices Micro artitions on t"e "ysical !PAHs -y dis atc"ing and un6 dis atc"ing t"e various virtual rocessors for t"e artitions running in t"e s"ared ool0 #f a artition "as multi le virtual rocessors, t"ey may or may not -e sc"eduled to run simultaneously on t"e "ysical rocessors Partition entitlement is t"e guaranteed resource availa-le to a artition0 A artition t"at is defined as ca ed, can only consume t"e rocessors units e7 licitly assigned as its entitled ca acity0 An unca ed artition can consume more t"an its entitlement -ut is limited -y a num-er of factors$ Anca ed artitions can e7ceed t"eir entitlement if t"ere is unused ca acity in t"e s"ared ool, dedicated artitions t"at s"are t"eir "ysical rocessors w"ile active or inactive, unassigned "ysical rocessors, !&D utility rocessors and suc"0 #f t"e artition is assigned to a virtual s"ared rocessor ool, t"e ca acity for all t"e artitions in t"e virtual s"ared rocessor ool may -e limited T"e num-er of virtual rocessors in an unca ed artition t"rottles on "ow muc" !PA it can consume0 *or e7am le$ o An unca ed artition wit" 2 virtual !PA can only consume 2 "ysical rocessor of !PA resource under any circumstances o An unca ed artition wit" 1 virtual !PAs can only consume 1 "ysical rocessors of !PA Virtual rocessors can -e added or removed from a artition using HM! actions0 EVirtual rocessors can -e added u to Ma7imum virtual rocessors of a 8PAR and virtual rocessors can -e removed u to Minimum virtual rocessors of a 8PARF

age >

IBM STG !ross Platform Systems Performance

1.2.1 Sizing/configuring virtual processors T"e num-er of virtual rocessors in eac" 8PAR in t"e system s"ould not Ie7ceedJ t"e num-er of cores availa-le in t"e system E!(!/framewor.F or if t"e artition is defined to run in s ecific virtual s"ared rocessor ool, t"e num-er of virtual rocessors s"ould not e7ceed t"e ma7imum defined for t"e s ecific virtual s"ared rocessor ool0 Having more virtual rocessors configured t"an can -e running at a single oint in time does not rovide any additional erformance -enefit and can actually cause additional conte7t switc"es of t"e virtual rocessors reducing erformance0 #f t"ere are sustained eriods of time w"ere t"ere is sufficient demand for all t"e s"ared rocessing resources in t"e system or a virtual s"ared rocessor ool, it is rudent to configure t"e num-er of virtual rocessors to matc" t"e ca acity of t"e system or virtual s"ared rocessor ool0 A single virtual rocessor can consume a w"ole "ysical core under two conditions$ 20 SP8PAR "ad given an entitlement of 20; or more rocessor C0 T"is is an unca ed artition and t"ere is idle ca acity in t"e system0 T"erefore t"ere is no need to configure more t"an one virtual rocessor to get one "ysical core0 *or e7am le$ A s"ared ool is configured wit" 2: "ysical cores0 *our SP8PARs are configured eac" wit" entitlement 10; cores0 To configure virtual rocessors, we need to consider t"e wor.loadHs sustained ea. demandHs ca acity0 #f two of t"e four SP8PARs would ea. to use 2: cores Ema7 availa-le in t"e oolF, t"en t"ose two SP8PARs would need 2: virtual !PAs0 T"e ot"er two ea.s only u to 9 cores, t"ose two would -e configured wit" 9 virtual !PAs T"e ma7imum virtual rocessors of a 8PAR s"ould not e7ceed t"e ma7imum !PA ca acity availa-le in t"e s"ared rocessor ool even t"oug" it is not restricted0 T"ere is no need to configure greater num-er virtual rocessors t"an t"e "ysical rocessors in t"e ool0 *or e7am le$ A s"ared ool is configured wit" <C "ysical cores0 (ig"t SP8PARs are configured in t"is case eac" 8PAR can "ave u to <C virtual rocessors0 T"is allows a 8PAR t"at "as <C virtual rocessors to get <C c us if all t"e ot"er l ars are not using t"eir entitlement0 Setting K <C virtual rocessors is not necessary as t"ere are only <C !PAs in t"e ool0 1.2.2 Entitlement vs. Virtual processors (ntitlement is t"e ca acity t"at a SP8PAR is guaranteed to get as its s"are from t"e s"ared ool0 Anca ed mode allows a artition to receive e7cess cycles w"en t"ere are free EunusedF cycles in t"e system0 (ntitlement also determines t"e num-er of SP8PARs t"at can -e configured for a s"ared rocessor ool0 T"at is, t"e sum of t"e entitlement of all t"e SP8PARs cannot e7ceed t"e num-er of "ysical cores configured in a s"ared ool0
age =

IBM STG !ross Platform Systems Performance

*or e7am le$ S"ared ool "as 9 cores, 2: SP8PARs are created eac" wit" ;02 core entitlement and 2 virtual !PA0 'e configured t"e artitions wit" ;02 core entitlement since t"ese artitions are not running t"at freLuently0 #n t"is e7am le, t"e sum of t"e entitlement of all t"e 2: SP8PARs comes to 20: cores0 T"e rest of :01 cores and any unused cycles from t"e 20: entitlement can -e dis atc"ed as unca ed cycles0 At t"e same time .ee ing entitlement low w"en t"ere is ca acity in t"e s"ared ool is not always a good ractice0 Anless t"e artitions are freLuently idle or t"ere is lan to add more artitions, t"e -est ractice is t"at t"e sum of t"e entitlement of all t"e SP8PARs configured s"ould -e close to t"e ca acity in t"e s"ared ool0 (ntitlement cycles are guaranteed, so w"ile a artition is using its entitlement cycles, t"e artition is not re6em ted, w"ereas a artition can -e reem ted w"en it is dis atc"ed to use e7cess cycles0 *ollowing t"is ractice allows t"e "y ervisor to o timi,e t"e affinity of t"e artitionHs memory and rocessors and also reduces unnecessary reem tions of t"e virtual rocessors0 1.2.3 Matching entitlement of a LPA close to its average utilization for !etter performance T"e aggregate entitlement Emin/desired rocessorF ca acity of all 8PARs in a system is a factor in t"e num-er of 8PARs t"at can -e allocated0 T"e minimum entitlement is w"at is needed to -oot t"e 8PARs, "owever t"e desired is w"at an 8PAR will get if t"ere are enoug" resources availa-le in t"e system0 T"e -est ractice for 8PAR entitlement would -e to matc" t"e entitlement ca acity to average utili,ation and let t"e ea. addressed -y additional unca ed ca acity0 T"e rule of t"um- would -e setting entitlement close to average utili,ation for eac" of t"e 8PAR in a system, "owever t"ere are cases w"ere a 8PAR "as to -e given "ig"er riority com ared to ot"er 8PARs in a system, t"is rule can -e rela7ed0 *or e7am le if t"e roduction and non6 roduction wor.loads are consolidated on t"e same system, roduction 8PARs would -e referred to "ave "ig"er riority over non6 roduction 8PARs0 #n w"ic" case, in addition to setting "ig"er weig"ts for unca ed ca acity, t"e entitlement of t"e roduction 8PARs can -e raised w"ile reducing t"e entitlement of non6 roduction 8PARs0 T"is allows t"ese im ortant roduction 8PARs to "ave -etter artition lacement EaffinityF and t"ese 8PARs will "ave additional entitled ca acity so not to rely solely on unca ed rocessing0 At t"e same time if roduction SP8PAR is not using t"eir entitled ca acity, t"en t"at ca acity can -e used -y non6 roduction SP8PAR and t"e non6 roduction SP8PAR will -e re6em ted if roduction SP8PAR needs its ca acity0 1.2." #hen to a$$ a$$itional virtual processors '"en t"ere is sustained need for a s"ared 8PAR to use additional resources in t"e system in unca ed mode, increasing virtual rocessors are recommended0 1.2.% &o' to estimate the num!er of virtual processors per uncappe$ share$ LPA

age H

IBM STG !ross Platform Systems Performance

T"e first ste would -e to monitor t"e utili,ation of eac" artition and for any artition w"ere t"e average utili,ation is M2;;N, and t"en add one virtual rocessors0 T"at is, use t"e ca acity of t"e already configured virtual rocessors -efore adding more0 Additional virtual rocessors are going to run concurrently if t"ere are enoug" free rocessors availa-le in t"e s"ared ool0 #f t"e ea. utili,ation is well -elow 5;N mar., t"en t"ere is no need for additional virtual rocessors0 #n t"is case, loo. at t"e ratio of virtual rocessors to configured entitlement and if t"e ratio is K 2, t"en consider reducing t"e ratio0 #n any case if t"ere are too many virtual rocessors configured, A#% can IfoldJ t"ose rocessors so t"at t"e wor.load would run on fewer virtual rocessors to o timi,e virtual rocessor erformance0 *or e7am le if a SP8PARs "as given a !PA entitlement of C0; cores and 1 virtual rocessors in an unca ed mode t"en t"e "y ervisor could dis atc" t"e virtual rocessors to 1 "ysical cores concurrently if t"ere are free cores availa-le in t"e system0 T"e SP8PARs leverages unused cores and t"e a lications can scale u to 1 cores0 However if t"e system does not "ave free cores t"en "y ervisor would "ave to dis atc" 1 virtual rocessors on C cores so t"e concurrency is limited to C cores0 #n t"is situation, eac" virtual rocessor is dis atc"ed for reduced time slice as C cores are s"ared across 1 virtual rocessors0 T"is situation could im act erformanceO t"erefore A#% o erating system rocessor folding su ort may -e a-le to reduce to num-er of virtual rocessors -eing dis atc" suc" t"at only C or < virtual rocessors are dis atc"ed across t"e C "ysical0 1.2.( Virtual Processor Management ) Processor *ol$ing A#% o erating system monitors t"e utili,ation of eac" virtual rocessor and aggregate utili,ation of a SP8PAR, if t"e aggregate utili,ation goes -elow 13N, A#% will start folding down t"e virtual !PAs so t"at fewer virtual !PAs will -e dis atc"ed0 T"is "as t"e -enefit of a virtual !PAs running longer -efore getting re6em ted w"ic" "el s to im rove erformance0 #f a virtual !PA gets lower dis atc" time slice, t"en more wor.loads are time6sliced on to t"e rocessor w"ic" can cause "ig"er cac"e misses0 #f t"e aggregate utili,ation of a SP8PAR goes -eyond 13N, A#% will start unfolding virtual !PAs so t"at additional rocessor ca acity can -e given to t"e SP8PAR0 Virtual rocessor management dynamically ado ts num-er of virtual rocessors to matc" t"e load on a SP8PAR0 T"is t"res"old Ev m=fold=t"res"oldF of 13N re resents t"e SMT t"read utili,ation starting A#% :02 T8:, rior to t"at v m=fold=t"res"old Ew"ic" was set to D;NF re resented t"e core utili,ation0

'it" a v m=fold=t"res"old value of 13N, t"e rimary t"read of a core is fully utili,ed -efore unfolding anot"er virtual rocessor to consume anot"er core from t"e s"ared ool on P&'(RD
age ?

IBM STG !ross Platform Systems Performance

systems0 #f free cores are availa-le in t"e s"ared rocessor ool, t"en unfolding anot"er virtual rocessor would result in t"e 8PAR getting anot"er core along wit" its associated cac"es0 As a result now t"e SP8PAR can run on two rimary t"reads of two cores instead of two t"reads E rimary and secondaryF on t"e same core0 'or.load running on two rimary t"reads of two cores would "ave "ig"er erformance t"an t"e wor.load running on rimary and secondary t"reads of a single core0 A#% virtual rocessor management default olicy aims at ac"ieving "ig"er erformance so it unfolds virtual rocessors using only t"e rimary t"read until all t"e virtual rocessors are unfolded t"en it starts leveraging SMT t"reads0 #f t"e system is "ig"ly utili,ed and t"ere are no free cycles in t"e s"ared ool, w"en all t"e SP8PARs in t"e system try to get more cores -y unfolding additional virtual rocessors and use only t"e rimary of t"read of eac" core and leverage ot"er t"ree t"reads only w"en load increases, t"e fact t"at all t"e virtual rocessors are unfolded lead to "y ervisor time slicing t"e "ysical cores across multi le virtual rocessors0 T"is would im act erformance of all t"e SP8PARs as time slicing a core across multi le virtual rocessors would increase cac"e misses, conte7t switc" cost etc0, #n suc" a situation unfolding fewer virtual rocessors so t"at a "ysical core is eit"er not s"ared or s"ared across fewer virtual rocessors would im rove overall system erformance0 A#% levels 61TL8 and 71TL2 rovides a new dis atc"ing feature t"roug" t"e sc"edo tunea-le v m=t"roug" ut=mode, w"ic" allows greater control over wor.load dis atc"ing0 T"ere are four o tions, ;,2,C,1 t"at can -e set dynamically0 Mode; and 2 cause t"e A#% artition to run in raw t"roug" ut mode, and modesC and 1 switc" t"e artition into scaled t"roug" ut mode0 T"e default -e"avior is raw t"roug" ut mode, same as legacy versions of A#%0 #n raw t"roug" ut mode Ev m=t"roug" ut=mode?;F, t"e wor.load is s read across rimary SMT t"reads0 (n"anced raw t"roug" ut mode Ev m=t"roug" ut=mode?2F, -e"aves similar to mode; -y utili,ing rimary SMT t"readsO "owever, it attem ts to lower !PA consum tion -y slig"tly increasing t"e unfold t"res"old olicy0 #t ty ically results in a minor reduction in !PA utili,ation0 Please note t"at t"is is different to c"anging t"e v m=fold=t"res"old tuna-le0 T"e new -e"avior c"oices are for scaled t"roug" ut mode SMTCEv m=t"roug" ut=mode?CF and scaled t"roug" ut mode SMT1Ev m=t"roug" ut=mode?1F0 T"ese new o tions allow for t"e wor.load to -e s read across C or 1 SMT t"reads, accordingly0 T"e t"roug" ut modes determine t"e desired level of SMT e7 loitation on eac" virtual rocessor core -efore unfolding anot"er core0 A "ig"er value will result in fewer cores -eing unfolded for a given wor.load0

age @

IBM STG !ross Platform Systems Performance

T"e A#% sc"eduler is o timi,ed to rovide -est raw a lication t"roug" ut on P&'(RD / P&'(RD40 Modifying t"e A#% dis atc"ing -e"avior to run in scaled t"roug" ut mode will "ave a erformance im act t"at varies -ased on t"e a lication0 T"ere is a correlation -etween t"e aggressiveness of t"e scaled t"roug" ut mode and t"e otential im act on erformanceO a "ig"er value increases t"e ro-a-ility of lowering a lication t"roug" ut and increasing a lication res onse times0 T"e scaled t"roug" ut modesO "owever, can "ave a ositive im act on t"e overall system erformance -y allowing t"ose artitions using t"e feature to utili,e less !PA Eunfolding fewer virtual rocessorsF, and t"us reducing t"e overall demand on t"e s"ared rocessing ool in a over rovisioned virtuali,ed environment0 T"e tune6a-le is dynamic and does not reLuire a re-ootO w"ic" sim lifies t"e a-ility to revert t"e setting if it does not result in t"e desired -e"avior0 #t is recommended t"at any e7 erimentation of t"e tune6a-le -e done on non critical artitions suc" as develo ment or test -efore -eing considered for roduction environments0 *or e7am le, a critical data-ase SP8PAR would -enefit most from running in t"e default raw t"roug" ut mode, utili,ing more cores even in a "ig"ly contended situation to ac"ieve t"e -est erformanceO "owever, t"e develo ment and test SP8PARs can sacrifice -y running in a scaled t"roug" ut mode, and t"us utili,ing fewer virtual rocessors -y leveraging more SMT t"reads er core00 However w"en a 8PAR utili,ation
age G

IBM STG !ross Platform Systems Performance

reac"es ma7imum level, A#% dis atc"er would use all t"e SMT1 t"reads of all t"e virtual rocessors irres ective of t"e mode settings0 T"e e7am le -elow illustrates t"e -e"avior wit"in a raw t"roug" ut mode environment0 Botice "ow t"e a lication t"reads are dis atc"ed on rimary rocessor t"reads, and t"e secondary and tertiary rocessor t"reads are utili,ed only w"en t"e a lication t"read count e7ceeds 9 and 2: accordingly0

1.2.+ Processor ,in$ings in Share$ LPA 'it" A#% :02 T85 and A#% D02 -inding rocessors is availa-le to a lication running in a s"ared 8PAR0 An a lication rocess can -e -ound to a virtual rocessor in a s"ared 8PAR0 #n a s"ared 8PAR a virtual rocessor is dis atc"ed -y PowerVM "y ervisor0 #n firmware level D<; and later, t"e PowerVM "y ervisor maintains t"ree levels of affinity for dis atc"ing, core, c"i and node level0 By maintaining affinity at "y ervisor level as well as in A#%, a lications may ac"ieve "ig"er level affinity t"roug" rocessor -indings0

1.3 Recommendation on page table size for LPAR

age 10

IBM STG !ross Platform Systems Performance

T"e "ardware age ta-le of a 8PAR is si,ed -ased on t"e ma7imum memory si,e of a 8PAR not w"at is currently assigned EdesiredF to t"e 8PAR0 T"ere are some erformance considerations if t"e ma7imum si,e is set significantly "ig"er t"an t"e desired memory0 20 8arger age ta-le tends to "el erformance of t"e wor.load as t"e "ardware age ta-le can "old more ages0 T"is will reduce translation age faults0 T"erefore if t"ere is enoug" memory in t"e system and would li.e to im rove translation age faults, set your ma7 memory to a "ig"er value t"an t"e 8PAR desired memory0 C0 &n t"e downside, more memory is used for "ardware age ta-le, t"is not only wastes memory -ut also ma.es t"e ta-le -ecome s arse w"ic" results in a cou le of t"ings$ a0 dense age ta-le tends to "el -etter cac"e affinity due to reloads -0 lesser memory consumed -y "y ervisor for "ardware age ta-le more memory is made availa-le to t"e a lications c0 lesser age wal. time as age ta-les are small

1.4 Recommendation for placing LPAR resources to attain higher memor affinit
P&'(RD PowerVM "as o timi,ed allocation of resources for -ot" dedicated and s"ared artitions as eac" 8PAR is activated0 Pro er lanning of 8PAR configuration would en"ance t"e ossi-ility of getting -ot" !PA and Memory in t"e same domain in relation to t"e to ology of a system0 PowerVM Hy ervisor selects t"e reLuired rocessors and memory configured for a 8PAR from t"e system free resource ool0 During t"is selection rocess "y ervisor ta.es t"e to ology of t"e system into consideration and allocates rocessors and memory w"ere -ot" resources are in close ro7imity as -est as it can0 T"is ensures t"at t"e wor.load on a 8PAR would "ave lower latency in accessing its memory0 1.".1 #hat $oes the SPPL option $o on Po'er +-% s.stem T"e S"ared Partition Processor 8imit ESPP8F attri-ute rovides "ints to "y ervisor w"et"er to contain artitions to minimum domains or s read artitions across multi le domains0 &n Power D35, a -oo. can "ost 1 c"i s totaling u to <C cores0 #f SPP8 is set to <C, t"en t"e ma7imum si,e of a 8PAR t"at can -e su orted is <C cores0 T"is "int ena-les "y ervisor to allocate -ot" "ysical cores and memory of a 8PAR wit"in a single domain as muc" as ossi-le0 *or e7am le in a t"ree -oo. configuration if t"e desired configuration is 1 8PARs eac" wit" C1 cores, < of t"ose 8PARs will -e contained in eac" of t"e t"ree -oo.s and t"e 1t" 8PAR would -e s read across < -oo.s #f SPP8 is set to MA% t"en a artition si,e can e7ceed <C cores0 T"is "int "el s "y ervisor to ma7imi,e t"e interconnect -andwidt" allocation -y s reading 8PARs across more domains as ossi-le for larger si,e 8PARs0

age 11

IBM STG !ross Platform Systems Performance

Bote t"at t"e SPP8 value can only -e set on D35 systems t"at contain 1 or more rocessing -oo.s w"ere Tur-o!ore mode is disa-led0 #f t"ere are < or less rocessor -oo.s, t"e SPP8 setting is controlled -y t"e system firmware0 &n a Power D35 system w"ere t"e SPP8 value is set to ma7, t"ere is a way to configure individual artitions to still -e ac.ed into minimum num-er of -oo.s0 T"is is ac"ieved -y using t"e HM! command line interface t"roug" t"e l ar= lacement rofile attri-ute on t"e c"syscfg command0 S ecifying a value of l ar= lacement?2 indicates t"at t"e "y ervisor s"ould try and minimi,e t"e num-er of domains assigned to t"e l ar0 Setting it to ; is t"e default setting and follows t"e e7isting rules w"en SPP8 is set to ma70 #n t"e D<; level of firmware, l ar= lacement?2 was only recogni,ed for dedicated rocessor artitions w"en SPP8?MA%0 Starting wit" t"e D:; firmware level, l ar= lacement?2 is also recogni,ed for s"ared rocessor artition wit" SPP8?MA% and systems configured to run in Tur-o!ore mode0

1.".2 &o' to $etermine if a LPA is containe$ 'ithin a $ra'er or !oo/ *rom an A#% 8PAR, lssrad command can -e used to dis lay t"e num-er of domains a 8PAR is s read0 T"e lssrad synta7 is$ lssrad -av #f all t"e cores and memory are located in a single -oo. REF1 SRAD MEM CPU 0 0 31806.31 0-31 1 31553.75 32-63 R(*2 is t"e second level domain0 #n t"e case of Power D35 t"is second level domain refers to -oo.0 *or Power DD;/D9; t"is second level domain refers to t"e drawer 0 SRAD refers to rocessor soc.et num-ers0 However IlssradJ does not re ort t"e actual "ysical domains Enamely -oo. num-ers and c"i num-ersF0 T"e out ut of lssrad indicates t"at t"e 8PAR is allocated 2: cores from two c"i s in t"e same -oo./drawer0 1.".3 0ptimizing esource Placement PowerVM w"en all t"e resources are free Ean initial mac"ine state or re-oot of t"e !(!F will allocate memory and cores as o timal as ossi-le0 At t"e artition -oot time, PowerVM is aware of all t"e 8PAR configurations so, lacement of rocessors and memory are irres ective of t"e order of activation of t"e 8PARs0 However after t"e initial configuration, t"e setu may not stay static0 T"ere will -e numerous o erations t"at ta.e lace suc" as$
age 12

IBM STG !ross Platform Systems Performance

20 Reconfiguration of e7isting 8PARs wit" new rofiles C0 Reactivating e7isting 8PARs and re lacing t"em wit" new 8PARs <0 Adding and removing resources to 8PAR dynamically ED8PAR o erationsF #f t"e current lacement is causing a erformance issue, t"ere are a cou le rocedures t"at can -e used to im rove lacement of t"e artitions across t"e server0 20 #f t"e Dynamic Platform & timi,er is availa-le on your server, t"is is t"e most efficient and least disru tive met"od of im roving t"e lacement0 Refer to 20101 & timi,ing Resource Placement ) Dynamic Platform & timi,er for more information a-out t"is o tion0 C0 '"en t"e Dynamic Platform & timi,er is not availa-le, t"e most effective -ut also t"e most disru tive met"od to im rove lacement is a server re-oot0 T"e rocedure to im rove t"e lacement t"roug" a server re-oot is as follows$ a0 Ma.e any rofile u dates t"at you need to ma.e to t"e artitions0 -0 Power off all t"e defined artitions c0 Activate E ower onF all t"e artitions s ecifying t"e rofiles wit" t"e desired attri-utes0 T"e artition need to -e activated to at least t"e SMS rom t0 d0 &nce all artitions are simultaneously activated wit" t"e u dated rofiles Ei0e0 every artition must -e activeF, start owering off all of t"e artitions0 e0 Power off t"e server f0 Power on t"e server g0 Activate t"e artitions from t"e management console0 &rder of activation is not im ortant0 <0 #f you are una-le to erform a server re-oot and all t"e rocessors in t"e system are licensed, you can im rove t"e lacement as follows$ a0 Power off all t"e defined artitions -0 Ase one of t"e following met"ods to free u all t"e resources assigned to all of t"e artitions$ i0 Ase t"e HM! command c""wres to remove all rocessors and memory from all of t"e artitions0 T"e command synta7 to remove t"e memory is Ic""wres )r mem )m Psystem nameK 6o r )L Pnum-er of M-tyesK 66id Pl ar idKJ0 T"e command synta7 to remove rocessor units from a s"ared rocessor artition is Ic""wres )r roc )m Psystem nameK 6o r ) rocunits Pnum-erK 66id Pl ar idKJ0 T"e management console can dis lay/re ort t"e resources assigned0 Before roceeding to t"e ne7t ste , confirm t"at no resources are assigned to any artition0 ii0 !reate a artition and indicate t"e artition rofile s"ould own all of t"e resources in t"e system0 Activate t"is artition wit" t"e all resource rofile to at least SMS menu0 Power off t"is artition and t"en delete t"is artition0
age 1F

IBM STG !ross Platform Systems Performance

c0 Activate t"e artitions in order of im ortance0 T"e largest, most im ortant artition s"ould -e activated first, ne7t most im ortant artition second and so on0 #f you wis" to try to im rove t"e lacement of a single artition, ower off t"e artition, use t"e c""wres command as descri-ed a-ove to remove all t"e rocessor and memory resources from t"e artition and t"en activate t"e artition wit" desired rofile0 T"e im rovement Eif anyF in t"e lacement will de end on a variety of factors suc" as w"ic" rocessors are free, w"ic" memory is free, w"at licensed resources are availa-le, configuration of ot"er artitions and so on0 *ragmentation due to freLuent movement of memory or rocessors -etween artitions can -e avoided t"roug" ro er lanning a"ead of t"e time0 D8PAR actions can -e done in a controlled way so t"at t"e erformance im act of resource addition/deletion will -e minimal0 Planning for growt" would "el to alleviate t"e fragmentation caused -y D8PAR o erations0 Qnowing t"e 8PARs t"at would need to grow or s"rin. dynamically and lacing t"em wit" 8PARs t"at can tolerate nodal crossing latency Eless critical 8PARsF would -e one a roac" to "andle t"e c"anges of critical 8PARs dynamically0 #n suc" a configuration, w"en growt" is needed for t"e critical 8PAR t"e resources assigned to t"e non6critical 8PAR can -e reduced so t"at critical 8PAR can grow0 1."." 0ptimizing esource Placement 1 2.namic Platform 0ptimizer #n firmware D:; level and later, on select Power servers, a feature is availa-le called t"e Dynamic Platform & timi,er0 T"is o timi,er automates t"e reviously descri-ed manual ste s to im rove lacement0 T"e following functions are availa-le on t"e HM! command line$ Ilsmemo t )m Psystem=nameK 6o currscoreJ will re ort t"e current affinity score for t"e server0 T"e score is a num-er in t"e range of ;62;; wit" ; -eing oor affinity and 2;; -eing erfect affinity0 Bote t"at t"e affinity score is -ased on "ardware c"aracteristics and artition configurations so a score of 2;; may not -e ac"ieva-le0 Ilsmemo t )m Psystem=nameK 6o calcscoreJ will re ort t"e otential score t"at could -e ac"ieved -y o timi,ing t"e system wit" t"e Dynamic Platform & timi,er0 Again, a calculated score of 2;; may not -e ossi-le0 T"e scoring is meant to rovide a gauge to determine if running t"e o timi,er is li.ely to rovide im roved erformance0 *or e7am le, if t"e current score is 95 and t"e calculated score is 3;, running t"e o timi,er may not "ave a noticea-le im act on overall systems erformance0 T"e amount of gain from doing an o timi,ation is de endent on t"e a lications running wit"in t"e various artitions and t"e artition lacement0 Also note t"at t"is is a system6wide score t"at reflects all of t"e resources assigned to all of t"e artitions0 T"e lacement of individual artitions contri-ute to t"e system wide score relative to t"e amount of resources assigned to t"e artition Elarge rocessor/memory artition contri-utes more to t"e system wide score t"an small artitionF0 Ma.ing any configuration c"ange Eeven activating a artition t"at wasnHt assigned resources reviouslyF can c"ange t"e overall score0

age 1>

IBM STG !ross Platform Systems Performance

Io tmem )m Psystem=nameK 6o start )t affinityJ will start t"e o timi,ation for all artitions on t"e entire server0 T"e time t"e o timi,ation ta.es is de endent u on t"e current lacement of artitions, t"e overall amount of rocessor and memory t"at need to -e moved, t"e amount of !PA cycles availa-le to run t"is o timi,ation and so on0 Ilsmemo t )m Psystem=nameKJ dis lays t"e status of t"e o timi,ation as it rogresses0 Io tmem )m Psystem=nameK 6o sto J will end an o timi,ation -efore it "as com leted all t"e movement of rocessor and memory0 T"is can result in affinity -eing oor for some artitions t"at were not com leted0 T"ere are additional arameters availa-le on t"ese commands t"at are descri-ed in t"e "el te7t on t"e HM! command line interface E!8#F commands0 &ne o tion on t"e o tmem command is t"e e7clude arameter E67 or )7idF w"ic" is a list of artition t"at s"ould not -e o timi,ed Eleft as is wit" regards to memory and rocessorsF0 T"e include o tion E6 or )idF is not a list of artitions to o timi,es as it may a ear, it is a list of artitions t"at are o timi,ed first followed -y any unlisted artitions and ignoring any e7 licitly e7cluded artitions0 *or e7am le, if you "ave artition ids 265 and issue Io tmem )m myserver )o start )t affinity )7id 1 )id CJ, t"e o timi,er would first o timi,e artition C, t"en from most im ortant to least im ortant artitions 2, < and 50 Since artition 1 was e7cluded, its memory and rocessors remain intact0 Some functions suc" as dynamic l ar and artition mo-ility cannot run concurrently wit" t"e o timi,er0 #f one of t"ese functions is attem ted, you will receive an error message on t"e M! indicating t"e reLuest did not com lete0 To run t"e o timi,er t"ere must -e some unlicensed memory installed or availa-le licensed memory0 T"e more free memory Eor unlicensed memoryF availa-le, t"e faster t"e o timi,ation will com lete0 Also, t"e -usier !PAs are on t"e system, t"e longer t"e o timi,ation will ta.e to com lete as t"e -ac.ground o timi,er tries to minimi,e its effect on running 8PARs0 8PARs t"at are owered off can -e o timi,ed very Luic.ly as t"e contents of t"eir memory does not need to -e maintained0 T"e Dynamic Platform & timi,er will s end t"e maRority of t"e time co ying memory from t"e e7isting affinity domain to t"e new domain0 '"en erforming t"e o timi,ation, t"e erformance of t"e artition will -e degraded and overall t"ere will -e a "ig"er demand laced on t"e rocessors and memory -andwidt" as data is -eing co ied -etween domains0 '"en t"e o timi,er com letes t"e o timi,ation, t"e o timi,er can notify t"e o erating systems in t"e artitions t"at "ysical memory and rocessors configuration "ave -een c"anged0 Partitions running on A#% D02 T8C Eor laterF, A#% :02 T89 Eor laterF, V#&S C0C0C0; and #BM i D02 PT* M*5:;59 "ave su ort for t"is notification0 *or older o erating system versions t"at do not su ort t"e notification, t"e dis atc"ing, memory management and tools t"at dis lay t"e affinity can -e incorrect0 *or dis atc"ing and memory management, for some configurations, t"e erformance may actually degrade after running t"e o timi,er even t"oug" t"e artition "as -etter affinity -ecause t"e o erating system is ma.ing decisions on stale information0 *or t"ese
age 1=

IBM STG !ross Platform Systems Performance

older versions, a re-oot of t"e artition will refres" t"e affinity0 Anot"er o tion would -e to use t"e e7clude o tion on t"e o tmem command to not c"ange t"e affinity of artitions wit" older o erating system levels0 Re-ooting artitions is usually less disru tive t"an t"e alternative of re-ooting t"e entire server0

1.".% Affinit. 3roups #n PowerVM *irmware level D<; and later, su ort was added t"at can -e used to lace multi le 8PARs Eallocate resourcesF wit"in a single c"i , drawer or -oo.0 T"e ma7imum si,e of t"e affinity grou is tied directly to t"e "ardware configuration0 *or e7am le, on Power D35 wit" <C cores in a -oo., t"e total "ysical core resources of an affinity grou s"ould not e7ceed <C cores or t"e memory "ysical contained wit"in a -oo.0 T"is affinity grou feature can -e used in multi le situations$ 20 8PARs t"at are de endent or related, suc" as server and client, a lication server and data-ase server, could -e grou ed so t"at t"ey can reside in t"e same -oo.0 C0 Affinity grou s can -e created large enoug" suc" t"at its forces t"e assignment of 8PARs to -e in different -oo.s0 *or e7am le, if you "ave a C -oo. system and t"e total resources Ememory and rocessorsF assigned to t"e two grou s e7ceed t"e ca acity of a single -oo., t"ese two grou s will -e forced to -e laced in se arate -oo.0 A sim le e7am le of t"is is if t"ere is a grou of artitions t"at totals 21 cores and a second grou t"at totals C; cores, since t"ese grou s e7ceed t"e <C cores in a D35 -oo. t"e grou s will -e laced in different -oo.s0 <0 #f a air of 8PAR are created wit" t"e intent of one -eing failover to anot"er artition, assuming t"at one artition fails, t"e ot"er artition w"ic" is laced in t"e same node, if -ot" are in t"e same affinity grou , would use all t"e resources freed u from t"e failed 8PAR0 T"e following HM! !8# command adds or removes a artition from an affinity grou $ chsysc ! -r "ro -# $sys%&#'(a#&) -* (a#&+$"ro *l&'(a#&) l"ar'(a#&+$"ar%*%*o('(a#&),a *(*%y'!ro-"'*d+$!ro-"'*d) w"ere !ro-"'*d is a num-er -etween 2 and C55 EC55 grou s can -e defined 6 a *(*%y'!ro-"'*d+(o(& removes a artition from t"e grou 0

'"en defining affinity grou s containing s"ared rocessors, t"e sum total of t"e rocessor units s"ould -e in full rocessor units0 *or e7am le, if you configured two grou s eac" wit" 905; rocessing units, t"e total rocessors in t"e s"ared ool would -e t"e sum of t"e entitlement or 2D in t"is case0 T"is is going to result in 9 rocessor in one affinity domain and 3 rocessors in t"e ot"er affinity domain0 #n t"is situation, round u eac" affinity grou 0 !"ange t"e entitlement
age 1H

IBM STG !ross Platform Systems Performance

of t"e artitions in t"e grou suc" t"at t"e total entitlement adds u to 30; er grou so t"e rocessors can -e fully allocated from a single domain0 '"en t"e "y ervisor laces resources at frame re-oot or w"en t"e dynamic latform o timi,er is started, t"e "y ervisor first laces all t"e 8PARs in grou C55, t"en t"e 8PARs in grou C51 and so on0 Because of t"is, t"e most im ortant artitions wit" res ect to affinity s"ould -e laced in t"e "ig"est configured grou 0 1.".( Lpar4placement52 #n firmware level D:; and later, a new value was added to t"e e7isting l ar= lacement o tion to rovide additional control over artition lacement0 T"e value of SCH indicates t"at t"e artition memory and rocessors s"ould -e ac.ed into t"e minimum num-er of domains0 #n most cases t"is is t"e default -e"avior of t"e "y ervisor, -ut certain configurations may s read across multi le c"i s/drawers0 *or e7am le, if t"e configuration is suc" t"at t"e memory would fit into a single c"i /drawer/-oo. -ut t"e rocessors donHt fit into a single c"i /drawer/-oo., t"e default -e"avior of t"e "y ervisor is to s read -ot" t"e memory and t"e cores across multi le c"i /drawer/-oo.0 #n t"e revious e7am le, if l ar= lacement?C is s ecified in t"e artition rofile, t"e memory would -e contained in a single c"i /drawer/-oo. -ut t"e rocessors would -e s read into multi le c"i s/drawers/-oo.s0 T"is configuration may rovide -etter erformance in a situation wit" s"ared rocessors w"ere t"e maRority of t"e time t"e full entitlement and virtual rocessors are not active0 #n t"is case t"e "y ervisor would dis atc" t"e virtual rocessor in t"e single c"i /drawer/-oo. w"ere t"e memory resides and only if t"e !PA e7ceeds w"at is availa-le in t"e domain would t"e virtual rocessors need to s an domains0 T"e l ar= lacement?C o tion is availa-le on all server models and a lies to -ot" s"ared and dedicated rocessor artitions0 *or AMS artitions, t"e attri-ute will ac. t"e rocessors -ut since t"e memory is su lied -y t"e AMS ool, t"e l ar= lacement attri-ute does not ac. t"e AMS ool memory0 Pac.ing s"ould not -e used indiscriminately, t"e "y ervisor may not -e a-le to satisfy all l ar= lacement?C reLuests0 Also, for some wor.loads, l ar= lacement?C can lead to degraded erformance0 1.".+ Lpar4placement consi$erations for failover/$isaster recover. Many customers use a air or cluster of systems for recover ur oses w"ere in t"e case of failure of artition or server t"e wor. is offloaded to anot"er server0 T"e lacement of t"ese artitions u on "ysical -oundaries Edrawers/-oo.sF is somet"ing to consider to o timi,e t"e erformance in a failover situation0 T"e first ste in ac"ieving good lacement is to follow t"e ste in I201 Recommendation for lacing 8PAR resources to attain "ig"er memory affinityJ0 T"e entitlement of t"e artitions needs to accurately reflect t"e !PA consum tion under normal load Eie0 if t"e artition is consuming 105 rocessor units on average, artition s"ould -e configured wit" at least 5 VPs and 105 rocessor units and not 5 VPs wit" ;05 rocessor units0 Since t"e "y ervisor uses t"e entitlement to determine "ow muc" !PA a artition will consume, t"e lacement of artitions will not -e o timal if t"e entitlement is undersi,ed0 After ensuring t"e entitlement is correct, t"e
age 1?

IBM STG !ross Platform Systems Performance

ne7t ste is to lace artitions into "ardware domains wit" s are ca acity suc" t"at w"en a failover occurs, t"ere is unused ca acity availa-le in t"e domain0 As an e7am le, lets assume two C6drawer D9; model wit" 2: cores er drawer0 &n server &B( is Partition A2 w"ic" is configured for 2: virtual rocessors EVPsF and 90; cores and Partition B2 also is configured for 2: VPs and 90; cores0 Server T'& is similar configuration wit" artitions AC and BC0 'it"out any ot"er directive, t"e "y ervisor may lace artitions A2 and B2 into t"e same drawer since t"e resources can -e contained wit"in a drawer0 '"en running normally Enon6failoverF, eac" server is only licensed for 2: of t"e <C installed cores0 #n t"e event of a failure of server, t"e transactions t"at were -eing rocessed -y A2 will failover to AC, same for B2 failing over t"e BC0 Also, for failover, all cores in t"e server are activated suc" t"at t"e wor.load can -e contained in a single server0 '"at would -e ideal for lacement in t"e failover scenario would -e t"at artition AC is contained in a drawer wit" 2: c us and similarly BC is contained in a different drawer0 #f all cores were licensed, one way to ac"ieve t"is lacement would -e to configure AC and BC wit" 2:0; entitlement even t"oug" only 90; entitlement is reLuired in t"e normal situation0 Since eac" drawer "as 2: cores, t"e "y ervisor would -e forced to lace t"ese artitions on different drawers0 Anot"er way to ac"ieve t"e desired lacement is t"oug" t"e configuration of memory0 *or e7am le, if eac" drawer "as C5:GB of memory, you could set eac" artition to 90; cores, 2: VPs and C<; GB0 #n t"is situation w"en t"e "y ervisor tries to lace t"e artitions, t"e memory reLuirements of AC and BC force t"e "y ervisor to lace t"e artitions in se arate drawers0 Allocation of memory -eyond 5;N of t"e ca acity of t"e drawers would force t"is lacement s lit0 Situations wit" multi le artitions er drawer can rely on t"e affinity grou su ort to -undle u smaller artitions into a single affinity grou 0 T"e affinity grou , not individual artitions, would -e t"e entity t"at is laced in t"e drawers/-oo.s0 '"en creating t"e grou s, t"e sum total of t"e memory and rocessor units s"ould -e suc" t"at it fits wit"in a drawer/-oo.s and is large enoug" to consume at least 5;N of t"e rocessor or memory suc" t"at multi le grou s could not -e laced in t"e same -oo./drawer0 Also, if s"ared rocessors are -eing utili,ed, t"e total rocessing units for a grou s"ould -e a multi le of full rocessor units0 Some users may consider using t"e Dynamic Platform & timi,er after a failover as a lacement strategy -ut t"ere is a erformance cost involved in running t"e o timi,er0 T"e artitions, as t"e resources are moved -etween domains, run a -it slower and overall t"ere is more !PA and memory -andwidt" demands for t"e system0 #f t"ere is sufficient unused ca acity in t"e system or t"e o timi,ation can -e delayed to a time w"en t"ere is unused ca acity t"en t"e Dynamic Platform & timi,er may "ave a role to lay in some failover scenarios0

1.! Po"erV# resource consumption for capacit planning considerations

PowerVM "y ervisor consumes a ortion of memory resources in t"e systemO during lanning stage ta.e t"at into consideration to lay out 8PARs0 *actors t"at affect t"e amount of memory consumed in t"e si,e of t"e "ardware age ta-les in t"e artitions, #V( resources, H!A resources, num-er of #/& devices, "y ervisor memory mirroring and ot"er factors0 T"e #BM
age 1@

IBM STG !ross Platform Systems Performance

system lanning tool E"tt $//www0i-m0com/systems/su ort/tools/system lanningtool/ F s"ould -e used to estimate t"e amount of memory t"at will -e reserved -y t"e "y ervisor0

1.$ Licensing resources %&'()

Power systems su ort ca acity on6demand w"ere customers can license ca acity on6demand as t"e -usiness needs for com ute ca acity grows0 #n addition to future growt" !&D resources can -e used to rovide erformance im rovement0 *or !&D memory, licensing is managed as an overall total of t"e memory consumed and t"ere is no s ecific licensing of individual logical memory -loc.s E8MBsF or D#MMs0 #f t"ere is !&D memory installed on t"e system, t"is can rovide erformance -enefits -ecause t"e "y ervisor can use all t"e "ysical memory installed to im rove artition lacement0 *or e7am le, on a two drawer system, eac" drawer "as C5:GB of installed memory -ut only <91GB of licensed memory, a C1;GB artition can -e created on one drawer and a 2C;GB artition on t"e second drawer0 #n t"is case, all <91GB is availa-le and can -e divided -etween t"e drawers in a manner t"at rovides t"e -est erformance0 *or !&D rocessor, t"e licensing is on an individual core -asis0 '"en t"ere are fewer licensed cores t"an installed cores, t"e "y ervisor identifies cores to unlicensed and uts t"ese cores into a low ower state to save energy0 #n firmware level D<; and later, during a server re-oot, t"e "y ervisor laces all defined artitions as o timally as ossi-le and t"en unlicenses t"e unused rocessor cores0 #n firmware level D:; and later, w"en activating additional cores t"e "y ervisor dynamically c"anges w"ic" cores are licensed and unlicensed to im rove artition lacement0 T"e dynamic latform o timi,er can -e utili,ed w"en configuration c"anges are made as a result of licensing additional cores and memory to o timi,e t"e system erformance0

age 1G