Consol i dat i ng OLTP

Wor k l oads on
Del l

Power Edge

11G
Ser ver s
ADellTechnicalWhitePaper
Dat abase Solut ions
Engineering
By Zafar Mahmood
Dell Product Group
July 2009
Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
2

THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND
TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF
ANYKIND.
2009 Dell Inc. All rights reserved. Reproduction in any manner whatsoever without the express written
permissionofDell,Inc.isstrictlyforbidden.Formoreinformation,contactDell.
Dell,theDELLlogo,andaretrademarksofDellInc.IntelandCorei7areregisteredtrademarksofIntelCorporation
in the U.S. and other countries. EMC is the registered trademark of EMC Corporation. Oracle is a registered
trademark of Oracle Corporation. Quest Software and Benchmark Factory are registered trademarks of Quest
Software, Inc. Other trademarks and trade names may be used in this document to refer to either the entities
claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of
others.

Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
3

EXECUTIVESUMMARY
TheDellenterpriseportfolioisevolvingtoincorporatebetterperforming,moreenergyefficient,andmore
highlyavailableproducts.WiththeintroductionofDellslatestserverproductline,customershaveanopportunity
toimprovetheirtotalcostofownershipbyconsolidatingdistributedlegacyenvironments.Thisisthesecondwhite
paperdiscussingserverconsolidationonDell11Gproductline.InthepreviouswhitepaperDSSworkloadandits
consolidationonDellPowerEdge11Gserverswasdiscussed:
http://www.dell.com/downloads/global/solutions/database_11g_consolidate.pdf?c=ec&l=en&s=gen
ThiswhitepaperfocusesonOnlineTransactionProcessing(OLTP)workloadsandconsolidation.
DellstrivestosimplifyITinfrastructurebyconsolidatinglegacyproductionenvironmentstoreducedatacenter
complexitywhilestillmeetingcustomersneeds.Thetoolsandproceduresdescribedinthiswhitepapercanhelp
administratorstest,compare,validate,andimplementthelatesthardwareanddatabasesolutionbundles.Dell
establishedtheseproceduresandguidelinesbasedonlabexperimentsanddatabaseworkloadsimulations
performedbytheDellDatabaseSolutionsEngineeringteam.Usingthetoolsandproceduresdescribedinthis
document,customersmaynotonlyselecttheappropriatedatabasesolutionhardwareandsoftwarestack,but
alsooptimizethesolutiontohelpoptimizetotalcostofownershipaccordingtothedatabaseworkloadsthey
choosetorun.Theintendedaudienceofthiswhitepaperincludesdatabaseadministrators,ITmanagers,and
systemconsultants.

Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
4

TableofContents
EXECUTIVESUMMARY.......................................................................................................................................3
TABLEOFCONTENTS..........................................................................................................................................4
INTRODUCTION..................................................................................................................................................5
TESTMETHODOLOGY.........................................................................................................................................6
TESTCONFIGURATION......................................................................................................................................17
RESULTS............................................................................................................................................................18
CONSOLIDATIONFACTOR..............................................................................................................................................18
SUMMARY........................................................................................................................................................21
REFERENCES......................................................................................................................................................21















Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
5

INTRODUCTION
ThiswhitepaperconcentratesonserverconsolidationforOracledatabaserunningOLTPworkloadsonlegacy
platforms.AnenterprisedatabasesystemmayberunningDSS,OLTPoramixedworkload.TheOLTPworkloads
typicallysendthousandsofsmallI/Orequestsfromthedatabaseserverstothebackendstoragesubsystem.The
largeamountofI/OrequestscharacteristicoftheOLTPworkload,meansthatthebackendstoragesubsystem
musthavesufficientnumberofdiskstohandletheI/Orequestscomingfromthehosts.Atypical15KRPMdiskcan
servicearound180200IOrequestspersecond(IOPS).
OLTPdatabasesystemstypicallyservicehundredsorthousandsofconcurrentusers.Anexampleofthistypeof
systemcouldbeofatravelreservationsystemwithlargenumberofcustomersandagentsperformingonline
travelreservations,orcheckingavailableflightsorflightschedules.TheOLTPdatabasetransactionsperformedby
thesethousandsofconcurrentusersgettranslatedintotensofthousandsofI/Orequeststothebackendstorage
subsystemdependingonthenatureoftheseOLTPtransactions.Forexample,anOracleAWRreportrevealsthata
typicalTPCCtransactionresultsinapproximately70PhysicaldatabaseI/Orequestsifthedatabasesizeisaround
300GBandlessthan1%ofthedataisinOracleSystemGlobalArea(SGA)cache.ThedatabasehostCPUsmayonly
beefficientlyutilizedifthebackendstoragesubsystemisconfiguredwithasufficientnumberofdiskstohandle
thelargenumberofI/Orequests.OtherwisethedatabasehostCPUsexhibitlargeIOWAITtimesinsteadofdoing
usefulwork.Inthisscenario,consolidating,upgradingormigratingtoafasterdatabaseserver,orscalingthe
numberofCPUsormemorydoesnothelp.Thecorrectapproachistoappropriatelyscalethebackenddisk
subsystemtohandletheI/Orequests,andthenmovetothenextstageofCPUandmemorysizingaswewill
discusslaterinthiswhitepaper.
Serverconsolidationcanbedefinedasmaximizingtheefficiencyofcomputerserverresources,therebyminimizing
theassociatedpower/cooling,rackfootprintandlicensingcosts.Itessentiallysolvesafundamentalproblem
calledserversprawlinwhichmultiple,underutilizedserverstakeupmorespaceandconsumemorepower
resourcesthantheworkloadrequirementindicates.
ConsideratwonodeOracleRACdatabasehostedontwoeighthgeneration(8G)PowerEdge2850dualsocket,
singlecoreordualcoreserversrunningOracle10gRelease2.Dellrecentlyannouncedtheavailabilityofits11G
serverproductlineequippedwithachipsetthatisdesignedtosupporttheIntelXeon5500seriesprocessors,
QuickPathInterconnect,DDR3memorytechnology,andPCIExpressGeneration2.Thenaturalreplacementof
eighthgeneration2UDellserversis2UDellR710serversthatsupportdualsocket,quadcoreprocessors.The
R710alsosupportstwodifferenttypesofenergyefficientCPUs,anditisdesignedwithahighlyefficientoverall
architecture.
ThegoalofthisstudyistodetermineifamultinodeOracleRACclustercanbereplacedwithaclusterconsistingof
fewerPowerEdge11Gnodes,andstillprocesstheOLTPworkloadfasterwithlesspowerconsumptionandlower
OracleRAClicensingfees.ThesavingsinRAClicensingfeesmaybeutilizedtoefficientlyconfigureandscalethe
backendstoragesystemwithenoughI/OmodulesanddiskstoremovetheI/Obottlenecksthatarealmostalways
anissueinanOLTPenvironment.Also,basedontheresultsofthisstudy,onemaydeterminehowmany
distributedstandalonelegacyenvironmentsrunningOLTPworkloadscanbeconsolidatedonasingleOracleRAC
solutionrunningonDellR710servers.

Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
6

Figure 1: System Architecture
TESTMETHODOLOGY
DellssolutionengineersusedQuestSoftwareBenchmarkFactoryTPCCworkloadtotestthelegacysystem,and
thenreranasimilarworkloadonatestenvironmentrunningthePowerEdge11Gservers.TheTPCCworkload
providedbytheBenchmarkFactoryschemasimulatesanorderentrysystemconsistingofmultiplewarehouses,
withdatapopulatedintableswithrowsaccordingtothescalefactordefinedduringtablecreation.Thelegacy
databasewasconfiguredwithascalefactorof3000thatcreated900Millionand300MillionrowsinNewOrder
andStocktablesrespectively.Thetotaldatabasesizethatresultedwiththisscalefactorwasaround290GB.Once
populated,westartedwith200concurrentusersandincreasedtheuserloadto1000inincrementsof200users
randomlyrunningtransactionsagainstthelegacydatabasewhilemakingsurethattheaveragequeryresponse
timealwaysstaysbelow2seconds.AveragequeryresponsetimeofanOLTPdatabaseenvironmentmaybe
describedastheaveragetimeittakesforanOLTPtransactiontocompleteanddelivertheresultsofthe
transactiontotheenduserinitiatingthattransaction;thisresponsetimemetricwaschosenasthebasisforour
ServiceLevelAgreementthatwechosetomaintainthroughoutourtesting.Theaveragequeryresponsetimeis
themostimportantfactorwhenitcomestofulfillingenduserrequirements,anditestablishestheperformance
criteriaforanOLTPdatabase.ThebackendstoragesubsystemconsistingofaDell/EMCCX4960storagearray
wasconfiguredwith1015KRPM136GBdisksinRAID10configuration.
Thetestmethodologyusedisasfollows:
1. Tosimulatethelegacyproductionenvironment,weselectedatwonodeOracle10gR2RACcluster
runningontwoPowerEdge2850singlecore,dualsocket3.4GHzCPUmachinesconnectedtoaCX4960
thathada400GBLUNforDATAanda100GBLUNforthedatabaseSYSTEMASMdiskgroups.Wealso
createda2GBLUN,andcreatedpartitionstohostthevotingandOracleClusterRegistry(OCR).
Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
7

2. WeappliedtheOracle10gR2patchset4(10.2.0.4)tothelegacyserversimulatedproduction
environment.
3. WeloadedTPCCschematestdatawithascalefactorof3000intothelegacyserversimulatedproduction
environment.
4. Afterdatapopulation,weusedtheOracleDataPumptoexportdataattheschemaleveltoavoidadata
reloadaftereachtestiteration.
expdp syst em/ or acl e@r acdb1 SCHEMAS=quest CONTENT=al l
di r ect or y=expor t ;
5. WeranourfirsttestiterationonthelegacyRACenvironmentstartingwithauserloadof200.Theuser
loadwasincreasedin200userincrementswhileconstantlymonitoringtheaveragequeryresponsetime.
Oncetheaveragequeryresponsetimereachedabove2seconds,thetestwasstopped.
6. InanOLTPenvironment,oncethebackendspindlesaresaturatedbeyond200IOPS,theystartexhibiting
largeI/OlatencywhichresultsinlargeIOWAITatthehostCPUandalargeaveragequeryresponsetime.
Onceourlegacyenvironmentreachedanaveragequeryresponsetimeofmorethan2secondswe
decidedtodoublethenumberofspindlesforourDATAASMdiskgroup,rebalanceTPCCdataacross
additionaldisksandperformanotheriterationofteststoseeifwecanlowertheaveragequeryresponse
below2secondswithauserloadhigherthan1000.
7. WetookanAutomaticWorkloadRepository(AWR)snapshotofdatabaseactivityinthelegacyproduction
environmentwhilerunningthepeakuserloadforlateranalysis.
8. UsingtheQuestBenchMarkFactory,wepopulatedtheOracle11gsinglenodeenvironmentrunningon
thetestenvironmentwiththesameTPCCscalefactorandbackenddiskconfigurationaswedidonthe
legacyenvironment.WeusedaPowerEdgeR710servertosimulateourtestenvironment.
9. Again,thesameuserloadwasrunonthetestenvironmenttodeterminethetransactionspersecondand
theaveragequeryresponsetimeofthe11Gtestenvironment.Theaverageresponsetimewasthen
comparedagainstthelegacyproductionenvironment.Again,withthebaseconfigurationconsistingof10
disksforDATAdiskgroup,theaveragequeryresponsetimecrossedourpredefinedSLAof2secondsat
1000userload.
10. Similartostep6,the11Gtestenvironmentbackenddisksweredoubledandincorporatedintothe
existingDATAASMdiskgroup.Anotheriterationoftestswasperformedtodetermineifwecansupporta
higheruserloadwhilekeepingtheaveragequeryresponsetimebelow2seconds.
11. Wedecidedthatifthememoryorthedatadisksofthetestenvironment11Gdatabaseserverbecamea
bottleneck,theywillbescaledfurtherandadditionaluserloadwouldbeapplieduntilthe11Gserver
CPUsbecomethebottleneck.Forthispurpose,the11Gtestenvironmentwastestedwithadditional
memoryconfigurationsof18GBand36GBtosupporttheadditionaluserloadwhilestayingbelowthe2
secondsresponsetime.Similarly,thebackendspindleswereagainscaledwithanadditional10spindles
tomakeatotalof30datadiskstobringdowntheresponsetimetolessthanorequalto2secondsif
disksbecamethebottleneck.

Figure2and3belowshowsacomparisonintermsoftransactionspersecondandaveragequery
responsetimebetweenthelegacyproductionandthe11Gtestenvironmentsusingthebase
configurationof10disksRAID10ASMdiskgroup.

Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
8

Figure 2: Base configuration TPS comparison between legacy and 11G environment

Figure 3: Base configuration Average Query Response Time comparison between legacy and 11G environment

InFigure2and3,weseethattheboththelegacyandthe11Genvironmentsexhibitsimilarperformance
intermsoftransactionspersecondandtheaveragequeryresponsetime.Donotbemisledbythese
results.UponfurtheranalysisoftheCPUutilizationintermsofUSERtimeandIOWAITtimes,itwas
revealedthatthelegacyproductionenvironmentwasexhibitinghigherUSERtimetoIOWAITtimeratioas
comparedtothe11GtestenvironmentasshowninFigure4and5.

0
10
20
30
40
50
200 400 600 800 1000
T
P
S
UserLoad
TPSlegacy10disks
TPSR71010disks
0
0.5
1
1.5
2
2.5
200 400 600 800 1000
A
Q
R
T
(
s
e
c
)
UserLoad
AvgResponseTime
legacy10disks
AvgResponseTime
R71010disks
Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
9

Figure 4: Base configuration CPU behavior for the legacy environment

Figure 5: Base configuration CPU behavior for the 11G test environment

Thesechartsrevealsomethingveryinteresting:the11GtestenvironmenthavingthefasterCPUand
overallmoreefficientdesignwasabletohandletheOLTPworkloadmuchfasterascomparedtothe
legacyproductionenvironment,andexhibitedalowUSERtoIOWAITtimeratioascomparedtothelegacy
productionenvironment(1.7forlegacyvs.0.24for11Gat1000userload).Sincebothenvironmentshad
anidenticalstorageconfiguration,thereasonforhigherIOWAITandlowerUSERCPUtimeonthe11G
testenvironmentwasduetothefasterprocessingpoweravailableonthatenvironmentascomparedto
thelegacyproductionenvironment.Overall,thecharts4and5revealthatinordertotakeadvantageof
0
10
20
30
40
50
60
70
80
90
100
1
9
1
1
8
1
2
7
1
3
6
1
4
5
1
5
4
1
6
3
1
7
2
1
8
1
1
9
0
1
9
9
1
1
0
8
1
1
1
7
1
1
2
6
1
1
3
5
1
1
4
4
1
1
5
3
1
1
6
2
1
1
7
1
1
C
P
U
U
t
i
l
i
z
a
t
i
o
n
Time
%IOWAIT
%USER
0
10
20
30
40
50
60
70
80
90
100
1
8
4
1
6
7
2
5
0
3
3
3
4
1
6
4
9
9
5
8
2
6
6
5
7
4
8
8
3
1
9
1
4
9
9
7
1
0
8
0
1
1
6
3
1
2
4
6
1
3
2
9
1
4
1
2
1
4
9
5
1
5
7
8
C
P
U
U
t
i
l
i
z
a
t
i
o
n
Time
%IOWAIT
%USER
Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
10

thefasterprocessingpowerofthe11Gtestenvironment,weneedtoremovetheI/Obottleneckand
reducetheIOWAITtime.
Thisrevelationledtofurthertestsandanalysis,andwedecidedtoverifyourconclusionsbytryingto
alleviatesomeoftheI/Obottlenecksfrombothourlegacyproductionandthe11Gtestenvironmentsby
doublingthespindlecountforourDATAdiskgroup.Thismethodologytoascertaintheperformance
deltabetweentwoenvironmentsrunningOLTPworkloadscanprovidereliableresultswithoutmaking
hugeinvestmentsintostoragewhichmayberequiredtoremoveIObottleneckstostudydatabasehost
performance.Figure6and7belowshowthetestresultsofonbothenvironmentsafterdoublingthe
spindlecount.

Figure 6: 20 disk configuration TPS comparison between legacy and 11G environment

Figure 7: 20 disk configuration Average Query Response Time comparison between legacy and 11G environment
0
10
20
30
40
50
60
200 400 600 800 1000
T
P
S
Userload
TPSlegacy20disks
TPSR71020disks
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
200 400 600 800 1000
A
Q
R
T
(
s
e
c
)
Userload
AvgResponseTime
legacy20disks
AvgResponseTime
R71020disks
Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
11

Asshowninthepreviousfigures,thelegacyproductionenvironmentshowedmarginalimprovementinits
averagequeryresponsetimeevenafterdoublingthespindlecount.Theperformancedeltawasonly
(2.111.83)/1.83=15.3%.Ontheotherhand,the11Gtestenvironmentexhibiteda(2.059
0.486)/0.486=323%performancedelta.Alsonotethatata1000userload,evenafterdoublingthespindle
countwecouldnotaddadditionaluserloadbeyond1000userswithoutviolatingourSLAof2seconds
AQRT(AverageQueryResponseTime).Whileatthe1000userload,the11Gtestenvironmentexhibited
only0.486secondsofAQRT.Apparently,thelegacytestenvironmentcannotbescaledanyfurther
withoutaddingadditionalprocessingpoweronlypossiblebyaddingadditionalRACnodestothecluster.
AlthoughtheadditionoftheRACnodestothelegacyenvironmentmayfixtheSLAviolation,thecost
implicationswouldbeenormousintermsofadditionalsystems,SANcomponents,power,andRAC
licenses.
Ontheotherhand,figure6and7revealthatthe11Gtestenvironmentexhibitedonly0.486secondsof
averagequeryresponsetimeanditwasonlylogicalthatifweincreasetheuserloadtodeterminehow
manyadditionaluserscouldbesustainedonoursinglenode11Gtestserverpopulatedwithasinglequad
coreprocessorwithoutviolatingourSLAof2secondsqueryresponsetime.Sofar,our11Gserverhas
beenconfiguredwith12GBofRAMwith2banksofallthree2GBDDR3memorychannelspopulatedto
achievetheoptimalmemoryconfigurationforasinglesocket.Ata1000userload,theserverthatwas
configuredwith2.5GBofOracletargetmemorywasalmostrunningoutofRAMtomakeadditionaluser
connections.So,wedecidedtoincreasetheRAMto18GBbypopulatingallthreebanksofthreechannels
with2GBDDR3RDIMMS.Duringthisiteration,weincreasedtheuserloadonthe11Gtestenvironment
allthewayto1600userswhilemonitoringourestablishedSLAoflessthanorequalto2secondsofAQRT
toensurenoviolationsoftheSLA.Theresultsareinfigure8and9below.

Figure 8: 12GB and 18GB configuration TPS comparison on 11G environment (20 disks)

0
10
20
30
40
50
60
70
80
90
200 400 600 800 1000 1200 1400 1600
T
P
S
UserLoad
TPSR71012GBRAM
TPSR71018GBRAM
Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
12

Figure 9: 12 GB and 18GB configuration AQRT comparison on 11G environment (20 disks)
TheabovefiguresrevealthatafterincreasingtheRAMonour11Gtestenvironmentnotonlywerewe
abletosustainahigheruserload,butwealsoimprovedouraveragequeryresponseat1000usersby
(0.4860.383)/0.383=26.8%whiletheTPSremainedthesameuntilreachinga1000userload.Alsoat
1600userload,ouraveragequeryresponsetimereached1.866,whichisalmostequaltotheaverage
queryresponsetimeofourlegacyproductionenvironmentat1000users,20spindlesand16GBofRAM.
Wedecidednottoincreasefurtherloadonthe11Gtestenvironmentatthispointsincethenext
incrementof1800usersloadviolatedourSLAof2seconds.
Tosummarizeourresultssofar,thetestenvironmentrunningOracle11gRelease1withoneIntelXeon
X5570processorwasabletohandle600moreuserstoruntheworkload,ascomparedtothelegacytwo
nodeclusterrunningonPowerEdgeeighthgeneration2850serverswhilemaintainourSLAof2second
averagequeryresponsetime.Alsothe11Gtestenvironmentexhibited(76.4647.6)/47.6=60.6%increase
intheintheresultingTPS.Wecanlookattheperformancegainfromtwodifferentdimensionsaverage
queryresponsetimeimprovementatthesameuserloadortheTPSimprovementresultingfrombeing
abletoincreasetheuserloadwhilemaintaintheSLA.Fromtheperspectiveofaveragequeryresponse
timeimprovementat1000userload,weseea(2.110.383)/0.383=450%performancegain.Fromthe
perspectiveofTPSimprovement,the11Genvironmentexhibitsa60%performancegainwhilesustaining
additionaluserloadandmaintainingtheSLA.Figures10and11belowdisplaytheperformancegainin
bothTPSandaveragequeryresponsetime.

0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
200 400 600 800 1000 1200 1400 1600
A
Q
R
T
(
s
e
c
)
UserLoad
AvgResponseTime
R71012GBRAM
AvgResponseTime
R71018GBRAM
Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
13

Figure 10: AQRT comparison between legacy(8GB+8GB) and 11G(18GB) environment (20 disks)


Figure 11: TPS comparison between legacy(8GB+8GB) and 11G(18GB) environment (20 disks)

ItisworthnotingthatourlegacyRACenvironmentexhibitedonlyamarginaldecreaseinIOWAITand
increaseinUSERCPUtimeasaresultofdoublingthenumberofspindleshostingtheDATAdiskgroup.On
theotherhand,our11Gtestenvironmentexhibiteda(67.148.33)/48.33=38.8%decreaseintheIOWAIT
timeanda(21.4116.34)/16.34=31%increaseinUSERtimethatshowsthatthesystemisspendingmore
timeperformingusefulworkaswescalethebackendstoragesubsystemwithadditionaldisks.The
comparisonisshowninfigure12and13below.
0
0.5
1
1.5
2
200 400 600 800 1000 1200 1400 1600
A
Q
R
T
(
s
e
c
)
UserLoad
AvgResponseTime
legacy20disks
AvgResponseTime
R71020disks
0
20
40
60
80
100
200 400 600 800 1000 1200 1400 1600
T
P
S
UserLoad
TPSlegacy20disks
TPSR71020disks
Consolida
14

Fi

Fi

Si
af
30
us
fr
th
ating OLTP W
igure 12: CPU ti
igure 13: CPU ti
nceweencou
fterscalingthe
0forthe11Gt
serload,TPSa
rom18GBto36
hediagrams14
0
10
20
30
40
50
60
C
P
U
U
t
i
l
i
z
a
t
i
o
n
0
10
20
30
40
50
60
70
C
P
U
U
t
i
l
i
z
a
t
i
o
n
Workloads on
ime comparison
ime comparison
nteredadecre
edisksforour
testenvironme
ndAQRT.Inor
6GB,scaledthe
4and15below
average
iowait
average
iowait
Dell PowerEd
n for legacy env
n for 11G enviro
easeintheIOW
11Gtestenviro
enttofigureou
rdertosuppor
ebackenddisk
w:
average
usertime
average
usertime
av
sy
t
dge

11G Serv
vironment after
onment after sca
WAITtimeand
onment,wede
utifthereisad
rtadditionalus
ksto30andre
average
system
time
verage
ystem
time
vers
scaling disks
aling disks
aproportiona
ecidedtoincre
dditionalroom
serload,weals
ranthetestite
Legacy:
users
Legacy:
users
R710:10Di
R710:20Di
lincreaseinth
easethespindl
forgrowthin
soscaledourm
eration.There
:10Disk1000
:20Disks1000
sk1000users
sks1000users
:

heCPUUSERti
ecountfrom2
termsofaddit
memoryfurthe
esultsareshow
0
s
me
20to
tional
er
wnin
Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
15

Figure 14: TPS comparison for 11G environment after scaling disks from 20 to 30

Figure 15: AQRT comparison for 11G environment after scaling disks from 20 to 30

Thefigure16revealsthattheadditionofspindlesfurtherimprovedtheAQRT,andat1600users,wesaw
animprovementof(1.866.563)/.563=231%.Thenextlogicalstepwouldbetofurtherincreasetheuser
loadonthe11Gtestenvironment,anddetermineatwhatuserloadourSLAof2secondsresponsetimeis
violated.Figures16and17belowshowboththemaximumuserloadthatcanbesustainedonour11G
testenvironmentwithoutviolatingtheSLA,aswellastheperformancegainafterscalingtheRAMfrom18
GBto36GB:
0
10
20
30
40
50
60
70
80
90
200 400 600 800 1000 1200 1400 1600
T
P
S
UserLoad
TPSR71020disks
TPSR71030disks
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
200 400 600 800 1000 1200 1400 1600
A
Q
R
T
(
s
e
c
)
UserLoad
AvgResponseTime
R71020disks
AvgResponseTime
R71030disks
Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
16

Figure 16: 18 GB and 36 GB configuration TPS comparison on 11G environment (20 disks)


Figure 17: 18 GB and 36 GB configuration AQRT comparison on 11G environment (20 disks)

Theabovefiguresrevealthatwith30DATAdisksand36GBofRAM,wearenowabletoscaletheuser
loadtoaround2500userswithoutviolatingourSLAaswellashavinga(25001000)/1000=150%
improvementfromthebaselegacyenvironmentintermsofuserload.

0
20
40
60
80
100
120
140
200 600 1000 1400 1800 2200 2600 3000
T
P
S
UserLoad
TPSR71030disks
18GBRAM
TPSR71030disks
36GBRAM
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
200 600 1000 1400 1800 2200 2600 3000
A
Q
R
T
(
s
e
c
)
UserLoad
AvgResponseTime
R71030disks18GB
RAM
AvgResponseTime
R71030disks36GB
RAM
Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
17

TESTCONFIGURATION
Table1describesthecompletesoftwareandhardwareconfigurationthatwasusedthroughouttestingonboth
thesimulatedlegacyproductionenvironmentandthe11Gtestenvironment.
Table1:Oracle 11g Database Replay Test Configuration
Component Legac y Pr oduc t i on Envi r onment Del l Pow er Edge 11G Test Envi r onment
Systems TwoPowerEdge28502U servers One PowerEdgeR7102Userver
Processors TwoIntelXeonCPU3.40GHzsinglecoreper
node
Cache:L2=1MperCPU
Test1:One Intel XeonX55702.93GHz
quadcore
CPUCache:L2=4x256KL3=8M

Memory 8GBDDR2pernode (16GBtotalfor2nodes) Iteration1:12GBDDR3

Iteration2:18GBDDR3
Iteration3:36GBDDR3
Internaldisks Two73GB3.5SCSIR1 Two73GB2.5SASR1
Network TwoIntel82544EIGigabitEthernet Four Broadcom NetXtremeIIBCM5709
GigabitEthernet
Externalstorage Dell/EMCCX4960with:
10x146GBFibreChanneldisksfor
DATABASEdiskgroup
Iteration1:10x146GBFibreChanneldisks
forDATAdiskgroup
Iteration2:20X146GBFibreChanneldisks
forDATAdiskgroup
Dell/EMCCX4960with
10x146GBFibreChanneldisksfor
DATABASEdiskgroup
Iteration1:10x146GBFibreChanneldisks
forDATAdiskgroup
Iteration2:20X146GBFibreChanneldisks
forDATAdiskgroup
Iteration3:30X146GBFibreChanneldisks
forDATAdiskgroup

HBA TwoQLE2460pernode TwoQLE2460

OS EnterpriseLinux4.6 EnterpriseLinux5.2
Oraclesoftware Oracle10gR210.2.0.4
FileSystem:ASM
Diskgroups:DATABASE,DATA
sga_target=1600M
pga_target=800M

Oracle11gR111.1.0.6
FileSystem:ASM
Diskgroups:DATABASE,DATA
memory_target=2400M
Workload QuestBenchmarkFactoryTPCClike
workload
Scalefactor:3000
Userconnections:2001000
QuestBenchmarkFactoryTPCC
likeworkload
Scalefactor:3000
Userconnections:
12GBRAM:2001000
18GBRAM:2001600
36GBRAM:2003000
Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
18

RESULTS
NOTE:Theresultswehaveprovidedareintendedonlyforcomparisonofthetwoenvironmentsconsistingof
specificconfigurationsinalabenvironment.Theresultsdonotportraythemaximumcapabilitiesofanysystem,
databasesoftware,orstorage.
Thefollowingtestresultsaddressquestionsregardingthelimitingperformancefactorsofthelegacyenvironment
runningOracleRAC,thecapabilitiestoscalethe11Gtestenvironment,andtheresultingconsolidationfactorof
the11Gtestenvironment.Thegoalsofthetestweretodetermine:
Themaximumperformancecapabilitiesofourlegacyproductionenvironment,andtodeterminewhether
itcanbeefficientlyscaledbyaddingadditionalresources.
AbaselinecomparisonofAQRTbetweenthelegacyenvironmentandthe11Gtestenvironment,witha
baselineconfigurationusingthesamebackenddiskconfigurationandthenumberofCPUcores.The
baselinealsodefinesanSLAofamaximumof2secondsofAQRT.
Thecapabilitiesofthe11Gtestenvironmenttoscaleafteraddingadditionalresourcestodeterminethe
scalefactor
TheconsolidationfactorresultingfrommigratingtoPowerEdge11GserversfromPowerEdge8Gsingle
coreserversrunningOLTPworkloads
Table2providedbelowsummarizesthetestresultsasdiscussedinthetestmethodologysectionofthiswhite
paper.Fromtheresultstablewecanseethatourlegacyproductionenvironmentcouldnotscaleanyfurtherafter
addingadditionaldisks.ThisisevidentfromthefacttheAQRT,IOWAITtimedidnotimprovesignificantlyeven
afterdoublingthespindlecount.TheonlywaytofurtherimproveperformancewouldbetoaddadditionalRAC
nodes,aswellasaddingadditionaldiskstoscalethelegacyenvironmentwhichcouldbecostprohibitivegiventhe
addedlicensecostsassociatedwithOracleRAC.
Ontheotherhand,wecanseethatthe11Gtestenvironmentconsistingofonlyonenodeand12GBofRAM
performedslightlybetterthanthetwolegacyRACnodesbutalsoexhibitedhugescalabilitypotential.Aswecan
seefromthedataprovidedintable2,the11Gtestenvironmentwasrepeatedlyscaledwithadditionaldisksand
memoryandeachtimeitcouldsustaineitheradditionaluserload,orimprovedtheTPSandAQRTatthesameuser
loaddependingontheusagemodelthatthecustomeradopts.Finally,duringthetestiterationconfiguredwith30
DATAdisks,36GBofRAMand2600userload,the11Gtestenvironmentstartedtoexhibitsimilarbottlenecksthat
thelegacyRACenvironmentwasshowing.TheaverageCPUutilizationstartedtoreachabove97%withaverylow
IDLEtime.Atthispoint,insteadoffurtherscalingtheDATAdisksormemory,itwillbemorebeneficialtoaddan
additionalquadcoreCPUtoincreasetheprocessingpower.Onceadditionalprocessingpowerhasbeenadded,
youcancontinuetotakeadvantageofthescalablearchitectureprovidedbythe11Genvironmentbyadding
additionalresourcesasneeded.
ConsolidationFactor
Afteracursoryanalysisoftheresults,onecouldreachtheconclusionthatour11Gtestenvironmentconsistingof
asingleserverrunningOracle11gdatabasewasabletohandletheOLTPworkloadofatwoserverlegacyRAC
environmentwhilemaintainingourSLAof2secondsAQRT.But,onemustnotignorethefactthatour11GR710
serverwasonlypopulatedwith1quadcoreCPU.Fromthatperspective,wecanextrapolatethatonesingle11G
R710serverpopulatedwithtwoquadcoreCPUswouldbeabletoconsolidatetheworkloadofafournodelegacy
RACenvironment,providedthatbothenvironmentsareconfiguredwithadequatediskandmemoryresourcesso
thattheydonotbecomethebottleneck.

Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
19

Table2:Results Summary

Ourresultsalsorevealedthatalthoughthebasecomparisonbetweenbothenvironmentsshowedalmostidentical
performanceintermsofAQRT,thelegacyRACenvironmentcouldnotbescaledanyfurtherevenafterremoving
theI/Obottleneck.However,the11GtestenvironmentstillhadalotofroomforgrowthinCPUIdletime.This
behaviorisdepictedinfigure18below.
Syst em DATA
Di sk s
RAM User
Load
Aver age
CPU
Ut i l i zat i on
User Syst em I OWAI T I DLE AQRT
2850 Legac y RAC
node1+node2
10 16 GB 1000 99.25 56.8 9.97 32.48 0.75 2.11
20 16 GB 1000 99.24 57.37 11.27 30.62 0.76 1.83

R710 (X5570)
10 12 GB 1000 85.87 16.34 2.43 67.10 14.13 2.05
20 12 GB 1000 78.59 21.41 2.84 48.33 27.42 0.486
20 18 GB 1000 66.25 18.02 2.32 45.91 33.75 0.383
30 18 GB 1000 63.74 21.23 2.57 39.94 36.26 0.287
30 36 GB 1000 48.24 13.13 2.21 32.9 51.76 0.159

20 18 GB 1600 88.29 22.81 3.82 61.66 11.71 1.866
30 18 GB 1600 84.17 29.82 4.07 50.28 15.83 0.563
30 36 GB 1600 74.65 21.09 3.83 49.73 25.45 0.372

30 36 GB 2400 94.58 33.83 5.55 55.2 5.42 1.43
30 36 GB 2600 97.29 37.67 8.62 51 2.95 2.481
30 36 GB 3000 97.15 38.67 6.33 52.15 2.85 3.419
Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
20

Figure 18: Adding disk and memory resources: CPU idle time behavior comparison between legacy and 11G
environment

Forexample,inthe20disk1000usertestiteration,thelegacyRACenvironmentCPUsonlyexhibited0.76%idle
timewhilethe11GRACenvironmentexhibiteda27.42%CPUidletimewhileperforming(1.830.486)/0.486=276%
or3.76timesfasterintermsofAQRT.Fromthisperspective,our11Gtestenvironmentshouldbeabletoperform
theworkloadofalmost7serverlegacyRACenvironment.Thistranslatesintoaconsolidationfactorof7to1.
Ofcourse,therearemultiplewaysinwhichtoanalyzetheseresults.Anotherperspectivecouldbethescalability
factorofthe11GenvironmentascomparedtothelegacyRACenvironment.OurlegacyRACenvironment
exhibitedonly15.2%AQRTimprovementwhenadditionaldiskswereaddedwhilethe11GRACenvironment
showed(2.05.486)/.486=321%improvementinAQRT.Thisshowsthatwithidenticalstorageconfigurations,the
11Gtestenvironmentexhibitsmorethan300%betterscalabilitywhenaddingdiskresources.
ItisalsoimportanttonotethefactthatOracleStandardEditionRAClicensingisbasedonmaximumof4CPU
socketsperRACirrespectiveofthenumberofcorespersockets.Fromthatperspective,youmayreplacethe
legacyEnterpriseEditionRACenvironmentwithatwonode11GR710StandardEditionRACenvironmentwith
eachserverpopulatedwith2sockets,consistingof4corespersockettotaling16coresresultingintremendous
performancegains,energysavingsandahugefuturescalabilitypotential.Wecanalsolookattheresultsfromthe
perspectiveofconsolidatingmultiplelegacydistributedstandaloneOLTPworkloadsonasingleOracleRAC
environmentrunningDellR710servers.Fromthisangle,onemayconcludethatasmanyasfourteenstandalone
legacynodesrunningOLTPworkloadsmaybeconsolidatedonanOracle11GRACenvironmentconsistingoftwo
0
10
20
30
40
50
60
1
0

d
i
s
k

1
2
G
B

R
A
M

(
1
0
0
0

u
s
e
r
s
)
2
0

D
i
s
k

1
2
G
B

R
A
M

(
1
0
0
0

u
s
e
r
s
)
2
0

D
i
s
k

1
8
G
B

R
A
M

(
1
0
0
0

u
s
e
r
s
)
3
0

D
i
s
k

1
8
G
B

R
A
M

(
1
0
0
0

u
s
e
r
s
)
3
0

D
i
s
k

3
6
G
B

R
A
M

(
1
0
0
0

u
s
e
r
s
)
2
0

D
i
s
k

1
8
G
B

R
A
M

(
1
6
0
0

u
s
e
r
s
)
3
0

D
i
s
k

1
8
G
B

R
A
M

(
1
6
0
0

u
s
e
r
s
)
3
0

D
i
s
k

3
6
G
B

R
A
M

(
1
6
0
0

u
s
e
r
s
)
3
0

D
i
s
k

3
6
G
B

R
A
M

(
2
4
0
0

u
s
e
r
s
)
3
0

D
i
s
k

3
6
G
B

R
A
M

(
2
6
0
0

u
s
e
r
s
)
3
0

D
i
s
k

3
6
G
B

R
A
M

(
3
0
0
0

u
s
e
r
s
)
C
P
U
%
I
D
L
E
CPU%IDLEtimevsDiskandMemory
Resources
CPU%IDLE(11G)
CPU%IDLE(Legacy)
Consolidating OLTP Workloads on Dell PowerEdge

11G Servers
21

DellR710serversprovidedthatthebackendstorageandmemoryisscaledaccordingtotheaggregatedIOPSand
concurrentuserconnectionsrespectivelyasdiscussedinthisstudy.
SUMMARY
DatabasesystemsrunningOnlineTransactionProcessingworkloadsrequiretheoptimalbackendstoragedisk
layoutanddiskquantitiestoefficientlyservicealargeconcurrentuserpopulation.Thelegacyserversrunning
thesetypesofworkloadshavebeensufferingarchitecturallimitationsoffrontsidebusdesignsthatwerealimiting
factorwhenitcametoefficientlyutilizingtheCPUresources.Thus,onlyalimitednumberofdisksormemory
couldbeservicedbyaCPUcoreinasystembasedonFSBdesign.Inthiswhitepaperwedemonstratedthat
PowerEdge11GserversequippedwithXeon5500SerieschipsetsforI/OandprocessorinterfacingremovetheFSB
bottleneckandprovideanidealplatformtoconsolidatelegacydatabaseenvironments.TheR710chipsetis
designedtosupportIntelsCorei7processorfamily,QuickPathInterconnect,DDR3memorytechnology,andPCI
ExpressGeneration2.Thisstudyalsodemonstratedthat11Gserversofferlargeperformancegainswhen
comparedtooldergenerationserverswithfrontsidebusarchitectures.Thedatabasesystemsrunningon
PowerEdge11Gserversexhibitbetterscalabilitywhenadditionalresources,suchasdisksandmemory,areadded.
CustomersrunningOracle9ior10gRACenvironmentsonlegacyserversandstoragecanfollowtheguidelinesand
proceduresoutlinedinthiswhitepapertoconsolidatepowerhungryRACnodesintofewer,faster,moreenergy
efficientnodes.TheresultinglegacyRACnodeconsolidationcanalsodrivedownOraclelicensingcosts,resultingin
savingsthatyoucanusetoadditionalbackendstorageresourcestoimproveaveragequeryresponsetime,
implementdisasterrecoverysitesandadditionalRACtestbedsitesforapplicationdevelopmentandtesting.The
reducednumberofnodesdoesnotcompromiseperformancewhenpairedwithPowerEdge11Gservers.The
resultislessclusteroverhead,simplifiedmanagement,andpositivemovementtowardanobjectiveofsimplifying
ITandreducingcomplexityindatacenters.

REFERENCES
ConsolidatingDSSWorkloadsonDellPowerEdge11GServersUsingOracle11g
DatabaseReplay
http://www.dell.com/downloads/global/solutions/database_11g_consolidate.pdf?c=us&cs=555&l=en
&s=biz
OracleDatabasePerformanceTuningGuide10gRelease2(10.2)
PartNumberB1421103