12/14/2016

Home

SettingthequeuedepthorExecutionthrottleparametersClariion/VNX|50muaboutstorage

About

Featured
Search:type,hitenter

50muaboutstorage

Storingdataandbeawesome

SettingthequeuedepthorExecutionthrottleparametersClariion
/VNX
PostedbyRobKoperonDecember22,2011

Gotocomments

Leaveacomment(5)

EachFCstoragearrayporthasamaximumqueuedepthof2048.Forperformancereasonswellhavetodo
themathwith1600.SupposealargenumberofHBAs(initiators)aregeneratingIOs,aspecificportqueue
canfilluptothemaximum.ThehostsHBAwillnoticethisbygettingqueuefull(QFULL)messagesandvery
poorresponsetimes.ItdependsontheOperatingsystemhowthisisdealtwith.OlderOSscouldloose
accesstoitsdrivesorevenfreezeorgetabluescreen.ModernOSswillthrottleIOsdowntoaminimumto
getridofthisinconvenience.VMwareESXforexampledecreasesitsLUNqueuedepthdownto1.When
thenumberofqueuefullmessagesdisappear,ESXwillincreasethequeuedepthabituntilitsbackatthe
configuredvalue.Thiscouldtakeuptoaroundaminute.
DuringtheQFULLeventsthehostsmayexperiencesometimeouts,eveniftheoverallperformanceofthe
CLARiiONisOK.TheresponsetoaQFULLisHBAdependent,butittypicallyresultsinasuspensionof
activityformorethanonesecond.Thoughrare,thiscanhaveseriousconsequencesonthroughputifthis
happensrepeatedly.
SomeOperatingSystemsand(HBA)driverscansetaceilingonthequeuedepthsperLUN.Thisis
commonlyreferredtoasthetargetqueuedepth.VMwareESXlimitsthequeuedepthonaperLUNbasis
foreachpath.
AnEMCCLARiiON(aswellasmanyotherstoragearrays)willreturnaQFULLflowcontrolcommandunder
thefollowingconditions:
1ThetotalnumberofconcurrentI/OrequestsontheFrontEndFCportisgreaterthan1600.
2ThetotalrequestsforaLUNisgreaterthanitmaximumqueuedepth(32+(14*LUNsdatadrive
quantity)).Forexample,fora5driveRAID5(4+1),maximumqueuedepth=32+4*14=88
TheHBAexecutionthrottlethresholdsonthehostsmaybesetattoohighavalue(suchas256).
OnaWindowsmachineusingQLogicHBAsusetheSANsurferutilitytochangetheExecutionThrottlefor
eachHBA.Thiscanbedoneonline.InnewversionsofSANsurfer,theExecutionThrottleisfoundonthe
AdvancedHBASettingsselectaHBAport,thenParameters,thenontheSelectSettingssectiondrop
down.ThedefaultsettingforExecutionThrottleinanEMCenvironmentis256ifthisishigherthan256,
thenchangeto256ifthesettingis256tryloweringitto32.
ThesametargetqueuelengthrestrictionsapplytoallotherHBAmakesandmodels.WithEmulex,these
settingscouldbechangedusedHBAAnywhere.
InordertoavoidhammeringthestorageprocessorFEFCports,youcancalculatethemaximumqueue
http://www.50mu.net/2011/12/22/settingthequeuedepthorexecutionthrottleparametersclariionvnx/

1/8

12/14/2016

SettingthequeuedepthorExecutionthrottleparametersClariion/VNX|50muaboutstorage

InordertoavoidhammeringthestorageprocessorFEFCports,youcancalculatethemaximumqueue
depthusingacombinationofthenumberofinitiatorsperStoragePortandthenumberofLUNsESXuses.
OtherinitiatorsarelikelytobesharingthesameSPports,sothesewillalsoneedtohavetheirqueue
depthslimited.Themathtocalculatethemaximumqueuedepthis:
QD=1600/(Initiators*LUNs)
QD=therequiredQueueDepthorExecutionThrottle,whichisthemaximumnumberofsimultaneousI/O
foreachLUNanyparticularpathtotheSP.
Initiators=thenumberofinitiators(HBAs)perStoragePort,whichisnormallyequivalenttothenumberof
ESXhosts,plusallotherhostssharingthesameSPports.
LUNs=thequantityofLUNsforESXwhicharesharingthesamepaths,whichisequivalenttothenumber
LUNsintheESXstoragegroup.
TwoESXparametersshouldbesettothisQvalue.Thesearethequeuedepthofthestorageadapterand
Disk.SchedNumReqOutstanding.Mostofthetime,theDisk.SchedNumReqOutstandingissettoalower
valuethantheHBAqueuedepthinordertopreventanyparticularvirtualmachinefromcompletelyfillingup
theHBAqueueandtherefornotallowingotherHBAstoperformanyIOrequests.Ifthisiscurrentlythe
caseintheESXenvironment,thesesettingsshouldevenlybedecreased.Forexample,iftheHBAqueue
depthis64andDisk.SchedNumReqOutstandingis32(thedefaultsetting),thenreducingtoreducethe
QFULL,theHBAqueuedepthcouldbesetto32andDisk.SchedNumReqOutstandingsetto16.
Forexample,afarmof16ESXservershavefourpathstotheCLARiiON(viatwoHBAeach)andtheseFC
portsarededicatedforusebyESX(whichmakeskeepingqueuedepthsundercontroleasier).Thereare
multiplestoragegroupsinthisexampletokeepeachESXserversbootLUNprivate,buteachstoragegroup
has5LUNs.
Thisleadstothefollowingqueuedepth:
QD=1600/(16*5)=20
InpracticeacertainamountofoversubscriptionwouldbefinebecauseallLUNsonallserversareunlikely
tobebusyatthesametime,especiallyifloadbalancingisused.Sointheexampleabove,aqueuedepthof
32shouldstillnotcauseQFULLeventsundernormalcircumstances.

AlsoseethefollowingVMwareknowledgebasearticles:
http://kb.vmware.com/kb/1267
http://kb.vmware.com/kb/1268

http://www.50mu.net/2011/12/22/settingthequeuedepthorexecutionthrottleparametersclariionvnx/

2/8

12/14/2016

SettingthequeuedepthorExecutionthrottleparametersClariion/VNX|50muaboutstorage

Youmayalsolike:

2FREE
VSPHERE
EBOOKS
ON
AMAZON!
2FreevSphereebooks
CXorVNXMirrorview
onAmazon!OnJune5
and62013only!!

withSnapviewactiveon
theremoteside

Howtostartcopyto
hotsparemanually

Acceleratingyour
storagearraybyusing
SSDtechnology

FREE100
GB
CLOUD
STORAGE
Maximumdistances
Free100GBCloud
usingfibercabletypes
storage
OM1/OM2/OM3/OM4/OS1

VCE
ANNOUNCES
NEW
VBLOCK
MODELS
VCEannouncesnew

AnnouncingtheEMC
Elect2016!

vBlockmodels

AdsbyShareaholic

Sharethis:
Share 1

Tweet

Share

AdsbyShareaholic

Email

More

Related

VMware:"adaptivequeue

Optimizingperformanceusing

SCSI,(P)ATA,SAS,NLSAS

http://www.50mu.net/2011/12/22/settingthequeuedepthorexecutionthrottleparametersclariionvnx/

3/8

12/14/2016

VMware:"adaptivequeue
depth"setting
June27,2013
In"Configuration"

Configuration

SettingthequeuedepthorExecutionthrottleparametersClariion/VNX|50muaboutstorage

Optimizingperformanceusing
VAAIandtheESX
MaxHWTransferSizesetting
June30,2014
In"Configuration"

SCSI,(P)ATA,SAS,NLSAS
andSATA,what'sthe
difference?(part1)
March27,2013
In"commonknowledge"

executionthrottle,HBA,queuedepth

AcceleratingyourstoragearraybyusingSSD

Facilitatetheconversation:saywhatyoumeanand

technology

dontmakeassumptions

Leaveacomment?

5Comments.

FeliponJune27,2013at11:19

Reply

Thankyouverymuchforthepost.Ihavebeenreadingaboutthisforawhileandthisis
thesitewheretheseconceptsarebestdescribed.
Ontheotherhand,Istillhaveadoubtregardingyourexample:duringtheQueuedepth
calculation,Iseethatthenumberofpathstothestoragedoesnotneedtobetakeninto
account,butwhathappenswiththemultipathpolicy?Doesitmatterifwehavefixed,mru
orRoundRobin?
Kindregards

RRRJune27,2013at16:48

Reply

Thatswhereitgetsfuzzy.IdontthinkitreallymattersifyouhaveRRonallpaths,since
theloadwillbeequalonallports.Ifyouhavefixed,itsyourresponsibilitytomanually
loadbalancethedatastoresovertheexistingstorageports.
WithMRUIhaveatrick:setiftofixedbriefly,pinittoacertainportandsetitbackto
MRUagain.ThedownsideofthisisthatMRUdoesntguaranteethatadatastorewillbe
usingthatsamepathadaylater,soIMHOitsbesttoswitchtofixedorRRasfastas
possible.
Fortherest:theresanicePrimuscasewhichdescribestheverysametopicandifyou
reallywanttoknowalltheinsandoutstheperformanceworkshopisTHEplacetoask
questionslikethis
ItriedtolocatethisPrimuscase,butIcantfinditthisfast,butbeawarethatthereissuch
acaseaboutqueuedepthandexecutionthrottleanditalsomentionsthequeuedepth
of88.

RRRJune27,2013at16:59

Reply

OntheEMCwebsiteyoucanfindasomewhatnewerarticle(000053727)onthesubject
here

benraz7August1,2014at21:37

Reply

Hi,
Ihaveaquestion:
Letssaythatwehave1ESXserverwith2HBAcards,connectedto4Targetports(2paths
foreachHBA)inaVMAXcontrollerforexampleto1LUN.
TheQDforeachHBAissetto32andtheDisk.SchedNumReqOutstandingisset32aswell.
willIbeabletoget64Queuesintotal(32foreachHBA)oronly32queuesbecause
Disk.SchedNumReqOutstandingislimitingus?

IfsowouldntitbesmarttolimiteachHBAto16QDwhilethe
Disk.SchedNumReqOutstandingstaysat32?

http://www.50mu.net/2011/12/22/settingthequeuedepthorexecutionthrottleparametersclariionvnx/

4/8

12/14/2016

SettingthequeuedepthorExecutionthrottleparametersClariion/VNX|50muaboutstorage

Disk.SchedNumReqOutstandingstaysat32?
Thanksinadvance,
BenRaz.

benraz7August1,2014at21:40

Reply

AndIforgottomentionweareworkinginaroundrobinfashioninthisexample

Wouldyouliketocommentonthispost?
Enteryourcommenthere...

Search
Search:type,hitenter

SEARCH

Awards
EMCElect2013,2014,2015and

theMonthlyarchivesection
September2016(2)
June2016(3)
May2016(3)
April2016(5)
March2016(3)
February2016(1)
http://www.50mu.net/2011/12/22/settingthequeuedepthorexecutionthrottleparametersclariionvnx/

5/8

12/14/2016

SettingthequeuedepthorExecutionthrottleparametersClariion/VNX|50muaboutstorage

December2015(2)
November2015(2)
October2015(2)
May2015(4)
April2015(8)
March2015(5)
February2015(6)
November2014(3)
September2014(1)
August2014(1)
July2014(2)
June2014(3)
May2014(4)
April2014(11)
March2014(4)
February2014(2)
January2014(5)
December2013(2)
October2013(3)
September2013(3)
August2013(3)
July2013(2)
June2013(7)
May2013(7)
April2013(5)
March2013(6)
February2013(2)
January2013(1)
November2012(2)
October2012(1)
August2012(1)
July2012(2)
December2011(1)
July2011(1)

Links
AllenWard'sBlogAStorageEducationfor
theUninitiated
BasRaayman'stechnicaldiatribe
blog.bjornhouben.com
blog.scottlowe.org
DamianKarlson
DaveHenryGeekFluent
FastStorage
G'sviewoftheworld
GabesVirtualWorld
Henriwithani
InfraBlogbyRobHeygele
http://www.50mu.net/2011/12/22/settingthequeuedepthorexecutionthrottleparametersclariionvnx/

InfraWorld

6/8

12/14/2016

SettingthequeuedepthorExecutionthrottleparametersClariion/VNX|50muaboutstorage

InfraWorld
MichelStevelmans
QuantumBackuptron
Rollingwindowsandstickybytes
RoyMikes.eu
StorageMonkey
TheCloudSurvivalGuide
theGinaMinksDaily
theMarkBrowneDaily
TommyTrogden"vTexan"
VipinV.K.
VirtualGeek
WookieCast

Populartags

AsktheExpertATECiscoClariion
Clone

CloudCWDMDWDMEFDEMC

EducationEMCElectEMC
ProvenProfessional

World

EMC

executionthrottle FASTCache

flashharddiskharddrivehardeschijfhot
spareIEEEIECSIGigabyteGibibyteIOps

VegasMirrorViewMusic

Las

NCQ

performancequeuedepthSAN
SDDCSDSSecuritySnapshotSnapViewspeed
oflightSSDTCQ VbarValleyofFireVCE

VMAXVMware
Windows

VNXWiFi

TopPosts&Pages
Maximumdistancesusingfibercabletypes
OM1/OM2/OM3/OM4/OS1
HowtotranslateWindowsdiskidstostorage
array'sLUNs
HowtobringdownanEMCVNXe(asifyou
everwanttodoso)
EMCSANCopybestpractices
Howtostartcopytohotsparemanually
WhichHotSparewillbeusedforafailed
drive?(EMCClariion/VNX)
OptimizingperformanceusingVAAIandthe
http://www.50mu.net/2011/12/22/settingthequeuedepthorexecutionthrottleparametersclariionvnx/

ESXMaxHWTransferSizesetting

7/8

12/14/2016

SettingthequeuedepthorExecutionthrottleparametersClariion/VNX|50muaboutstorage

ESXMaxHWTransferSizesetting
CiscoSmartZoning:isitreallyworththe
effort?YES!
HowtosettheNTPserver,timeandtimezone
inaBrocadeswitch
LiveVMwaredatastoreexpansionwithrunning
VMs

RecentComments
HowtomatchWindowsDiskstoVMwareHard
Disks|TheTiredAdminonHowtotranslate
WindowsdiskidstostoragearraysLUNs
DellEMCWorld2016Austin,TX,USA|50mu
aboutstorageonEMCMidrangeMega
Launch:thenewVNX
EMCAnnouncesUnityPart2|
penguinpunk.netonEMCUnity:simplicity
redefined
EMCUnity:simplicityredefined|50mu
aboutstorageonWanttoplayaroundwith
VMwareVVOLs?Heresyourchance!The
SoftwareDefinedVNXishere!
EMCWorld2016Howtofollowifyouare
notinVegas.|VipinV.KonAlistof(social)
activitiesduringEMCWorldinLasVegas2016

Followmeonsocialmedia

Disclaimer
Mytweetsaremyown,myblogismyown.The
postsfoundherereflectmypersonalopinions.
Thepublishedcontentisnotapprovedoreven
readinadvancebyEMCoranyothervendorand
doesnotnecessarilyreflectopinionsofthis
vendor.Thisismyblog,notanEMCblog.

Copyright201650muaboutstorage|PoweredbyzBenchandWordPress

http://www.50mu.net/2011/12/22/settingthequeuedepthorexecutionthrottleparametersclariionvnx/

8/8