CopyrightRedHat,Inc.20062010Allrightsreserved.

TheRedHatClusterSuiteNFSCookbook
SettingupaLoadBalancedNFSClusterwithFailoverCapabilities
ByBobPeterson
rpeterso@redhat.com
Updated:15July2010

15July2010:ChangedthedocumenttoreflectthemaximumofsixteennodessupportedbyRedHat.
23March2007:Iaddedasectiononclusteringbasics,afewnotesabouthowRHEL5isdifferentand
additionalnotesabouttwonodeclustersandquorumdisks.
SpecialthankstoRiaanvanNiekerk(riaan@obsidian.co.za)forhissuggestionsandimprovements.

NFSfileservingletsyousharevaluabledatawithpotentiallyhundredsofclientcomputers
onyourcompany'snetwork,buttherearepitfalls.Suppose,forexample,thatyourcompanyhas
purchasedfiftydisklessworkstationsforyouremployees.Thefactthattheworkstationshavenohard
drivessavedthecompanybigmoneyonpurchasing,andthereliabilityisbetterbecausetherearefewer
movingpartstofail.Soundsgoodintheory,right?Buttherearetwobigproblems:First,howdoyou
distributetheNFSworkloadsothatalltheclientsaren'thammeringonthesameserver?Second
problem:whatifaserverfails?Howdoyoupreventitfromlockingupdozensofworkstationsand
paralyzingyourbusiness?
ThesolutiontothefirstproblemistodistributetheworkthroughoutseveralNFSservers.
Thisisknownasloadbalancing.RatherthanhavingoneNFSserverforfiftyclients,youcanhavefive
NFSservers,eachofwhichprovidedatatotenclients,thusdistributingtheworkload.
ThesolutiontothesecondproblemistoconfigureyourNFSserverswithRedHatCluster
SuiteNFSfailoverservice.Basically,allfiveservershaveastethoscopeoneachoftheotherfour
servers,listeningforaheartbeat.Ifaserverfails,oneoftheremainingserversautomaticallytakeson
itsdutiesseamlesslyandtheclientwon'tevenknowaboutthefailure.Thisisknownasfailoverandit
isaccomplishedbytheRedHatClusterSuite's(RHCS's)ResourceGroupManager,rgmanager.
ThisdocumentwillcoverthebasicsofsettingupandrunningtheRedHatClusterSuite
andNFSsothatyournetworkissomewhatloadbalancedandprovidesfailoverservice.Itassumesthe
readerhasabasicunderstandingofNFS.I'mgoingtouselotsofspecificexamplesthatwillbepretty
basic.Inotherwords,I'mgoingtousebabysteps.I'mnotgoingtoexpoundonallthedetailsand
optionsavailabletoyou,butIwillexplainwhatI'mdoingalongtheway.Tokeepthingssimple,I'm
goingtostickwithmyfiveserverexample,althoughitshouldeasilyscaletosixteenservers.
Thegoalhereistoshowyouhowtodoitbygivingexamples,ratherthanprovidingyou
withacrypticmanualandforcingyoutofigureitoutyourself.Ifyou'relookingforadetailedmanual
onsettingupRHCS,itmaybefoundhere:
http://www.redhat.com/docs/manuals/csgfs/browse/rhcsen/

CopyrightRedHat,Inc.20062010Allrightsreserved.

Step1SetyourGoalsandDesignyourCluster
Takesometimetocarefullydesignyourclusterbeforeyouinvesttimeormoney
implementingit.Designthenetworksothatitprovidesenoughredundancytoguaranteeno
interruptionofservice.Makesureyoubuyhardwarethatprovidesscalabilitytoallowyourclusterto
growasyourcompany'sneedsgrow.Workfromyourgoals.Yourgoalsshoulddictatehowcomplex
yourconfigurationgets.

Decidehowmanynodestouse
Thefirstquestionyoushouldaskishowbigtomakeyourcluster.Inotherwords,how
manycomputers(nodes)tohaveinyourcluster.Therearetwomainissuestoconsider:fault
toleranceandworkload.
Thefirstissue,faulttolerance,relatestohowmuchyoucantoleratedowntime.Atthe
heartoftheissueisaspecialclusterrequirementcalledquorum.Quorumisarulethatprotectsyour
data.Letmeexplainwithanexample.
SupposeyouhavetwonetworkswitchestiedtogetherwithasingleEthernetcableandeach
switchhashalfofyournodes.Ifsomeoneaccidentallyunplugsthatcable,youhaveabigproblem:
insteadofonebigcluster,yousuddenlyhavetwosmallerclusters.Ifthesenodeshaveaccesstothe
samedataonsharedstorage,eachofthesetworogueclusters,withoutknowledgeoftheother,would
quicklycorruptthefilesystem,unlesstheyhavesomeruletopreventit.Tomakesurethatnever
happens,weonlyallowwritingtothefilesystemifthereisamajorityofnodescommunicating.Any
rogueclustersthatarecutofffromthemajorityrefusetooperatebecausetheydon'thaveamajorityof
votes;theydon'thavequorum.Inotherwords,atleast(half+1)ofyournodesneedtobe
communicatingnormally.
Sinceamajorityofnodesisneededfornormaloperation,ifyouhavetoofewnodesinthe
cluster,you'remorelikelytohavedowntime.IfyourcompanyrarelygetsordersthroughtheInternet,
athreenodeclusterwillcontinueservingyourdataifonenodefails,butiftwooutofthreegodown,
yourclusterisdown,andthatmaybefineforyourbusiness.
Ontheotherhand,ifeveryminuteofdowntimecostsyourcompanyamilliondollars,you
maywantaclusterofsixteennodessothatyouronlydowntimewouldbeintherareeventthateight
ormorenodesfail.Rightnow,RedHatonlysupportsuptosixteennodesinacluster,althoughthe
softwarecanprobablyhandlemore.
Bytheway,wemadeaspecialexceptionfortwonodeclusterssotheycancontinue
runningifonenodefails,butthathastobespeciallyconfigured.Therearealsowaysyoucanspecifya
tiebreakerfortwonodeclusters.
Thesecondissuetoconsiderwhenchoosingthesizeofyourclusterisworkload.Workload
relatestohowmuchworkisrequiredofthecluster.Ifyouhavemorenodesinyourcluster,youcan
distributetheworkloadbetweentheadditionalserversandhandleaheavierworkload.Forexample,if
you'reusingyourclusterforwebserving,youprobablywantenoughnodestoensurethatyourweb
serversdon'tgetswampedbylotsofInternettrafficwhenyourcompanyisdiscovered;customers
don'thavemuchpatience.

CopyrightRedHat,Inc.20062010Allrightsreserved.

LoadBalancingIssues
Youshouldalsothinkaboutloadbalancingissues.Ifyou'replanningtoserveNFS,ask
yourselfwhyyouwantloadbalancedNFScapabilitiesinthefirstplace.Ifit'sonlytoservedatato
webservers,thatmightbemoreefficientlydonebyleavingNFSoutofthepicturealtogether.Since
thergmanagerservicecanmanagewebserversaswellasNFSservers,itmaybebettertoforgetNFS
anduseamanagedapache/httpdserviceonyourcluster.
Next,determineifyouneedtrueloadbalancingforyoursituation.Withtrueload
balancing,allrequestsaredistributedequallybetweentheservers,andallserversdothesameamount
ofworkatanygiventime.Requestsaredistributedtotheserversbyarequesttrafficcopknownasa
LinuxVirtualServer(LVS)router.ThisisknownasNATrouting.Theroutermuststandbetween
yourclientmachinesandtheserversandthisisknownasatier.Multipletiersmaybeusedreducethe
workloadontheLVSrouter,andmultipleLVSroutersmaybeintroducedforfailover,buteachtier
addsalayerofcomplexity.Formoreinformation,seethesectiononTrueLoadBalancingbelow.
Ifyoudon'tcareaboutoccasionalservercongestion,youcanuseasimpletechniqueIcall
roundrobinpseudoloadbalancing.Withpseudoloadbalancing,clientsaresplitupequallybetween
theservers,andanyclient'sindividualrequestsareallhandledbythesameserver.Individualservers
maystilloccasionallygetswampedwithrequestssincetherequeststhemselvesarenotloadbalanced.
ForsimpleNFSapplications,thismightbeacceptable.
Forthesakeofsimplicity,I'llusepseudoloadbalancinginthisdocument.

Scalability
Designyourclusterwithscalabilityinmind.SetupyourIPaddressesandVirtualIP
addressessothereisroomtoaddmore.
BecauseofthewayI'vechosenmyIPaddresses,thisexampleallowsyoutoaddnew
serverstotheclusterasdesired.I'venamedmyserverstrin10throughtrin14andassignedthemIP
addresses192.168.0.10through14.Youcaneasilyaddnewservers,startingwithtrin15andassign
themIPsstartingwith192.168.0.15andhavethemworkingintheclusterwithminimaleffort.
Forthisdocument,thegoalisonlytoprovidebasicNFSservicetoseveralclientcomputers
withpseudoloadbalancingandfailover.I'llassume:
FiftydisklessNFSclients.
FiveNFSservers(calledtrin10,11,12,13,and14)runningRedHatEnterpriseLinux4
(RHEL4)withstaticIPaddresses192.168.0.10through192.168.0.14.
BasicNFSfileservingthroughfivevirtualIPaddresses10.0.0.1through10.0.0.5.
Pseudoloadbalancing.EachofthefiveserverswillbeassignedoneofthevirtualIPs.DHCP
servicecanbesetupononeoftheserverstodynamicallyassignclientIPsanddistributethem
equallybetweenthefiveVirtualIPs.VirtualIP10.0.0.1willserviceclients1,6,11,16,21and
soon.Address10.0.0.2willserviceclients2,7,12,17,22,etc.
We'llmanageourVirtualIPswithroundrobinDNS,whichordinarilymayhave
somedisadvantages.Inthiscase,itbecomesmorepalatablebecausetheVirtualIPswillbe
managedandmadehighlyavailablebyClusterSuite,soevenifasinglenodegoesdown,its

CopyrightRedHat,Inc.20062010Allrightsreserved.

VirtualIPwillberelocatedtoanothernode,andintheory,theVIPsshouldbealwaysavailable.
Forfailover,theVirtualIPofthefailingservermaybemovedtoanyoftheothers.Wecanalso
setupfailoverfortheDHCPservice.

Step2DesignandSetuptheClusterHardware
Thefirststepistodesignandsetupyourclusterofservers.Forthatyou'regoingtoneed
somehardware.Formyexample,I'musingaconfigurationthatconsistsof:

FiveNFSservers,eachofwhichhasanIDEbootdrive(hda),atleastoneNetworkInterface
Card(NIC)providingdeviceeth0andaFibreChannelHostBusAdapter:QLogicQLA2310.
Afibrechannelswitch.
HarddiskstorageconsistingofthreeLUNs(whichlooklikeSCSIdrivessda,
sdb,sdc)connectedtothefibreswitch.
Networkswitchorhub.Inmyexample,anygoodhighspeedEthernetswitchwilldo.
ANetworkPowerSwitch.Inmyexample,I'lluseanAPCMasterSwitchPlus.

NetworkConnections
EachserverisattachedtoanetworkswitchwithanEthernetcable.TheEthernet
connectionswillbeassignedstaticIPswhichareshowninblueonthediagram.Later,wewillassign
virtualIPaddressestothemaswell,showninred.
Theclientconnectionsareshowninredtoindicatethey'regettingtheirNFSdatafromthe
VirtualIP,eventhoughtheydohaveaphysicalconnection.Thediagramshowsthemconnected
directlytothenetworkswitch,butinreallifethenetworkwouldprobablybemorecomplex.
PowerConnections
AllserversarepluggedintoaNetworkPowerSwitch(NPS)thatalsohasanEthernet
connectiontothenetworkswitch.Powercablesareshowningreen.TheNPSservesanimportant
role:ifanodefails,theothernodesmustbeabletopowercycle(turnoffandbackon)thefailedoneto
preventitfrompotentiallydamagingthesharedstorage.
StorageConnections
EachserverhasaccesstothesamephysicalstoragethroughtheFibreChannelswitch.The
harddrivesandserversplugintotheswitchwithfiberopticcables(shownindarkblue).
Hereisadiagramofthehardware:

CopyrightRedHat,Inc.20062010Allrightsreserved.

Step3SetupRedHatClusterSuiteandRedHatGlobalFileSystem
Inordertosetuprgmanagerfailoverservice,youmustfirstsetupyourserversina
clusteredenvironment.Forthat,weinstallRedHatClusterSuite(RHCS)4.0.andRedHatGlobalFile
System(GFS)6.1.Herearethebasics:
Step3a.ApplythenecessaryRPMsonallfiveservers
YourNFSserversshouldhavethelatestRedHatClusterSuite,DeviceMapper,LVM2and
LinuxkernelRPMsinstalled.TheseRPMsshouldbeaddedtoallNFSserverstobeclustered(called
nodes).Hereisanexample:
[root@trin10~]#cd/usr/src/redhat/RPMS/i686/

CopyrightRedHat,Inc.20062010Allrightsreserved.
[root@trin10i686]#rpmikernel2.6.942.EL.i686.rpm
Repeatthisstepforallfiveservers.Afterthenewkernelisinstalled,rebootyourserversto
pickupthenewkernel.Next,weapplytheRPMsfordevicemapper,lvm2,andtheRedHatCluster
Suite.
[root@trin10i686]#rpmidevicemapper1.02.074.0.RHEL4.i686.rpm
[root@trin10i686]#rpmilvm22.02.066.0.RHEL4
[root@trin10i686]#rpmiccs1.0.70.i686.rpmcman1.0.110.i686.rpmcmankernelsmp2.6.9
45.2.i686.rpmdlm1.0.11.i686.rpmdlmkernelsmp2.6.942.10.i686.rpmfence1.32.251.i686.rpm
GFS6.1.61.i686.rpmGFSkernelsmp2.6.958.0.i686.rpmgulm1.0.70.i686.rpmiddev2.0.0
3.i686.rpmlvm2cluster2.02.067.0.RHEL4.i686.rpmmagma1.0.60.i686.rpmmagmaplugins1.0.9
0.i686.rpmperlNetTelnet3.033.noarch.rpmrgmanager1.9.530.i686.rpmsystemconfigcluster
1.0.271.0.noarch.rpm
Repeatthisstepforallfiveservers.Thisisjustanexample;theversionsfortheseRPMs
areamovingtarget,sotakethelatestforyourrelease.
NOTE:ForRHEL5andequivalent,youwillneedadifferentsetofRPMs.Forexample,thereisno
gulm,ormagma,ccs,fenceandcmanareallcombinedintoonecman,anddlmisbuiltintothe
kernel.ForRHEL5,you'llneedopenais.
Step3b.Createyour/etc/cluster/cluster.confconfigurationfile.
Ifyou'rerunningRHEL5orRHEL4.5,you'reinluckbecausethere'sanewwebbased
clustersetupandmanagementguicalledconga.Itwillmakeyourlifeawholeloteasierthanbefore.
Ifyou'renotusingconga,thereisagraphicaltoolforfirsttimesetupofaclustercalledcs
deploytoolthatautomatesmostofthesetup.There'sanothergraphicaltoolcalledsystemconfig
clusterthatallowsyoutoconfigureyourclusterevenafterit'sbeenconfigured.Togettoitfromthe
GUI,followthispath:
Applications>SystemSettings>ServerSettings>ClusterManagement.
Ifyouwanttoconfigureyourclustermanually,edit/etc/cluster/cluster.confwithyour
editorofchoice.HereisthefileI'llbeusingformyexample:
<?xmlversion="1.0"?>
<clusterconfig_version="1"name="bob_cluster">
<cman/>
<clusternodes>
<clusternodename="trin10">
<fence>
<methodname="single">
<devicename="trinapc"nodename="trin10"/>
</method>

CopyrightRedHat,Inc.20062010Allrightsreserved.
</fence>
</clusternode>
<clusternodename="trin11">
<fence>
<methodname="single">
<devicename="trinapc"nodename="trin11"/>
</method>
</fence>
</clusternode>
<clusternodename="trin12">
<fence>
<methodname="single">
<devicename="trinapc"nodename="trin12"/>
</method>
</fence>
</clusternode>
<clusternodename="trin13">
<fence>
<methodname="single">
<devicename="trinapc"nodename="trin13"/>
</method>
</fence>
</clusternode>
<clusternodename="trin14">
<fence>
<methodname="single">
<devicename="trinapc"nodename="trin14"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedeviceagent="fence_apc"ipaddr="192.168.0.9"login="apc"name="trinapc"passwd="apc"/>
</fencedevices>
<fence_daemonpost_fail_delay="0"post_join_delay="3"/>
</cluster>

Ifyou'reonlyplanningtohavetwonodes,you'llneedtodosomemore.Twonodeclusters
areaspecialcasethatrequiresomeforethought.Abiggerclusterwillonlyrunaslongasitmaintains
quorum,whichmeansthattherehastobeamajorityofnodesworkinginthecluster.Soforaseven
nodecluster,therehastobeatleastfournodesormorerunning.Withatwonodecluster,thereisno
majority,soyouhavetodecidebetweentwopossiblesolutions:Thefirstsolutionistobendthe
quorumrulesbyaddingexpected_votes="1"two_node="1"tothe<cman/>statement.Thesecond
solutionistouseaquorumdiskasatiebreaker.
Thefirstoption,bendingtherules,hasanegativesideeffect.Ifthetwonodeslosecontact
witheachother,forwhateverreason,theywillbothrealizeitatthesametimeandeachwilltrytofence
theother.Thiscanresultinashootoutinwhichthefastestnodekillstheothertoprotectthedata,and
dependingonthekindoffencingyouuse,theymaybothendupdying.
Thesecondoptionistouseatiebreaker.Ifanodestartsexperiencingcommunication
failures,there'snowaytoknowwhethertheproblemiswithitselforwiththeothernode,unlessithas

CopyrightRedHat,Inc.20062010Allrightsreserved.
someothersharedresourcelikeaaspecialpartitiononsharedstorageorasecondcommunication
channel.Thisistypicallycalledaquorumdisk,butthat'samisnomer;youcansetupawidevarietyof
heuristicstodeterminewhichnodewillbreakthetie.Unfortunately,settingupaquorumdiskisnot
trivial;yourbestresourceistoreadtheqdiskmanpage.
Step3c.Sendthecluster.conftoallnodesinthecluster.
Usesometoollikescptomovethecluster.conffiletoalloftheservers.Oncetheyhave
theirinitialcluster.conffile,changestothefilearenormallypropagatedbyusingtheccs_toolupdate
command,butthatwon'tworkuntilwehavetheclusterupandrunning.Sofornow,we'lldoitby
hand:
[root@trin10~]#scp/etc/cluster/cluster.confroot@trin11:/etc/cluster/
[root@trin10~]#scp/etc/cluster/cluster.confroot@trin12:/etc/cluster/
[root@trin10~]#scp/etc/cluster/cluster.confroot@trin13:/etc/cluster/
[root@trin10~]#scp/etc/cluster/cluster.confroot@trin14:/etc/cluster/

Step4.Bringupthecluster
Thenextstepistogetyourserversthinkingasateam.Inotherwords,youneedtobring
upthecluster.Thefirsttime,we'lldothismanually,butwe'llalsoconfigurethesystemstodothis
automaticallyonstartup.
YoucanfindsomecooltoolsoutontheInternetthatallowyoutotypethesamecommand
simultaneouslyonallofyournodes.Iuseonecalledcssh(clusterssh).It'sprobablynotthebestand
it'snotpartoftheregularinstallation,sofornowI'llassumeyou'lljustuseregularsshandaforloop.
Step4a.Tellallserverstostarttheccsdservice.
Thisistheservicethatsharescluster.confinformationinthecluster.Notonlydowestart
theservice,wealsosetituptostartautomaticallyeverytimethesystemsarerebooted.
[root@trin10~]#serviceccsdstart
[root@trin10~]#foriin`seq1114`;dosshtrin$iserviceccsdstart;done
Step4b.Tellallserverstojoininthecluster.
Nowweusetheclustermanagertooltogettheserverstalkingtoeachother.Asbefore,we
starttheserviceandconfigureittostartautomaticallyatstartup.
[root@trin10~]#servicecmanstart
[root@trin10~]#foriin`seq1114`;dosshtrin$iservicecmanstart;done
Nowyoursystemsarenotjustservers,theyarenodesinacluster.

CopyrightRedHat,Inc.20062010Allrightsreserved.
FYI:Thecmanserviceessentiallyexecutesthiscommand:cman_tooljoinw
Step4c.Tellallnodestostartthefencedservice.
Thisistheservicethatdetectsnodefailuresandpowercyclesserversthathavefailed(an
actknownasfencing).Thisisnecessarytopreventdamagetothefilesystem.
[root@trin10~]#servicefencedstart
[root@trin10~]#foriin`seq1114`;dosshtrin$iservicefencedstart;done
Thefencedserviceessentiallyexecutesthiscommand:fence_tooljoinw
NOTE:forRHEL5andequivalent,thesethreeservicesareallcombinedintoonecalledcman.So
forRHEL5,youdon'tneedsteps4aor4c.
Step4d.Tellallnodestostarttheclvmdservice.
ThisistheclusterLVM(LogicalVolumeManager)daemon.ItsjobistodistributeLVM
metadatainformationbetweenclusternodes.
[root@trin10~]#serviceclvmdstart
[root@trin10~]#foriin`seq1114`;dosshtrin$iserviceclvmdstart;done
Step4e.StarttheGFSservice.
ThisenablesclusterlockingandtheGFSfilesystemwithintheLinuxkernel.The
necessarykernelmoduleswillbeloadedautomaticallyatstartupbythegfsserviceifyouhaveaGFS
filesystemtobemountedinyour/etc/inittab.
[root@trin10~]#servicegfsstart
[root@trin10~]#foriin`seq1114`;dosshtrin$iservicegfsstart;done

Step5.Makesureyourserverscanseeyoursharedstorage
Fromoneofthenodes,youneedtoprepareyourvolumegroup,physicalvolume,logical
volume,thenformatyourGFSfilesystem.
Step5a.Makesureallnodeshavethehostbusadapterdevicedriversloaded.
Thefirststepistomakesureyourfibrechannelhostbusadapterdevicedriverisloaded.
Formyexample,I'musingaQLogicQLA2310hostbusadapter,soI'llneedtheqla2300andqla23xx
devicedrivers.

CopyrightRedHat,Inc.20062010Allrightsreserved.
[root@trin10~]#lsmod|grepqla
qla23001244810
qla2xxx1792978qla2300
scsi_transport_fc78731qla2xxx
scsi_mod1212933qla2xxx,scsi_transport_fc,sd_mod
Ifyoudon'tseethedevicedrivers,youmayneedtousedepmodatoresolvemodule
dependenciesforyourkernelandmodprobetoloadthem.
Step5b.Makesureallnodescanproperlyseethephysicaldrives.
Nowthatourdevicedriversareloaded,weneedtofindoutiftheycanactuallyseethehard
drivesthatareattached.ThebestwayisbyaskingtheLinuxkernel.IfthekernelcanseetheSCSI
drives,theyshouldshowupassdXpartitionsin/proc/partitions:
[root@trin10~]#cat/proc/partitions
majorminor#blocksname
3039082680hda
31104391hda1
3238973690hda2
808388608sda
8168388608sdb
8328388608sdc
8488388608sdd
IthrewinafourthSCSIdevice,sdd,justtomakeapoint:makesureyouknowwhichhard
drivesarewhich.Youdon'twanttoaccidentallydestroysomeoneelse'sdatabyrepartitioningahard
driveyouassumedwasyours.Youcanusethepvsandpvdisplaycommandstoseewhatphysical
volumesexistandwhethertheyareassignedtovolumegroups.
IfyourSANisn'tinthelist,youmaywanttousethedmesgcommandtoseeifthereare
anycomplaintsfromtheLinuxkernel.Iftherearen'tanycomplaints,youmayneedtoconfigureyour
SAN,whichissometimesdonethroughawebinterface.

Step6.CreateyourGFSfilesystem
Step6a.Updatethelvmconfigurationfilesontheservers.
Ifyouswitchedfromnormallvmtotheclusteredversion,youmayneedtochangeyour
lvmconfigurationabit.
[root@trin10gfs]#vi/etc/lvm/lvm.conf
Locatetheglobalsectionandchangelocking_typeandlocking_libraryasfollows:

CopyrightRedHat,Inc.20062010Allrightsreserved.
locking_type=2
locking_library="liblvm2clusterlock.so"
NOTE:ForRHEL5,youshoulduselocking_type=3.
Thechangesneedtobepropagatedtoalltheservers.
[root@trin10gfs]#vgchangealy
[root@trin10gfs]#vgscan
[root@trin10gfs]#foriin`seq1115`;doscp/etc/lvm/lvm.confroot@trin$i:/etc/lvm/;done
Step6b.Preparethephysicalvolumesforuse.
Thistakestheplaceofusingfdisktopartitionthedrives.
[root@trin10~]#pvcreate/dev/sda
[root@trin10~]#pvcreate/dev/sdb
[root@trin10~]#pvcreate/dev/sdc
Step6c.CreateavolumegroupforyourGFSfilesystem.
Thisgroupsallthreephysicalvolumesintoavolumegroup.
[root@trin10~]#vgcreatebobs_vg/dev/sdadev/sdbdev/sdc
Step6d.Createalogicalvolumeoutofyourvolumegroup.
[root@trin10~]#lvcreateL750Gbobs_vg
Step6e.MakeyourGFSfilesystemwithinthelogicalvolume.
[root@trin10~]#gfs_mkfstbob_cluster:bobs_gfsplock_dlmj8/dev/bobs_vg/lvol0
HereIspecifiedeight(8)journalsforthefilesystem.Theremustbeatleastonejournalper
serverinthecluster.Byspecifyingeight,I'mleavingmyselfroomtoaddmoreserversinthefuture.
Step6f.Edit/etc/fstabsothatitmountstheGFSfilesystemtobeexported.
Usingyoureditorofchoice,editthe/etc/fstabfiletoautomaticallymountthenewGFS
filesystematsystemstartup.Inmyfstab,I'vegotanentrythatlookslikethis:
/dev/bob_vg/lvol0/mnt/bobgfsdefaults00

CopyrightRedHat,Inc.20062010Allrightsreserved.
Step6g.Mountthenewfilesystemandpropagatethefstabtotheothernodes.
[root@trin10~]#mounta
[root@trin10~]#foriin`seq1115`;doscp/etc/fstabroot@trin$i:/etc/;done
[root@trin10~]#foriin`seq1115`;dosshtrin$imounta;done

Step7ChooseandConfigureNFSFailoverService
TherearetwobasictypesofNFSfailover:managedNFSserviceandManagedVirtualIPs
withRHCS'sGlobalFileSystem,GFS.Youneedtodecidewhichyou'regoingtouseandthereare
advantagesanddisadvantagestoboth.Ineitherscenario,NFSclientscontinuetoreadandwritefiles
ontheNFSmountedpathwithoutevendetectingwhenafailureoccurs.
WithmanagedNFSservice,rgmanagermanagestheentireNFSservice,automatically
monitoringtheserviceandrelocatingittoothernodeswhennodefailuresaredetected.
WithManagedVirtualIPs,NFSserviceisrunningonallofthenodesandrgmanager
monitorsavirtualIPaddress.Whennodefailuresaredetected,theVirtualIPaddressismovedfrom
nodetonode.ThismethodiscurrentlydesignedtoworkonlyinconjunctionwithGFS.
Whichshouldyouchoose?IfyouneedtoexportfilesystemsotherthanGFS,forexample,
EXT3,onyoursharedstorage,thenyoushoulduseManagedNFSService.Ifyouneedtomaintain
yourown/etc/exportsfileandexportamixtureoffilesystemswheresomeareonsharedstorageand
someareonlocalstorage,useManagedVirtualIPs.
Ifyou'reonlygoingtobeexportingGFSonsharedstorageyoucanuseeithermethod.
However,ManagedNFSServicehastwoadvantages:First,it'seasiertousesincergmanagertakes
careoftheexportsforyou,soyoudon'tneedtomanagethe/etc/exportsfile.Second,ifforsome
reasontheNFSservicestopsononeofyourserverswhichadmittedlyisn'tlikelythergmanager
willdetectandhandletheproblem.WithManagedVirtualIPs,iftheNFSservicestops,youmay
neverknowitunlessyoucheckitperiodically.

ManagedNFSService
ThismethodofmanagingNFSfailoverismorepowerfulthanusingManagedVirtualIPs
becauseitmaybeusedtomanageanyfilesystemNFScanexportprovidedthestorageisshared
amongthenodesintheclusterwhereasVirtualIPsshouldonlybeusedwithGFS.Itisrelatively
easytosetupbecauseyoucanusethesystemconfigclusterGUItooltomanageryourresources.Itis
alsoeasiertomaintainbecauseitautomatesthemonitoring,startingandstoppingoftheNFSserviceon
theservers.However,itisnotasflexibleasusingManagedVirtualIPsbecausergmanagermanages
allofyourNFSsharesthroughtheclusterconfigurationfile(cluster.conf)ratherthanlettingyoupick
andchoosewiththe/etc/exportsfile.
TherearetwowaystosetupManagedNFSServicewithrgmanager:manuallyorwiththe
systemconfigclusterGUItool.

SettingupManagedNFSServiceusingsystemconfigcluster:

CopyrightRedHat,Inc.20062010Allrightsreserved.
Step7a.Createourfailoverdomainsfortheservers
1. Startthesystemconfigclustertool.FromtheStartmenu:
Applications>SystemSettings>ServerSettings>ClusterManagement
2. UnderManagedResourceshighlightFailoverDomains.
3. ClicktheCreateaFailoverDomainbutton.
4. Next,youwanttobuildalistofnodesthatmayruntheservice.Todothat,usetheAvailable
ClusterNodespulldowntoaddnodestothemembernodebox.
5. Ifyouwanttheservicetobeprioritized,checkthePrioritizedListcheckbox.Youmay
thenhighlightthenodeslistedinthemembernodesboxandclicktheAdjustPriority
buttonstoselecttheorderinwhichthefailovershouldoccur.Thenodewiththehighest
priority(priority1)willbetheinitialnodeforrunningtheservice.Ifthatnodefails,theservice
willberelocatedtothenodewiththenexthigherpriority(priority2)andsoforth.Ifmultiple
nodeshavethesamepriority,rgmanagerwillchooseamongthemasitseesfit.HittheClose
buttonwhenyouaredone.
Createfivefailoverdomains,oneforeachserver.Foreachdomain,setthepriorityto1
forthehomeserver,and2fortheotherservers.Thatway,eachserverwillberesponsiblefora
groupofclients,andifthatserverfails,theservicewillberelocatedtoanotherserver.
Step7b.Createourresources
Next,foreachserver,wecreateourresources,theNFSexports.
6. UnderManagedResourcesHighlightResources.
7. ClicktheCreateaResourcebutton.
8. SelectresourcetypeGFSandfillinName,MountPoint,DeviceandOptions
9. ClicktheCreateaResourcebutton.
10. SelectNFSExportandfillinthename.
11. ClicktheCreateaResourcebutton.
12. SelectNFSClientandfillinthename,target,readwriteorreadonlyandoptions.
Step7c.SetuptheNFSservices
NowwehavetosetuptheNFSservicestoberunoneachoftheservers.
13. UnderManagedResourceshighlightServices.
14. ClicktheCreateaServicebuttonandgivetheserviceaname.Forexample,nfssvc.
AServiceManagementpopupwindowwillappear.
15. UsetheFailoverDomainpulldowntoselectthedomainyouspecifiedfromstep4.
16. UnderRecoveryPolicyselectRestartorRelocate.
17. ClickthebuttonlabeledCreateanewresourceforthisservice.

CopyrightRedHat,Inc.20062010Allrightsreserved.
18. UsepulldowntoselectResourceTypeGFS.
19. EnterName(bobfs1),MountPoint(/mnt/bob),Device(/dev/bob_vg/lvol0),andOptions
(acl).
20. Putinallofyourexportsthisway.
Step7d.Saveyourchangesandpropagatetothecluster
21. Whenyouarefinished,saveyourchangeswithFile>Saveorbypressing<Ctrl+S>
22. ClicktheSendToClusterbuttontosendyournewconfigurationtothenodesinthecluster.
23. YoumaythenclickontheClusterManagementtabandstarttheservice.
Whenyou'redone,your/etc/cluster/cluster.conffileshouldbesimilartotheoneshownin
thenextsection.

SettingupManagedNFSServicemanually:

Step7a.Addresourcemanagerdomains,resourcesandservicestocluster.conf
Editthe/etc/cluster/cluster.conffileusingyoureditorofchoiceandadda<rm>sectionthat
defineswhatrgmanagershouldmanageandhow.Forthisexample,weaddthefollowinglinestothe
cluster.confnearthebottom,beforethe<fence_daemon>statement:
<rm>
<failoverdomains>
<failoverdomainname="nfsdomain1"ordered="1"restricted="1">
<failoverdomainnodename="trin10"priority="1"/>
<failoverdomainnodename="trin11"priority="2"/>
<failoverdomainnodename="trin12"priority="2"/>
<failoverdomainnodename="trin13"priority="2"/>
<failoverdomainnodename="trin14"priority="2"/>
</failoverdomain>
<failoverdomainname="nfsdomain2"ordered="1"restricted="1">
<failoverdomainnodename="trin10"priority="2"/>
<failoverdomainnodename="trin11"priority="1"/>
<failoverdomainnodename="trin12"priority="2"/>
<failoverdomainnodename="trin13"priority="2"/>
<failoverdomainnodename="trin14"priority="2"/>
</failoverdomain>
<failoverdomainname="nfsdomain3"ordered="1"restricted="1">
<failoverdomainnodename="trin10"priority="2"/>
<failoverdomainnodename="trin11"priority="2"/>
<failoverdomainnodename="trin12"priority="1"/>
<failoverdomainnodename="trin13"priority="2"/>
<failoverdomainnodename="trin14"priority="2"/>
</failoverdomain>
<failoverdomainname="nfsdomain4"ordered="1"restricted="1">
<failoverdomainnodename="trin10"priority="2"/>
<failoverdomainnodename="trin11"priority="2"/>

CopyrightRedHat,Inc.20062010Allrightsreserved.
<failoverdomainnodename="trin12"priority="2"/>
<failoverdomainnodename="trin13"priority="1"/>
<failoverdomainnodename="trin14"priority="2"/>
</failoverdomain>
<failoverdomainname="nfsdomain5"ordered="1"restricted="1">
<failoverdomainnodename="trin10"priority="2"/>
<failoverdomainnodename="trin11"priority="2"/>
<failoverdomainnodename="trin12"priority="2"/>
<failoverdomainnodename="trin13"priority="2"/>
<failoverdomainnodename="trin14"priority="1"/>
</failoverdomain>
</failoverdomains>
<resources>
<clusterfsdevice="/dev/bob_vg/lvol0"force_unmount="0"fstype="gfs"
mountpoint="/mnt/bob"name="bobfs1options="acl"/>
<nfsexportname="NFSexports1"/>
<nfsclientname="client1"options="rw"target="client1.mycompany.com"/>
<clusterfsdevice="/dev/bob_vg/lvol0"force_unmount="0"fstype="gfs"
mountpoint="/mnt/bob"name="bobfs2"options="acl"/>
<nfsexportname="NFSexports2"/>
<nfsclientname="client2"options="rw"target="client2.mycompany.com"/>
<clusterfsdevice="/dev/bob_vg/lvol0"force_unmount="0"fstype="gfs"
mountpoint="/mnt/bob"name="bobfs3"options="acl"/>
<nfsexportname="NFSexports3"/>
<nfsclientname="client3"options="rw"target="client3.mycompany.com"/>
<clusterfsdevice="/dev/bob_vg/lvol0"force_unmount="0"fstype="gfs"
mountpoint="/mnt/bob"name="bobfs4"options="acl"/>
<nfsexportname="NFSexports4"/>
<nfsclientname="client4"options="rw"target="client4.mycompany.com"/>
<clusterfsdevice="/dev/bob_vg/lvol0"force_unmount="0"fstype="gfs"
mountpoint="/mnt/bob"name="bobfs5"options="acl"/>
<nfsexportname="NFSexports5"/>
<nfsclientname="client5"options="rw"target="client5.mycompany.com"/>
</resources>
<serviceautostart="1"domain="nfsdomain1"name="nfssvc1">
<ipaddress="10.0.0.1"monitor_link="1"/>
<clusterfsref="bobfs1">
<nfsexportname="bobfs1">
<nfsclientref="client1"/>
</nfsexport>
</clusterfs>
</service>
<serviceautostart="1"domain="nfsdomain2"name="nfssvc2">
<ipaddress="10.0.0.2"monitor_link="1"/>
<clusterfsref="bobfs2">
<nfsexportname="bobfs2">
<nfsclientref="client2"/>
</nfsexport>
</clusterfs>
</service>
<serviceautostart="1"domain="nfsdomain3"name="nfssvc3">
<ipaddress="10.0.0.3"monitor_link="1"/>
<clusterfsref="bobfs3">

CopyrightRedHat,Inc.20062010Allrightsreserved.
<nfsexportname="bobfs3">
<nfsclientref="client3"/>
</nfsexport>
</clusterfs>
</service>
<serviceautostart="1"domain="nfsdomain4"name="nfssvc4">
<ipaddress="10.0.0.4"monitor_link="1"/>
<clusterfsref="bobfs4">
<nfsexportname="bobfs4">
<nfsclientref="client4"/>
</nfsexport>
</clusterfs>
</service>
<serviceautostart="1"domain="nfsdomain5"name="nfssvc5">
<ipaddress="10.0.0.5"monitor_link="1"/>
<clusterfsref="bobfs5">
<nfsexportname="bobfs5">
<nfsclientref="client5"/>
</nfsexport>
</clusterfs>
</service>
</rm>

ThisexampleessentiallydefinesthemanagedNFSservice.
Noticethattheprioritiesforeachfailoverdomainaredistributedbetweentheservers.I'm
settinguppseudoloadbalancingbymakingeachfailoverdomainanordered(prioritized)listandusing
theprioritysettingtodistributetheworkloadbetweentheservers.
Thesharedfilesystemdefinedaspath/mnt/bobisexportedthroughNFStoclientsnamed
client1throughclient50althoughthey'renotallshownhere.Thepreferrednodeforrunningthe
NFSservicenfssvc1istrin10sincetrin10'spriorityissethigherthantheothernodesinthe
failoverdomain.Iftrin10fails,thergmanageroftheothernodeswilldetectthefailureandrelocate
theNFSservicetooneoftheothernodesdependingoncircumstances.Likewise,thepreferrednode
forthesecondNFSexportistrin11.
Step7b.Copythecluster.conffiletoallnodesinthecluster:
Whentheclusterwasdown,wehadtousescptocopythecluster.conffiletotheservers.
Nowthattheclusterisrunning,weshoulddoittherightway.Thisisaccomplishedwiththeuseof
twotools,ccs_toolandcman_toolasshownbelow.
First,propagateyourchangedcluster.conffiletothecluster:
[root@trin10~]#ccs_toolupdate/etc/cluster/cluster.conf
Next,determinetheinmemoryconfigurationversionnumber.
[root@trin10~]#cman_toolstatus|grep"Configversion"
Configversion:1

CopyrightRedHat,Inc.20062010Allrightsreserved.
Updatetheversionnumbertoyournewerversion.Sincetheoldversionwas1,wetellitto
useversion2.
[root@trin10~]#cman_toolversionr2
Step7c.Deletethe/etc/exportsfile
IfyouwantrgmanagertomanageyourNFSexports,rgmanagershouldmanagethemall.
Inotherwords,youshouldn'thaveotherNFSexportsin/etc/exportsorNFSwillgetconfused.
Ifyouneeda/etc/exportsfiletomanageNFSexportsthatarenotpartoftheclustered
filesystem,youshouldreadfurtheranduseManagedVirtualIPServiceinstead.
Otherwiseyoucanskipthenextsectionandproceedtostep8.

ManagedVirtualIPService
ThismethodofmanagingNFSfailoverismoreflexiblethanusingManagedNFSService
becauseitallowsyoutohandcraftyourNFSexportsasyouseefit.However,itisonlydesignedto
workonGFSfilesystems.UsingitfornonGFSfilesystemsisnotrecommended.
Inthisconfiguration,youmustensurethat/etc/exportsfileisinsynconallnodes,andthat
nfsdisalwaysrunningonallnodes(it'snotmonitoredbythecluster;theservicedoesn'tuse
<nfsexport>inthiscase),andthattheGFSfilesystemismountedbeforeNFSisstarted.Therefore,it
requiresmoreplanning,maintenanceandmonitoring.
Step7a.Addresourcemanagerdomains,resourcesandservicestocluster.conf
Theclusterconfigurationismuchsimplerthanbefore.Thefailoverdomainsarethesame,
buttheresourcesandservicesaresimpler.Usingyourfavoriteeditor,changethermsectionofthe
/etc/cluster/cluster.conffiletolooksomethinglikethis:
<rm>

<failoverdomains>
<failoverdomainname="igridnodes1"ordered="1"restricted="1">
<failoverdomainnodename="trin10"priority="1"/>
<failoverdomainnodename="trin11"priority="2"/>
<failoverdomainnodename="trin12"priority="2"/>
<failoverdomainnodename="trin13"priority="2"/>
<failoverdomainnodename="trin14"priority="2"/>
</failoverdomain>
<failoverdomainname="igridnodes2"ordered="1"restricted="1">
<failoverdomainnodename="trin10"priority="2"/>
<failoverdomainnodename="trin11"priority="1"/>
<failoverdomainnodename="trin12"priority="2"/>
<failoverdomainnodename="trin13"priority="2"/>
<failoverdomainnodename="trin14"priority="2"/>
</failoverdomain>
<failoverdomainname="igridnodes3"ordered="1"restricted="1">

CopyrightRedHat,Inc.20062010Allrightsreserved.
<failoverdomainnodename="trin10"priority="2"/>
<failoverdomainnodename="trin11"priority="2"/>
<failoverdomainnodename="trin12"priority="1"/>
<failoverdomainnodename="trin13"priority="2"/>
<failoverdomainnodename="trin14"priority="2"/>
</failoverdomain>
<failoverdomainname="igridnodes4"ordered="1"restricted="1">
<failoverdomainnodename="trin10"priority="2"/>
<failoverdomainnodename="trin11"priority="2"/>
<failoverdomainnodename="trin12"priority="2"/>
<failoverdomainnodename="trin13"priority="1"/>
<failoverdomainnodename="trin14"priority="2"/>
</failoverdomain>
<failoverdomainname="igridnodes5"ordered="1"restricted="1">
<failoverdomainnodename="trin10"priority="2"/>
<failoverdomainnodename="trin11"priority="2"/>
<failoverdomainnodename="trin12"priority="2"/>
<failoverdomainnodename="trin13"priority="2"/>
<failoverdomainnodename="trin14"priority="1"/>
</failoverdomain>
</failoverdomains>
<resources>
<ipaddress="10.0.0.1"monitor_link="1"/>
<ipaddress="10.0.0.2"monitor_link="1"/>
<ipaddress="10.0.0.3"monitor_link="1"/>
<ipaddress="10.0.0.4"monitor_link="1"/>
<ipaddress="10.0.0.5"monitor_link="1"/>
</resources>
<serviceautostart="1"domain="igridnodes1"name="10.0.0.1">
<ipref="10.0.0.1"/>
</service>
<serviceautostart="1"domain="igridnodes2"name="10.0.0.2">
<ipref="10.0.0.2"/>
</service>
<serviceautostart="1"domain="igridnodes3"name="10.0.0.3">
<ipref="10.0.0.3"/>
</service>
<serviceautostart="1"domain="igridnodes4"name="10.0.0.4">
<ipref="10.0.0.4"/>
</service>
<serviceautostart="1"domain="igridnodes5"name="10.0.0.5">
<ipref="10.0.0.5"/>
</service>
</rm>

Again,noticethattheprioritiesforeachfailoverdomainaredistributedbetweenthe
servers.I'msettinguppseudoloadbalancingbymakingeachfailoverdomainanordered(prioritized)
listandusingtheprioritysettingtodistributetheworkloadbetweentheservers.ThefirstVirtualIP
serviceissettorunonthefirstserver,trin10,atstartupbecauseitspriorityissethigher(1)thanthe
prioritiesoftheotherservers(2).ThesecondVirtualIPserviceissettorunonthesecondserver,trin
11,andsoforth.

CopyrightRedHat,Inc.20062010Allrightsreserved.
Step7b.ConcoctyourNFSexportsfile
First,youneedtodecideyourNFSexportsandputtheminyour/etc/exportsfile.For
example:
/mnt/bob/bin*(ro,insecure,no_root_squash)
/mnt/bob/sbin*(ro,insecure,no_root_squash)
/mnt/bob/usr/bin*(ro,insecure,no_root_squash)
CarefullychooseyourNFSexportoptionswithyourcompany'ssecurityinmind.
Exportingtoeveryone(*)isgenerallynotagoodidea.Likewise,optionslikeinsecureand
no_root_squashmightbeokayifyou'reexportingdirectorieslike/binor/usr/bin,butit'sdefinitely
notagoodideaforvaluablecompanydata.Remember,thisisjustanexample.
Step7c.Next,sendyour/etc/exportsfiletoallserversinthecluster.
SincetheNFSexportsfileisnotmanagedbytheclustersoftwareinthisconfiguration,you
needtoensurethatit'sthesameforallofyourNFSservers.
[root@trin10~]#foriin`seq1114`;doscp/etc/exportsroot@trin$i:/etc/;done
Step7d.StartyourNFSserviceandsetituptostartatboottimeonallservers:
[root@trin10~]#servicenfsstart
[root@trin10~]#foriin`seq1114`;dosshtrin$iservicenfsstart;done

Step8StarttheResourceManagerService
Next,weneedtostarttheResourceManagerservice,rgmanager,onallservers.Like
before,westarttheserviceandsetituptostartatboottime.
[root@trin10~]#servicergmanagerstart
[root@trin10~]#foriin`seq1114`;dosshtrin$iservicergmanagerstart;done
Ifwe'vedoneourjobscorrectly,thergmanagershouldstarttheNFSserviceorVirtualIP
servicewesetupearlierontheappropriateservers.Ifitdoesn'tworkforsomereason,wecanlook
in/var/log/messagestofindoutwhy.

Step9SetupDHCP
ItmaybeoutofthescopeofthisdocumenttosetupDHCP,butsomehowyouneedtoget
yourfiftyNFSclientstousethefivevirtualIPaddressesinapseudoloadbalancedway.Onewayyou

CopyrightRedHat,Inc.20062010Allrightsreserved.
candothatistoconfigureDHCPononeoftheserverstoassignIPaddressestoyourdisklessNFS
clientsandexporttheirrootdirectories.
Forexample,youcansetuprootpartitionsforeachofyourclientsonyourGFSfilesystem.
Youmighthavefiftyentriesin/etc/dhcpd.confthatlooksomethinglikethis:
hostclient1{

optionrootpath"/mnt/bob/client1/";
filename"/mnt/bob/bootImage";
hardwareethernet00:60:18:3F:00:01;
fixedaddress10.0.1.1;
}
hostclient2{
optionrootpath"/mnt/bob/client2/";
filename"/mnt/bob/bootImage";
hardwareethernet00:60:18:3F:00:02;
fixedaddress10.0.1.2;
}
(andsoforthforeveryclient)

Whentheclientsarebooted,theyrequesttheirIPaddressfromtheDHCPserver.Once
assigned,theycanusesomemechanismliketftptoloadtheirkernelandNFStomounttheirownroot
directoriesRead/Write.
Forstaticdirectoriesthatcanbereadonly,allofyourclientscouldaccessthesame
physicaldirectoriesthroughNFSanddistributetheworkloadthroughoutthefiveserversbyusingthe
fiveVirtualIPs.Forexample,/bin,/usr/bin,and/usr/libcanallbeNFSmounts.Inclient1'sfstab
(file/mnt/bob/client1/etc/fstabontheservers)wehavesomethinglike:
/dev/nfs/nfsdefaults00
none/procprocdefaults00
10.0.0.1:/mnt/bob/bin.i686/binnfsdefaults,ro00
10.0.0.1:/mnt/bob/usrbin.i686/usr/binnfsdefaults,ro00
10.0.0.1:/mnt/bob/usrlib.i686/usr/libnfsdefaults,ro00

Thefstabfilewouldbethesameforclients6,11,16,21,26,andsoforth,soeveryfifth
clientwouldbeusingthesameVirtualIPandtherefore,thesamephysicalserver.
Thefstabforclient2(/mnt/bob/client2/etc/fstab)issimilar,butaccomplishespseudoload
balancingbyusingadifferentVirtualIPwhichaccessesthesamedatafromadifferentserver.Same
forclients17,22,27,32,etc.Wehave:
/dev/nfs/nfsdefaults00
none/procprocdefaults00
10.0.0.2:/mnt/bob/bin.i686/binnfsdefaults,ro00
10.0.0.2:/mnt/bob/usrbin.i686/usr/binnfsdefaults,ro00
10.0.0.2:/mnt/bob/usrlib.i686/usr/libnfsdefaults,ro00

Andofcourse,clients3,8,13,18,23,etc.,wouldmountfrom10.0.0.3.Clients4,9,14,
19,24wouldallmountfrom10.0.0.4and5,10,15,20,25wouldallmountfrom10.0.0.5.Allfiftyof
theclientswouldhavesimilarconfigurationstoaccomplishthepseudoloadbalancing.
Icouldhavealsospecifiedclients1through10touseserver1,11through20touseserver

CopyrightRedHat,Inc.20062010Allrightsreserved.
2,andsoforth.Ipurposelychosetostaggertheminstead,andthatgoesbacktothedesignofthe
network.Ifyouassignclients1through10toonedepartmentand11through20toanotherdepartment
(therebymakingiteasiertofindhardwareproblemsdowntheroad),ourserverworkloadmaybecome
unbalancedduetothedemandsofthedifferentdepartments.Youdon'twantonedepartmentto
hammeraserverhardwhileanotherdepartment'sserverdoesnothingbutsitandeatbonbons.It's
betterforalldepartmentstotaxtheserversequally.Whichbringsustoournexttopic.

TrueLoadBalancing
Whatwe'vedonesofarisnottrueloadbalancing.Wedistributedtheclientsequallyacross
theNFSservers,butsomeofyourserversmaystillbepoundedbyrequestswhileotherssitidle.If
yourfiftyworkstationsaredividedamongdifferentdepartments,youmighthaveonedepartment
hammeringoneoftheserversontheweekend(forexample,production)whileanotherserveris
completelyidle(forexample,becausetheshippingdepartmentisclosedfortheweekend.)
Ifyoureallyneedtodistributetheworkloadequallyamongtheservers,youneedtrueload
balancing,andthataddsanotherlayerofcomplexity.ThebestwaytoachievethatisbyusingLinux
VirtualServer(LVS).LVSusesamechanismcalledNetworkAddressTranslation(NAT)toroute
requestsfromoneIPaddresstoanotherandthat'swhatachievestrueloadbalancing.
WithLVS,youhaveasecondsetofservers(calledLVSrouters)whosejobistoequally
distributetherequests.Onlyonerouterisactiveatatime;additionalroutersareneededtoprovide
failovercapabilitiesincasethefirstrouterfails.Ourexamplediagrammightlooklikethis:

CopyrightRedHat,Inc.20062010Allrightsreserved.
WithLVS,allrequestscomeintoacentralLVSserverknownastheActiveRouter.The
routerdecideswhichservertogivetherequestto,basedononeofseveralselectablemethods.A
secondrouter(calledthebackuprouter)monitorsthenetworkandtakesoveriftheactiverouterfails.
RedHatClusterSuitecomeswithagraphicalLVSconfigurationtoolcalledPiranha.Ithas
awebinterfaceandrequiresphptorun.
Ifyoureallywanttogetfancy,youcanaddasecondtierofservers,onetierdoingtheNFS
fileservingandthesecondtierservingthatNFSdatatotheweb.Ofcourse,itgetshardertoconfigure
andmaintainthesenetworkswitheachlevelofcomplexity.

MinimizingDowntimeonMissionCriticalServers
I'veshownyouhowRedHatClusterSuitecanprovidefailovercapabilitiesincaseaserver
fails,butthat'snottheendofthestory.Ifyoulookatthediagramofourclusterabove,therearestill
severalpointsoffailure.IfyourNFSserversaremissioncritical,youmaywanttodoeverythingin
yourpowertomakesurethereisnodowntime.
Thereareseveralstepsyoucantaketoensureyoursystemsstayupbarringacitywide
powerfailure.It'soutofthescopeofthisdocumenttoshowyouhowtosetthoseup,butI'dberemiss
ifIdidn'tatleastmentionthem.Mostofthemarehardwarerelated.

RedundantPowerSuppliesInsteadofhavingonepowersupplyinaserver,youhavetwo.
Thisprovidesredundantpowerifonepowersupplyfails.
UninterruptiblePowerSupply(UPS)Providestemporarycontinuityofpowerifthe
electricitygoesout.
MultipathingThebasicideahereistohavetwoSAN/Fibreswitcheswithredundantcables.
Thatallowsthedataontheharddrivestobereadevenifoneoftheswitcheshasaproblem.
RAID,Mirroring,etc.Thedataontheharddrivesiscopied(mirroring)orarrangedonthe
diskstoallowfordatarecoveryincaseaharddrivefails.RAIDstandsforRedundantArrayof
IndependentDisks.
NetworkRedundancyTheideahereistohavemorethanonenetworkswitchorhuband
redundantNICconnections.Thatway,ifaswitchorhubfails,theclustercanstillreachthe
networkthroughtheotherswitchorhub.Also,ifaNICoranetworkcablefailsinaserver,
thereisstillapathfromthatservertothenetworkthroughasecondNICandcable.
SiteRedundancyTheideahereistohavetwodifferentlocationsforyourservers,maybe
evenindifferentcities.Thatway,eveniftheentiresiteistakendownbysomedisasterorloss
ofpower,you'llhaveanothersitetobackthemup.ThisissometimesknownasSiteMirroring.

Wheretogoforhelp
Atfirstthiscanallseemoverwhelming,butonceyougetintoit,it'snotthathard.
Fortunately,thereareplaceswhereyoucangoforhelpandtogetquestionsanswered.Youcan:

ReadtheClusterProjectPage:
http://sources.redhat.com/cluster/wiki/

CopyrightRedHat,Inc.20062010Allrightsreserved.

SearchtheClusterSuiteFrequentlyAskedQuestions:
http://sources.redhat.com/cluster/wiki/FAQ
SubscribetotheLinuxClusterMailingListandaskthere.Formoreinformationvisit:
https://www.redhat.com/mailman/listinfo/linuxcluster