12/5/2016

5 Ways to Improve your Production Web Application Server Setup | DigitalOcean

Menu

Communit

uscrie

:MitchellAnicas

5WastoImproveourProductionWeApplicationerveretup
Posted Feruar11,2015

58.4k

RVROPTIMIZATION

ACKUP

LOGGING

LOADALANCING

HIGHAVAILAILITY

MONITORING

108

CONFIGURATIONMANAGMNT

Introduction
Onceourapplicationisupandrunninginacloudserverenvironment,oumaewonderinghowoucanimproveourserverenvironmentto
maketheleapfrom"itworks"toafull-fledgedproductionenvironment.Thisarticlewillhelpougetstartedwithplanningandimplementinga
productionenvironmentcreatingaloosedefinitionof"production",inthecontextofaweapplicationinacloudserverenvironment,and
showingousomecomponentsthatoucanaddtoourexistingarchitecturetomakethetransition.
Forthepurposesofthisdemonstration,let'sassumethatwe'restartingwithasetupsimilartoonedescriedin5Commonerveretups,likethis
two-serverenvironmentthatsimplservesaweapplication:

Youractualsetupmightesimplerormorecomplex,utthegeneralideasandcomponentsdiscussedhereinshouldappltoanserver
environmenttoasomeextent.
Let'sgetstartedwithdefiningwhatwemeanwhenwesa"productionenvironment".

https://www.digitalocean.com/community/tutorials/5-ways-to-improve-your-production-web-application-server-setup?utm_source=Customerio&utm_medium=

1/10

12/5/2016

5 Ways to Improve your Production Web Application Server Setup | DigitalOcean

WhatisaProductionnvironment?
Aserverenvironmentforaweapplication,inageneralsense,consistsofthehardware,software,data,operationalplans,andpersonnelthatare
necessartokeeptheapplicationworking.Aproductionenvironmenttpicallreferstoaserverenvironmentthatwasdesignedandimplemented
withutmostconsiderationforacceptalelevelsofthesefactors:
Availailit:Theailitfortheapplicationtoeusaleitsintendedusersduringadvertisedhours.Availailitcanedisruptedanfailure

thataffectsacriticalcomponentseverelenough(e.g.theapplicationcrashesduetoaug,thedataasestoragedevicefails,orthesstem
administratoraccidentallpowersofftheapplicationserver).
Onewatopromoteavailailitistodecreasethenumerofsinglepointsoffailureinanenvironment.Forexample,usingastaticIPanda
monitoringfailoverserviceensuresthatusersonlaccesshealthloadalancers.Tolearnmore,readthissectionofHowToUseFloatingIPsand
thisarticleonloadalancing.
Recoverailit:Theailittorecoveranapplicationenvironmentintheeventofsstemfailureordataloss.Ifacriticalcomponentfails,andisnot

recoverale,availailitwillecomenon-existent.Improvingmaintainailit,arelatedconcept,reducesthetimeneededtoperformagiven
recoverprocessintheeventofafailure,andthereforecanimproveavailailitintheeventofafailure
Performance:Theapplicationperformsasexpectedunderaverageorpeakload(e.g.itisreasonalresponsive).Whileverimportanttoour

users,performanceonlmattersiftheapplicationisavailale
Takesometimetodefineacceptalelevelsforeachoftheitemsjustmentioned,inthecontextofourapplication.Thiswillvardependingonthe
importanceandnatureoftheapplicationinquestion.Forexample,itisproalacceptaleforapersonallogthatservesfewvisitorstosufferfrom
occasionaldowntimeorpoorperformance,aslongasthelogcanerecovered,utacompan'sonlinestoreshouldstriveverhighmarksacross
theoard.Ofcourse,itwouldenicetoachieve100%inevercategor,foreverapplication,utthatisoftennotfeasileduetotimeandmone
constraints.
Notethatwehavenotmentioned(a)hardwarereliailit,theproailitthatagivenhardwarecomponentwillfunctionproperlforaspecified
amountoftimeeforefailure,or()securitasfactors.Thisisecauseweareassuming(a)thecloudserversouareusingaregenerallrelialeut
havethepotentialforfailure(astherunonphsicalservers),and()ouarefollowingsecuritestpracticestotheestofourailitiessimpl
put,theareoutsideofthescopeofthisarticle.Youshouldeaware,however,thatreliailitandsecuritarefactorsthatcandirectlaffect
availailit,andothcancontriutetheneedforrecoverailit.
Insteadofshowingouastep--stepprocedureforcreatingaproductionenvironment,whichisimpossileduethevaringneedsandnatureof
everapplication,wewillpresentsometangilecomponentsthatcanutilizetotransformourexistingsetupintoaproductionenvironment.
Let'stakealookatthecomponents!

1.ackupstem
Aackupsstemwillgrantouwiththeailittocreateperiodicackupsofourdata,andrestoredatafromackups.ackupsalsoallowfor
rollacksinourdata,toapreviousstate,intheeventofaccidentaldeletionorundesiredmodification,whichcanoccurduetoavarietofreasons
includinghumanerror.Allcomputerhardwarehasachanceoffailureatsomepointintime,whichcanpotentiallcausedataloss.Withthisinmind,
oushouldmaintainrecentackupsofallourimportantdata.
RequiredforProduction?Yes.Aackupsstemcanmitigatetheeffectsofdataloss,whichisnecessartoachieverecoverailitand,therefore,

aidsavailailitintheeventofdatalossutitmusteusedinconjunctionwithsolidrecoverplans,whicharediscussedinthenextsection.Note
thatDigitalOcean'ssnapshot-asedackupsmanotesufficientforallofourackupneeds,asitisnotwell-suitedformakingackupsofactive
dataasesandotherapplicationswithhighdiskwriteI/Oifourunthesetpesofapplications,orwantmoreackupschedulingflexiilit,esure
touseanotherackupsstemsuchasacula.

https://www.digitalocean.com/community/tutorials/5-ways-to-improve-your-production-web-application-server-setup?utm_source=Customerio&utm_medium=

2/10

12/5/2016

5 Ways to Improve your Production Web Application Server Setup | DigitalOcean

Thediagramaoveisanexampleofaasicackupsstem.Theackupserverresidesinthesamedatacenterastheapplicationservers,where
theinitialackupsarecreated.Later,off-sitecopiesoftheackupsaremadetoaserverinadifferentdatacentertoensurethedataispreservedin
thecaseof,sa,anaturaldisaster.

Considerations
ackupelection:Thedatathatouwillackup.Minimall,ackupandatathatoucan'trelialreproducefromanalternativesource
ackupchedule:Whenandhowfrequentlouwillperformfullorincrementalackups.pecialconsiderationsmustetakenforackupsof

certaintpesofdata,suchasactivedataases,whichcanaffectourackupschedule
DataRetentionPeriod:Howlongouwillkeepourackupseforedeletingthem
Diskpaceforackups:Thecominationofthreepreviousitemsaffectstheamountofdiskspaceourackupsstemwillrequire.Take

advantageofcompressionandincrementalackupstodecreasethediskspacerequiredourackups
Off-siteackups:Toprotectourackupsagainstlocaldisasters,withinaparticulardatacenter,itisadvisaletomaintainacopofourackups

inageographicallseparatelocation.Inthediagramaove,theackupsofNYC3arecopiedtoFO1forthispurpose
ackupRestorationTests:Periodicalltestourackuprestorationprocesstomakesurethatourackupsworkproperl

RelatedTutorials
HowToChooseanffectiveackuptrategforourVP
HowToInstallaculaerveronUuntu14.04
HowToUseRsnctoncLocalandRemoteDirectoriesonaVP
UnderstandingDigitalOceanDropletackups

2.RecoverPlans
Recoverplansareasetofdocumentedprocedurestorecoverfrompotentialfailuresoradministrationerrorswithinourproductionenvironment.
Atminimum,ouwillwantarecoverplanforeachcripplingscenariothatoudeemwillinevitaloccur,suchasserverhardwarefailureor
accidentaldatadeletion.Forexample,averasicrecoverplanforaserverfailurecouldconsistofalistofthestepsthatoutooktoperformour
initialserverdeploment,withextraproceduresforrestoringapplicationdatafromackups.Aetterrecoverplanmight,inadditiontogood
documentation,leveragedeplomentscriptsandconfigurationmanagementtools,suchasAnsile,Chef,orPuppet,tohelpautomateandquicken
therecoverprocess.
RequiredforProduction?Yes.Althoughrecoverplansdon'texistassoftwareinourserverenvironment,theareanecessarcomponentfora

productionsetup.Theenaleoutoutilizeourackupseffectivel,andprovidealueprintforreuildingourenvironmentorrollingacktoa
desiredstatewhentheneedarises.

https://www.digitalocean.com/community/tutorials/5-ways-to-improve-your-production-web-application-server-setup?utm_source=Customerio&utm_medium=

3/10

12/5/2016

5 Ways to Improve your Production Web Application Server Setup | DigitalOcean

Thediagramaoveisanoverviewofarecoverplanforafaileddataaseserver.Inthiscase,thedataaseserverwillereplacedanewone
withthesamesoftwareinstalled,andthelastgoodackupwilleusedtorestoretheserverconfigurationanddata.Lastl,theappserverwille
configuredtousethenewdataaseserver.

Considerations
ProcedureDocumentation:Thesetofdocumentsthatshouldefollowedinafailureevent.Agoodstartingpointisuildingastep--step

documentthatoucanfollowtoreuildafailedserver,thenaddingstepsforrestoringthevariousapplicationdataandconfigurationfrom
ackups
AutomationTools:criptsandconfigurationmanagementsoftwareprovideautomation,whichcanimprovedeplomentandrecoverprocesses.

Whilestep--stepguidesareoftenadequateforsimplrecoveringfromafailure,themusteexecutedapersonandthereforearenotas
fastorconsistentasanautomatedprocess
CriticalComponents:Thecomponentsthatarenecessarforourapplicationtofunctionproperl.Intheexampleaove,oththeapplication

anddataaseserversarecriticalcomponentsecauseifeitherfails,theapplicationwillecomeunavailale
inglePointsofFailure:Criticalcomponentsthatdonothaveanautomaticfailovermechanismareconsideredtoeasinglepointoffailure.You

shouldattempttoeliminatesinglepointsoffailure,totheestofourailit,toimproveavailailit
Revisions:Updateourdocumentationasourdeplomentandrecoverprocessimproves

3.Loadalancing
Loadalancingcaneaddedtoaserverenvironmenttoimproveperformanceandavailailitdistriutingtheworkloadacrossmultipleservers.
Ifoneoftheserversthatisloadalancedfails,theotherserverswillhandletheincomingtrafficuntilthefailedserverecomeshealthagain.Ina
cloudserverenvironment,loadalancingtpicallcaneimplementedaddingaloadalancerserver,thatrunsloadalancer(reverseprox)
software,infrontofmultipleserversthatrunaparticularcomponentofanapplication.
RequiredforProduction?Notnecessaril.Loadalancingisnotalwasrequiredforaproductionenvironmentutitcaneaneffectivewato

reducethenumerofsinglepointsoffailureinasstem,ifimplementedcorrectl.Itcanalsoimproveperformanceaddingmorecapacitthrough
horizontalscaling.

Thediagramaoveaddsanadditionalappservertosharetheload,andaloadalancertospreaduserrequestsacrossothappservers.This
setupcanhelpwithperformance,ifthesingleappserverwasstrugglingtokeepupwiththetraffic,anditcanalsohelpkeeptheapplication
https://www.digitalocean.com/community/tutorials/5-ways-to-improve-your-production-web-application-server-setup?utm_source=Customerio&utm_medium=

4/10

12/5/2016

5 Ways to Improve your Production Web Application Server Setup | DigitalOcean

availaleifoneoftheapplicationserversfails.However,itstillhastwosinglepointsoffailureinthedataaseserverandtheloadalancerserver
itself.

Considerations
LoadalancealeComponents:Notallcomponentsinanenvironmentcaneloadalancedeasil.pecialconsiderationmustemadefor

certaintpesofsoftwaresuchasdataasesorstatefulapplications
ApplicationDataReplication:Ifaloadalancedapplicationserverstoresapplicationdatalocall,suchasuploadedfiles,thisdatamustemade

availaletotheotherapplicationserversviamethodssuchasreplicationorsharedfilesstems.Thisisnecessartoensurethattheapplication
datawilleavailalenomatterwhichapplicationserverischosentoserveauserrequest
Performanceottlenecks:Ifaloadalancerdoesnothaveenoughresourcesorisnotconfiguredproperl,itcanactualldecreasethe

performanceofourapplication
inglePointsofFailure:Whilealoadalancingcaneusedtoeliminatesinglepointsoffailure,poorlplannedloadalancingcanactualladd

moresinglepointsoffailure.Loadalancingisenhancedwiththeinclusionofasecondload-alancerwithastaticIPinfrontofthepairthat
sendstraffictooneortheotherdependingonavailailit.

RelatedTutorials
AnIntroductiontoHAProxandLoadalancingConcepts
HowToImplementLTerminationWithHAProxonUuntu14.04
HowToUseHAProxAsALaer7LoadalancerForWordPressandNginxOnUuntu14.04
UnderstandingNginxHTTPProxing,Loadalancing,uffering,andCaching*HowToUseFloatingIPs.

4.Monitoring
Monitoringcansupportaserverenvironmenttrackingthestatusofservicesandthetrendsofourserverresourceutilization,thusproviding
greatvisiilitintoourenvironment.Oneoftheiggestenefitsofmonitoringsstemsisthatthecaneconfiguredtotriggeranaction,suchas
runningascriptorsendinganotification,whenaserviceorservergoesdown,orifacertainresource,suchasCPU,memor,orstorage,ecomes
over-utilized.Thesenotificationsenaleoutoreacttoanissuesassoonastheoccur,whichcanhelpminimizeorpreventthedowntimeofour
application.
RequiredforProduction?Notnecessaril,uttheneedformonitoringincreasesasaproductionenvironmentgrowsinsizeandcomplexit.It

providesaneaswatokeeptrackofourcriticalservicesandserverresources.Inturn,monitoringcanimprovetherecoverailit,andinformthe
planningandmaintenanceofoursetup.

ignupforournewsletter.
GetthelatesttutorialsonsAdminandopensourcetopics.
nterouremailaddress

ignUp

Thediagramaoveisanexampleofamonitoringsstem.Tpicall,themonitoringserverwillrequeststatusdatafromagentsoftwarerunningon
C R O L L TO TO P
theapplicationanddataaseservers,andeachagentwillrespondwithsoftwareandhardwarestatusinformation.Theadministrator(s)ofthesstem

canthenusethemonitoringconsoletolookattheoverallstateoftheapplication,anddrilldowntomoredetailedinformation,asneeded.

Considerations
ervicestoMonitor:Theservicesandsoftwarethatouwillmonitor.Minimall,oushouldmonitorthestateofalloftheservicesthatneedtoe

inahealthrunningstateforourapplicationtofunctionproperl
ResourcestoMonitor:Theresourcesthatouwillmonitor.xamplesofresourcesincludeCPU,memor,storage,andnetworkutilization,andthe

stateofserverasawhole
DataRetention:Theperiodoftimethatouretainmonitoringdataeforediscardingit.This,alongwithourchoiceofitemstomonitor,willaffect

theamountofdiskspacethatourmonitoringsstemwillrequire

https://www.digitalocean.com/community/tutorials/5-ways-to-improve-your-production-web-application-server-setup?utm_source=Customerio&utm_medium=

5/10

12/5/2016

5 Ways to Improve your Production Web Application Server Setup | DigitalOcean

ProlemDetectionRules:ThethresholdsandrulesthatdeterminewhetheraserviceorresourceisinaOKstate.Forexample,aserviceorserver

maeconsideredtoehealthifitisrunningandservingrequests,whereasaresource,suchasstorage,mighttriggerawarningifitsutilization
reachesacertainthresholdforacertainamountoftime
NotificationRules:Thethresholdsandrulesthatdetermineifanotificationshouldesent.Whilenotificationsareimportant,itisequallimportant

totuneournotificationrulessothatoudon'treceivetooman;aninoxfullofwarningsandalertswilloftengoignored,makingthemalmostas
uselessasnonotificationsatall

RelatedTutorials
HowToInstallNagios4andMonitorYourerversonUuntu14.04
HowToUseIcingaToMonitorYourerversandervicesOnUuntu14.04
HowToInstallZaixonUuntu&ConfigureittoMonitorMultipleVPervers
MonitoringandManagingourNetworkwithNMP
HowToConfigureensuMonitoring,RaitMQ,andRedisonUuntu14.04

5.CentralizedLogging
Centralizedloggingcansupportaserverenvironmentprovidinganeaswatoviewandsearchourlogs,whicharenormallstoredlocallon
individualserversacrossourentireenvironment,inasingleplace.Asidefromtheconvenienceofnothavingtologintoindividualserverstoread
logs,centralizedloggingalsoallowsoutoeasilidentifissuesthatspanmultipleserverscorrelatingtheirlogsandmetricsduringaspecific
timeframe.Italsograntsmoreflexiilitintermsoflogretentionecauselocallogscaneoff-loadedfromapplicationserverstoacentralizedlog
serverthathasitsown,independentstorage.
RequiredforProduction?No,utlikemonitoring,centralizedloggingcanprovideinvalualeinsightintoourserverenvironmentasitgrowsinsize

andcomplexit.Inadditiontoeingmoreconvenientthantraditionallogging,itenalesoutorapidlauditourserverlogswithgreatervisiilit.

Thediagramaoveisasimplifiedexampleofacentralizedloggingsstem.Alogshippingagentisinstalledoneachserver,andconfiguredtosend
importantappanddataaselogstothecentralizedloggingserver.Theadministrator(s)ofthesstemcanthenview,filter,andsearchallofthe
importantlogsfromasingleconsole.

Considerations
ignupforournewsletter.
LogstoGather:Theparticularlogsthatouwillshipfromourserverstothecentralizedloggingserver.Youshouldgathertheimportantlogs

fromallofourservers
GetthelatesttutorialsonsAdminandopensourcetopics.
DataRetention:
Theperiodoftimethatouretainlogseforediscardingthem.This,alongwithourchoiceoflogstogather,willaffectthe
nterouremailaddress

ignUp
amountofdiskspacethatourcentralizedloggingsstemwillrequire
LogFilters:Thefiltersthatparseplainlogsintostructuredlogdata.Filteringlogswillimproveourailittoquer,analze,andgraphthedatain

meaningfulwas

C R O L L TO TO P

erverClocks:nsurethattheclocksofourserversaresnchronizedandusingsettothesametimezone,soourlogtimelineisaccurate

acrossourentireenvironment

RelatedTutorials
HowToInstalllasticsearch,Logstash,andKiana4onUuntu14.04
HowToUsetheDigitalOceanLKtackOne-ClickApplication
IntroductiontoTrackingtatisticsonervers
HowToInstallGralog2AndCentralizeLogsOnUuntu14.04

https://www.digitalocean.com/community/tutorials/5-ways-to-improve-your-production-web-application-server-setup?utm_source=Customerio&utm_medium=

6/10

12/5/2016

5 Ways to Improve your Production Web Application Server Setup | DigitalOcean

Conclusion
Whenouputallthecomponentstogether,ourproductionenvironmentmightlooksomethinglikethis:

Nowthatouarefamiliarwithcomponentsthatcaneusedtosupportandimproveaproductionserversetup,oushouldconsiderhowoucan
integratethemourownserverenvironments.Ofcourse,wedidn'tcovereverpossiilit,utthisshouldgiveouanideaofwheretogetstarted.
Rememertodesignandimplementourserverenvironmentasedonaalanceofouravailaleresourcesandourownproductiongoals.
Ifouareinterestedinsettingupanenvironmentliketheoneaove,checkoutthistutorial:uildingforProduction:WeApplications.

Upvote

:MitchellAnicas

(108)

uscrie

pinupanDcloudserverinunderaminute.
implesetup.Fullrootaccess.
traightforwardpricing.
DPLOYRVR

RelatedTutorials
ignupforournewsletter.

HowToAddthegzipModuletoNginxonUuntu16.04

HowToConfigurePeriodicTRIMforDtorageonLinuxervers
GetthelatesttutorialsonsAdminandopensourcetopics.
HowToCentralizeLogswithRsslog,Logstash,andlasticsearchonUuntu14.04
nterouremailaddress

ignUp
HowToChangeYourPHPettingsonUuntu14.04

HowToAddthegzipModuletoNginxonCentO7
C R O L L TO TO P

11Comments

Leaveacomment...

https://www.digitalocean.com/community/tutorials/5-ways-to-improve-your-production-web-application-server-setup?utm_source=Customerio&utm_medium=

7/10

12/5/2016

5 Ways to Improve your Production Web Application Server Setup | DigitalOcean

LogIntoComment

IMsupporting Feruar16,2015
0

Re:Loadalancing.
Whatdoourecommendsoftwarewise?

manicas
1

MOD

Feruar16,2015

NginxandHAProxarepopularreverseproxiesthatcaneusedforloadalancing.CheckouttheRecommendedTutorialsintheloadalancer
sectionofthisarticle.

IMsupporting Feruar17,2015
0

Thanksman,Illtakealook.IfiguredIshoulditetheulletandproaluseaproperLratherthanroundroinDNwithshortTTL's

FlevasGR March5,2015
0

Greatartictle!Iwouldliketoseemoreproduction-readarticles

korkoszrafal March6,2015
0

Greatarticle!Iwouldaddcacheservertothisstack(Redis,Varnish)

agnasrodaln Ma25,2015
0

Re:CentralizedLogging
Whatlogshippingagentwouldourecommend?

manicas
0

MOD

Ma26,2015

Itdependsonwhichsoftwareouareusingtocollectthelogs,andcompatiilitwiththedevicesthatouwanttogatherlogsfrom.
Logstash,ApacheFlume,Fluentd,andrsslogarepopularloggatheringoptions.

lmouden June19,2015
0

thanksforthegreateffort
howmandropletsthisconfigurationneeds?theminimumconfiurationandhowtoscaleitwithtimewhenthetraffic/storageisgettingigger?

ignupforournewsletter.

GetthelatesttutorialsonsAdminandopensourcetopics.

avildosola Jul5,2015
1

nterouremailaddress

ignUp
Manicas,congatulationsmfriend,excelenttutorial!!!,hereinChileDigitalOceanisverfamouseplatform,pleasefollowmakinfgoodthings;)

C R O L L TO TO P

katsum0to Octoer16,2015
0

Thanks,thesharingknowledgeisverexcitinganduseful

zaix August15,2016
0

Re:Increaseservers'availailitusingZaixserver3.03
DearMitchellAnicas
Couldoukindlhelpmehowto"Increaseservers'availailit"usingZaixserver3.03?
pleaseprovidesomeinformationorwelinksforfurtherreading.reallreallappreciateourhelp
andcomments.

https://www.digitalocean.com/community/tutorials/5-ways-to-improve-your-production-web-application-server-setup?utm_source=Customerio&utm_medium=

8/10

12/5/2016

5 Ways to Improve your Production Web Application Server Setup | DigitalOcean

ThisworkislicensedunderaCreative
CommonsAttriution-NonCommercialhareAlike4.0InternationalLicense.

Copright2016DigitalOceanInc.
Communit Tutorials Questions Projects Tags Newsletter R
Distros&One-ClickApps Terms,Privac,&Copright ecurit Reportaug GetPaidtoWrite hop

ignupforournewsletter.
GetthelatesttutorialsonsAdminandopensourcetopics.
nterouremailaddress

ignUp

C R O L L TO TO P

https://www.digitalocean.com/community/tutorials/5-ways-to-improve-your-production-web-application-server-setup?utm_source=Customerio&utm_medium=

9/10

12/5/2016

5 Ways to Improve your Production Web Application Server Setup | DigitalOcean

ignupforournewsletter.
GetthelatesttutorialsonsAdminandopensourcetopics.
nterouremailaddress

ignUp

C R O L L TO TO P

https://www.digitalocean.com/community/tutorials/5-ways-to-improve-your-production-web-application-server-setup?utm_source=Customerio&utm_medium

10/10